2023-08-25T09:31:34.993 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-25T09:31:34.994 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --archive /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327 --owner scheduled_rishabh@teuthology --name rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi --verbose -- /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/orig.config.yaml 2023-08-25T09:31:35.041 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-25T09:31:35.137 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327 branch: wip-rishabh-2023aug3-b5 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7379327' kernel: flavor: default kdb: true sha1: a6a69670761025e039dc5ae86619b0cf04c46dc2 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-2023aug3-b5 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: test-nfs-pr-52556 sha1: 4b369cf18ed1391a426ab4ae86da834e9c074f81 owner: scheduled_rishabh@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 sleep_before_teardown: 0 suite: fs suite_branch: test-nfs-pr-52556 suite_path: /home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa suite_relpath: qa suite_repo: https://github.com/rishabh-d-dave/ceph suite_sha1: 4b369cf18ed1391a426ab4ae86da834e9c074f81 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGK5rZRhD9wTEpAxb8Xg5U6DSQJKyvneMkKBiMVKngkXaBc4/SOTGnZ0oZgIeugfCUb2mfTjamoEClErqT06Dgg= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKTJ01r1w1pvwRpAElBNol36OwGpEbAZ+Te5ggmgsQD6Q7jm4EX3vtmJk4+R/SANSZWTU3UzOFyPs4IP3cYEwg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 449a1bc2027504e7b3c3d7b30fa4178906581da7 timestamp: 2023-08-25_06:38:25 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2224 2023-08-25T09:31:35.137 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa; will attempt to use it 2023-08-25T09:31:35.138 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks 2023-08-25T09:31:35.138 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-25T09:31:35.141 INFO:teuthology.task.internal:Checking packages... 2023-08-25T09:31:35.172 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ba6b39e4f4f77fb95c747099b53db0467d435f56' 2023-08-25T09:31:35.173 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-25T09:31:35.173 INFO:teuthology.packaging:ref: None 2023-08-25T09:31:35.174 INFO:teuthology.packaging:tag: None 2023-08-25T09:31:35.174 INFO:teuthology.packaging:branch: wip-rishabh-2023aug3-b5 2023-08-25T09:31:35.174 INFO:teuthology.packaging:sha1: ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T09:31:35.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-2023aug3-b5 2023-08-25T09:31:35.395 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5703.gba6b39e4 2023-08-25T09:31:35.396 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-25T09:31:35.404 INFO:teuthology.task.internal:no buildpackages task found 2023-08-25T09:31:35.404 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-25T09:31:35.438 INFO:teuthology.task.internal:Saving configuration 2023-08-25T09:31:35.456 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-25T09:31:35.462 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T09:31:35.487 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 09:26:12.730620', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGK5rZRhD9wTEpAxb8Xg5U6DSQJKyvneMkKBiMVKngkXaBc4/SOTGnZ0oZgIeugfCUb2mfTjamoEClErqT06Dgg='} 2023-08-25T09:31:35.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 09:26:12.732064', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKTJ01r1w1pvwRpAElBNol36OwGpEbAZ+Te5ggmgsQD6Q7jm4EX3vtmJk4+R/SANSZWTU3UzOFyPs4IP3cYEwg='} 2023-08-25T09:31:35.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-25T09:31:35.515 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-08-25T09:31:35.515 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-08-25T09:31:35.515 INFO:teuthology.run_tasks:Running task console_log... 2023-08-25T09:31:35.611 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f76f9fbce50>, signals=[15]) 2023-08-25T09:31:35.613 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-25T09:31:35.623 INFO:teuthology.task.internal:Opening connections... 2023-08-25T09:31:35.623 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:31:35.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:31:35.712 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T09:31:35.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:31:35.797 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-08-25T09:31:35.805 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2023-08-25T09:31:35.823 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2023-08-25T09:31:35.824 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2023-08-25T09:31:35.881 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2023-08-25T09:31:35.881 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2023-08-25T09:31:35.881 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2023-08-25T09:31:35.881 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2023-08-25T09:31:35.881 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2023-08-25T09:31:35.881 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-25T09:31:35.882 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-25T09:31:35.883 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-25T09:31:35.883 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-08-25T09:31:35.903 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2023-08-25T09:31:35.920 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2023-08-25T09:31:35.920 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2023-08-25T09:31:35.976 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2023-08-25T09:31:35.976 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2023-08-25T09:31:35.977 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2023-08-25T09:31:35.978 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-25T09:31:35.978 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2023-08-25T09:31:35.978 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-25T09:31:35.978 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-25T09:31:35.978 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-25T09:31:35.978 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2023-08-25T09:31:36.003 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-08-25T09:31:36.014 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-08-25T09:31:36.029 INFO:teuthology.task.internal:Checking for old test directory... 2023-08-25T09:31:36.030 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2023-08-25T09:31:36.032 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2023-08-25T09:31:36.047 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-08-25T09:31:36.055 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-08-25T09:31:36.056 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2023-08-25T09:31:36.089 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2023-08-25T09:31:36.116 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-08-25T09:31:36.172 INFO:teuthology.run_tasks:Running task kernel... 2023-08-25T09:31:36.188 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': 'a6a69670761025e039dc5ae86619b0cf04c46dc2'} 2023-08-25T09:31:36.189 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': 'a6a69670761025e039dc5ae86619b0cf04c46dc2'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': 'a6a69670761025e039dc5ae86619b0cf04c46dc2'}}, timeout 300 2023-08-25T09:31:36.190 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-25T09:31:36.190 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-25T09:31:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T09:31:36.206 DEBUG:teuthology.task.kernel:sha1 for mon.b is a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:31:36.206 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "a6a69670761025e039dc5ae86619b0cf04c46dc2"... 2023-08-25T09:31:36.207 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2023-08-25T09:31:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T09:31:36.208 DEBUG:teuthology.task.kernel:sha1 for mon.a is a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:31:36.208 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "a6a69670761025e039dc5ae86619b0cf04c46dc2"... 2023-08-25T09:31:36.208 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-08-25T09:31:36.261 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-239.el9.x86_64 2023-08-25T09:31:36.262 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-239.el9.x86_64 vs a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:31:36.262 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-08-25T09:31:36.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:31:36.266 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-239.el9.x86_64 2023-08-25T09:31:36.266 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-239.el9.x86_64 vs a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:31:36.266 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-08-25T09:31:36.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:31:36.404 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum upgrade -y linux-firmware 2023-08-25T09:31:36.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum upgrade -y linux-firmware 2023-08-25T09:31:38.362 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 14 MB/s | 6.2 MB 00:00 2023-08-25T09:31:38.811 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 9.1 MB/s | 6.2 MB 00:00 2023-08-25T09:31:39.671 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2023-08-25T09:31:40.117 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2023-08-25T09:31:40.869 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 7.8 MB 00:00 2023-08-25T09:31:40.960 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.8 MB 00:00 2023-08-25T09:31:42.728 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-08-25T09:31:44.279 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 7.3 MB/s | 18 MB 00:02 2023-08-25T09:31:46.337 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 14 kB 00:00 2023-08-25T09:31:47.546 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 16 MB/s | 18 MB 00:01 2023-08-25T09:31:47.796 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 14 kB 00:00 2023-08-25T09:31:49.044 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 16 MB/s | 18 MB 00:01 2023-08-25T09:31:53.375 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-08-25T09:31:53.387 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-08-25T09:31:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-08-25T09:31:53.388 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-08-25T09:31:53.388 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2023-08-25T09:31:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: linux-firmware noarch 20230814-139.el9 CentOS-BaseOS 307 M 2023-08-25T09:31:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:31:53.389 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-08-25T09:31:53.389 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2023-08-25T09:31:53.389 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 1 Package 2023-08-25T09:31:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:31:53.389 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 307 M 2023-08-25T09:31:53.389 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-08-25T09:31:55.031 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-08-25T09:31:55.044 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-08-25T09:31:55.044 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-08-25T09:31:55.044 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-08-25T09:31:55.044 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout: linux-firmware noarch 20230814-139.el9 CentOS-BaseOS 307 M 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 1 Package 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:31:55.045 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 307 M 2023-08-25T09:31:55.046 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-08-25T09:32:03.789 INFO:teuthology.orchestra.run.smithi164.stdout:linux-firmware-20230814-139.el9.noarch.rpm 35 MB/s | 307 MB 00:08 2023-08-25T09:32:03.790 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-08-25T09:32:03.791 INFO:teuthology.orchestra.run.smithi164.stdout:Total 35 MB/s | 307 MB 00:08 2023-08-25T09:32:03.791 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-08-25T09:32:03.877 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-08-25T09:32:03.877 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-08-25T09:32:05.794 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-08-25T09:32:05.794 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-08-25T09:32:11.784 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-08-25T09:32:12.666 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : linux-firmware-20230814-139.el9.noarch 1/2 2023-08-25T09:32:12.756 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : linux-firmware-20221214-129.el9.noarch 2/2 2023-08-25T09:32:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: linux-firmware-20221214-129.el9.noarch 2/2 2023-08-25T09:32:14.242 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 1/2 2023-08-25T09:32:14.637 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : linux-firmware-20221214-129.el9.noarch 2/2 2023-08-25T09:32:14.638 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:32:14.638 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2023-08-25T09:32:14.638 INFO:teuthology.orchestra.run.smithi164.stdout: linux-firmware-20230814-139.el9.noarch 2023-08-25T09:32:14.638 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:32:14.638 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-08-25T09:32:14.967 INFO:teuthology.task.kernel:Downloading kernel a6a69670761025e039dc5ae86619b0cf04c46dc2 on mon.b... 2023-08-25T09:32:14.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:32:15.112 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/a6a69670761025e039dc5ae86619b0cf04c46dc2/centos/9/flavors/default/x86_64/ 2023-08-25T09:32:15.112 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc7_ga6a696707610-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/a6a69670761025e039dc5ae86619b0cf04c46dc2/centos/9/flavors/default/x86_64/ --input-file=- 2023-08-25T09:32:15.115 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi164.front.sepia.ceph.com to complete... 2023-08-25T09:32:16.636 INFO:teuthology.orchestra.run.smithi071.stdout:linux-firmware-20230814-139.el9.noarch.rpm 13 MB/s | 307 MB 00:23 2023-08-25T09:32:16.637 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-08-25T09:32:16.638 INFO:teuthology.orchestra.run.smithi071.stdout:Total 13 MB/s | 307 MB 00:23 2023-08-25T09:32:16.638 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-08-25T09:32:16.726 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-08-25T09:32:16.727 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-08-25T09:32:18.658 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-08-25T09:32:18.658 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-08-25T09:32:25.009 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-08-25T09:32:25.788 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : linux-firmware-20230814-139.el9.noarch 1/2 2023-08-25T09:32:25.874 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : linux-firmware-20221214-129.el9.noarch 2/2 2023-08-25T09:32:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25 09:32:26 URL:https://4.chacra.ceph.com/r/kernel/testing/a6a69670761025e039dc5ae86619b0cf04c46dc2/centos/9/flavors/default/x86_64/kernel-6.5.0_rc7_ga6a696707610-1.x86_64.rpm [319273036/319273036] -> "/tmp/kernel.x86_64.rpm" [1] 2023-08-25T09:32:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:FINISHED --2023-08-25 09:32:26-- 2023-08-25T09:32:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:Total wall clock time: 11s 2023-08-25T09:32:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:Downloaded: 1 files, 304M in 11s (28.1 MB/s) 2023-08-25T09:32:26.265 INFO:teuthology.task.kernel:Installing kernel a6a69670761025e039dc5ae86619b0cf04c46dc2 on mon.b... 2023-08-25T09:32:26.265 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-08-25T09:32:26.940 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: linux-firmware-20221214-129.el9.noarch 2/2 2023-08-25T09:32:26.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 1/2 2023-08-25T09:32:27.371 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : linux-firmware-20221214-129.el9.noarch 2/2 2023-08-25T09:32:27.371 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:32:27.371 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2023-08-25T09:32:27.372 INFO:teuthology.orchestra.run.smithi071.stdout: linux-firmware-20230814-139.el9.noarch 2023-08-25T09:32:27.373 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:32:27.373 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-08-25T09:32:27.715 INFO:teuthology.task.kernel:Downloading kernel a6a69670761025e039dc5ae86619b0cf04c46dc2 on mon.a... 2023-08-25T09:32:27.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a6a69670761025e039dc5ae86619b0cf04c46dc2 2023-08-25T09:32:28.071 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/a6a69670761025e039dc5ae86619b0cf04c46dc2/centos/9/flavors/default/x86_64/ 2023-08-25T09:32:28.071 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc7_ga6a696707610-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/a6a69670761025e039dc5ae86619b0cf04c46dc2/centos/9/flavors/default/x86_64/ --input-file=- 2023-08-25T09:32:28.074 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi071.front.sepia.ceph.com to complete... 2023-08-25T09:32:28.107 INFO:teuthology.orchestra.run.smithi164.stdout:Verifying... ######################################## 2023-08-25T09:32:28.146 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing... ######################################## 2023-08-25T09:32:28.196 INFO:teuthology.orchestra.run.smithi164.stdout:Updating / installing... 2023-08-25T09:32:35.324 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-6.5.0_rc7_ga6a696707610-1 ######################################## 2023-08-25T09:32:38.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25 09:32:38 URL:https://4.chacra.ceph.com/r/kernel/testing/a6a69670761025e039dc5ae86619b0cf04c46dc2/centos/9/flavors/default/x86_64/kernel-6.5.0_rc7_ga6a696707610-1.x86_64.rpm [319273036/319273036] -> "/tmp/kernel.x86_64.rpm" [1] 2023-08-25T09:32:38.929 INFO:teuthology.orchestra.run.smithi071.stderr:FINISHED --2023-08-25 09:32:38-- 2023-08-25T09:32:38.929 INFO:teuthology.orchestra.run.smithi071.stderr:Total wall clock time: 11s 2023-08-25T09:32:38.929 INFO:teuthology.orchestra.run.smithi071.stderr:Downloaded: 1 files, 304M in 11s (28.9 MB/s) 2023-08-25T09:32:38.930 INFO:teuthology.task.kernel:Installing kernel a6a69670761025e039dc5ae86619b0cf04c46dc2 on mon.a... 2023-08-25T09:32:38.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-08-25T09:32:40.768 INFO:teuthology.orchestra.run.smithi071.stdout:Verifying... ######################################## 2023-08-25T09:32:40.816 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing... ######################################## 2023-08-25T09:32:40.865 INFO:teuthology.orchestra.run.smithi071.stdout:Updating / installing... 2023-08-25T09:32:47.899 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-6.5.0_rc7_ga6a696707610-1 ######################################## 2023-08-25T09:33:15.098 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-08-25T09:33:15.099 DEBUG:teuthology.orchestra.run.smithi164:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-08-25T09:33:15.184 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/System.map-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/config-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/vmlinuz-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel 2023-08-25T09:33:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch 2023-08-25T09:33:15.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86 2023-08-25T09:33:15.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto 2023-08-25T09:33:15.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-08-25T09:33:15.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-08-25T09:33:15.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-08-25T09:33:15.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-08-25T09:33:15.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/crc32c-intel.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-08-25T09:33:15.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-08-25T09:33:15.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/serpent-avx2.ko 2023-08-25T09:33:15.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-08-25T09:33:15.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-08-25T09:33:15.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-08-25T09:33:15.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel/cpu 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel/cpu/mce 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-08-25T09:33:15.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm 2023-08-25T09:33:15.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm/kvm-amd.ko 2023-08-25T09:33:15.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm/kvm-intel.ko 2023-08-25T09:33:15.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm/kvm.ko 2023-08-25T09:33:15.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto 2023-08-25T09:33:15.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/ansi_cprng.ko 2023-08-25T09:33:15.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/anubis.ko 2023-08-25T09:33:15.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/arc4.ko 2023-08-25T09:33:15.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx 2023-08-25T09:33:15.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_memcpy.ko 2023-08-25T09:33:15.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_pq.ko 2023-08-25T09:33:15.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_raid6_recov.ko 2023-08-25T09:33:15.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_tx.ko 2023-08-25T09:33:15.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_xor.ko 2023-08-25T09:33:15.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/raid6test.ko 2023-08-25T09:33:15.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/authenc.ko 2023-08-25T09:33:15.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/authencesn.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/blake2b_generic.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/blowfish_common.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/blowfish_generic.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/camellia_generic.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cast5_generic.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cast6_generic.ko 2023-08-25T09:33:15.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cast_common.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/ccm.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cmac.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/crc32_generic.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/crypto_engine.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/crypto_user.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cts.ko 2023-08-25T09:33:15.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/des_generic.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/dh_generic.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/echainiv.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/essiv.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/fcrypt.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/khazad.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/lz4.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/lz4hc.ko 2023-08-25T09:33:15.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/md4.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/michael_mic.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/pcbc.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/pcrypt.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/rmd160.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/seed.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/serpent_generic.ko 2023-08-25T09:33:15.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/tcrypt.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/tea.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/twofish_common.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/twofish_generic.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/vmac.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/wp512.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/xcbc.ko 2023-08-25T09:33:15.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/xor.ko 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/xxhash_generic.ko 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/acpi_ipmi.ko 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/acpi_pad.ko 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/custom_method.ko 2023-08-25T09:33:15.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/ec_sys.ko 2023-08-25T09:33:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/platform_profile.ko 2023-08-25T09:33:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/sbs.ko 2023-08-25T09:33:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/sbshc.ko 2023-08-25T09:33:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/video.ko 2023-08-25T09:33:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata 2023-08-25T09:33:15.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/acard-ahci.ko 2023-08-25T09:33:15.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/ahci_platform.ko 2023-08-25T09:33:15.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/ata_generic.ko 2023-08-25T09:33:15.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/libahci_platform.ko 2023-08-25T09:33:15.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_acpi.ko 2023-08-25T09:33:15.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ali.ko 2023-08-25T09:33:15.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_amd.ko 2023-08-25T09:33:15.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_artop.ko 2023-08-25T09:33:15.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_atiixp.ko 2023-08-25T09:33:15.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_atp867x.ko 2023-08-25T09:33:15.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_cmd640.ko 2023-08-25T09:33:15.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_cmd64x.ko 2023-08-25T09:33:15.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_cypress.ko 2023-08-25T09:33:15.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_efar.ko 2023-08-25T09:33:15.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt366.ko 2023-08-25T09:33:15.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt37x.ko 2023-08-25T09:33:15.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt3x2n.ko 2023-08-25T09:33:15.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt3x3.ko 2023-08-25T09:33:15.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_it8213.ko 2023-08-25T09:33:15.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_it821x.ko 2023-08-25T09:33:15.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_jmicron.ko 2023-08-25T09:33:15.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_marvell.ko 2023-08-25T09:33:15.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_mpiix.ko 2023-08-25T09:33:15.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_netcell.ko 2023-08-25T09:33:15.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ninja32.ko 2023-08-25T09:33:15.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ns87410.ko 2023-08-25T09:33:15.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ns87415.ko 2023-08-25T09:33:15.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_oldpiix.ko 2023-08-25T09:33:15.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_opti.ko 2023-08-25T09:33:15.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_optidma.ko 2023-08-25T09:33:15.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_pcmcia.ko 2023-08-25T09:33:15.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_pdc2027x.ko 2023-08-25T09:33:15.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-08-25T09:33:15.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_piccolo.ko 2023-08-25T09:33:15.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_rdc.ko 2023-08-25T09:33:15.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sch.ko 2023-08-25T09:33:15.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_serverworks.ko 2023-08-25T09:33:15.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sil680.ko 2023-08-25T09:33:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sis.ko 2023-08-25T09:33:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sl82c105.ko 2023-08-25T09:33:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_triflex.ko 2023-08-25T09:33:15.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_via.ko 2023-08-25T09:33:15.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pdc_adma.ko 2023-08-25T09:33:15.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_inic162x.ko 2023-08-25T09:33:15.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_mv.ko 2023-08-25T09:33:15.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_nv.ko 2023-08-25T09:33:15.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_promise.ko 2023-08-25T09:33:15.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_qstor.ko 2023-08-25T09:33:15.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sil.ko 2023-08-25T09:33:15.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sil24.ko 2023-08-25T09:33:15.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sis.ko 2023-08-25T09:33:15.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_svw.ko 2023-08-25T09:33:15.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sx4.ko 2023-08-25T09:33:15.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_uli.ko 2023-08-25T09:33:15.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_via.ko 2023-08-25T09:33:15.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_vsc.ko 2023-08-25T09:33:15.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/auxdisplay 2023-08-25T09:33:15.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/auxdisplay/ks0108.ko 2023-08-25T09:33:15.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base 2023-08-25T09:33:15.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base/regmap 2023-08-25T09:33:15.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base/regmap/regmap-i2c.ko 2023-08-25T09:33:15.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base/regmap/regmap-mmio.ko 2023-08-25T09:33:15.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/bcma 2023-08-25T09:33:15.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/bcma/bcma.ko 2023-08-25T09:33:15.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block 2023-08-25T09:33:15.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/aoe 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/aoe/aoe.ko 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/brd.ko 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/drbd 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/drbd/drbd.ko 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/floppy.ko 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/loop.ko 2023-08-25T09:33:15.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/mtip32xx 2023-08-25T09:33:15.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-08-25T09:33:15.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/nbd.ko 2023-08-25T09:33:15.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/null_blk 2023-08-25T09:33:15.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/null_blk/null_blk.ko 2023-08-25T09:33:15.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/pktcdvd.ko 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/rbd.ko 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/virtio_blk.ko 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/xen-blkback 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/xen-blkfront.ko 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/zram 2023-08-25T09:33:15.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/zram/zram.ko 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hangcheck-timer.ko 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/amd-rng.ko 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/intel-rng.ko 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-08-25T09:33:15.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/via-rng.ko 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/virtio-rng.ko 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_si.ko 2023-08-25T09:33:15.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/lp.ko 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/mwave 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/mwave/mwave.ko 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ppdev.ko 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tlclk.ko 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm 2023-08-25T09:33:15.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_atmel.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_infineon.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_nsc.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_tis.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/uv_mmtimer.ko 2023-08-25T09:33:15.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/virtio_console.ko 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/p4-clockmod.ko 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/powernow-k8.ko 2023-08-25T09:33:15.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/speedstep-lib.ko 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/ccp 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/ccp/ccp.ko 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_common 2023-08-25T09:33:15.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/padlock-aes.ko 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/padlock-sha.ko 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/virtio 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-08-25T09:33:15.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dca 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dca/dca.ko 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/devfreq 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/dw 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/dw/dw_dmac.ko 2023-08-25T09:33:15.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/ioat 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/ioat/ioatdma.ko 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/amd64_edac.ko 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/e752x_edac.ko 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/edac_mce_amd.ko 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i3000_edac.ko 2023-08-25T09:33:15.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i3200_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i5100_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i5400_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i7300_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i7core_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i82975x_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/ie31200_edac.ko 2023-08-25T09:33:15.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/sb_edac.ko 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/x38_edac.ko 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/firmware 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/firmware/edd.ko 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/firmware/iscsi_ibft.ko 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/gpio 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/gpio/gpio-ich.ko 2023-08-25T09:33:15.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/gpio/gpio-viperboard.ko 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-appleir.ko 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-aureal.ko 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-axff.ko 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-dr.ko 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-elecom.ko 2023-08-25T09:33:15.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-elo.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-emsff.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-gaff.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-gt683r.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-gyration.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-holtek-kbd.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-holtek-mouse.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-holtekff.ko 2023-08-25T09:33:15.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-hyperv.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-icade.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-keytouch.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-kye.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-lcpower.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-led.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-lenovo.ko 2023-08-25T09:33:15.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-logitech-dj.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-multitouch.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-ortek.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-penmount.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-petalynx.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-picolcd.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-pl.ko 2023-08-25T09:33:15.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-plantronics.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-primax.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-rmi.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-arvo.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-common.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-isku.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-kone.ko 2023-08-25T09:33:15.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-konepure.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-lua.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-pyra.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-ryos.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-savu.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat.ko 2023-08-25T09:33:15.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-saitek.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-samsung.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sensor-hub.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sjoy.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sony.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-speedlink.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-steelseries.ko 2023-08-25T09:33:15.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sunplus.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-thrustmaster.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-tivo.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-tmff.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-topseed.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-twinhan.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-uclogic.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-waltop.ko 2023-08-25T09:33:15.238 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-wiimote.ko 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-xinmo.ko 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-zpff.ko 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-zydacron.ko 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/uhid.ko 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/wacom.ko 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv 2023-08-25T09:33:15.239 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv/hv_balloon.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv/hv_utils.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv/hv_vmbus.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/abituguru.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/abituguru3.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/acpi_power_meter.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ad7414.ko 2023-08-25T09:33:15.240 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ad7418.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adc128d818.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1025.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1026.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1029.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1031.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm9240.ko 2023-08-25T09:33:15.241 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ads7828.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7410.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7411.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7462.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7470.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7475.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7x10.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/amc6821.ko 2023-08-25T09:33:15.242 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/applesmc.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/asb100.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/asc7621.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/asus_atk0110.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/atxp1.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/coretemp.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-08-25T09:33:15.243 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/dme1737.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ds1621.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ds620.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/emc1403.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/emc6w201.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/f71805f.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/f71882fg.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/f75375s.ko 2023-08-25T09:33:15.244 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/fam15h_power.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/fschmd.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/g760a.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/g762.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/gl518sm.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/gl520sm.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/hwmon-vid.ko 2023-08-25T09:33:15.245 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/i5500_temp.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/i5k_amb.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ibmaem.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ibmpex.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ina209.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ina2xx.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/it87.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/k10temp.ko 2023-08-25T09:33:15.246 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/k8temp.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lineage-pem.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm63.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm73.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm75.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm77.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm78.ko 2023-08-25T09:33:15.247 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm80.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm83.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm85.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm87.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm90.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm92.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm93.ko 2023-08-25T09:33:15.248 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm95234.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm95241.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm95245.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc2945.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4151.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4215.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4222.ko 2023-08-25T09:33:15.249 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4245.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4260.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4261.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max16065.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max1619.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max1668.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max197.ko 2023-08-25T09:33:15.250 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max6639.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max6650.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max6697.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/mcp3021.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct6683.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct6775-core.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct6775.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct7802.ko 2023-08-25T09:33:15.251 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pc87360.ko 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pc87427.ko 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pcf8591.ko 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-08-25T09:33:15.252 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/max16064.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/max34440.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/max8688.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-08-25T09:33:15.253 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/powr1220.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sch5627.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sch5636.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sch56xx-common.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sht15.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sht21.ko 2023-08-25T09:33:15.254 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/shtc1.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sis5595.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/smsc47b397.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/smsc47m1.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/smsc47m192.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/thmc50.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp102.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp103.ko 2023-08-25T09:33:15.255 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp401.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp421.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/via-cputemp.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/via686a.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/vt1211.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/vt8231.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83627ehf.ko 2023-08-25T09:33:15.256 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83627hf.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83781d.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83791d.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83792d.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83793.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83795.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83l785ts.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83l786ng.ko 2023-08-25T09:33:15.257 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/algos 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-08-25T09:33:15.258 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-i801.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-isch.ko 2023-08-25T09:33:15.259 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-parport.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-08-25T09:33:15.260 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-via.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-dev.ko 2023-08-25T09:33:15.261 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-mux.ko 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-smbus.ko 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-stub.ko 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_cm.ko 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_core.ko 2023-08-25T09:33:15.262 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_umad.ko 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/iw_cm.ko 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/rdma_cm.ko 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/cxgb4 2023-08-25T09:33:15.263 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-08-25T09:33:15.264 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx4 2023-08-25T09:33:15.264 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-08-25T09:33:15.264 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx5 2023-08-25T09:33:15.264 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-08-25T09:33:15.264 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mthca 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/ipoib 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/iser 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-08-25T09:33:15.265 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/isert 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srp 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srpt 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input 2023-08-25T09:33:15.266 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/fm801-gp.ko 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/gameport.ko 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/lightning.ko 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/ns558.ko 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/joydev.ko 2023-08-25T09:33:15.267 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/keyboard 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/keyboard/gpio_keys.ko 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/appletouch.ko 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/bcm5974.ko 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/cyapatp.ko 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/elan_i2c.ko 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/sermouse.ko 2023-08-25T09:33:15.268 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/synaptics_usb.ko 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/vsxxxaa.ko 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/rmi4 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/rmi4/rmi_core.ko 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/altera_ps2.ko 2023-08-25T09:33:15.269 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/arc_ps2.ko 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/serio_raw.ko 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/sparse-keymap.ko 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/iommu 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/iommu/amd 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/iommu/amd/iommu_v2.ko 2023-08-25T09:33:15.270 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-blinkm.ko 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-lm3530.ko 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-lp3944.ko 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-ss4200.ko 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-08-25T09:33:15.271 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md 2023-08-25T09:33:15.272 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/bcache 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/bcache/bcache.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-bio-prison.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-cache-smq.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-cache.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-crypt.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-delay.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-flakey.ko 2023-08-25T09:33:15.273 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-log-userspace.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-multipath.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-queue-length.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-raid.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-round-robin.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-service-time.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-switch.ko 2023-08-25T09:33:15.274 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-thin-pool.ko 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-verity.ko 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/faulty.ko 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/linear.ko 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/multipath.ko 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/persistent-data 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-08-25T09:33:15.275 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid0.ko 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid1.ko 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid10.ko 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid456.ko 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptbase.ko 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptctl.ko 2023-08-25T09:33:15.276 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptfc.ko 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptlan.ko 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptsas.ko 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptscsih.ko 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptspi.ko 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/lpc_ich.ko 2023-08-25T09:33:15.277 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/lpc_sch.ko 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/mfd-core.ko 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/sm501.ko 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/viperboard.ko 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/vx855.ko 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/wl1273-core.ko 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/altera-stapl 2023-08-25T09:33:15.278 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/apds9802als.ko 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/apds990x.ko 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/bh1770glc.ko 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/cb710 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/cb710/cb710.ko 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom 2023-08-25T09:33:15.279 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/at24.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/eeprom.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/max6875.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/enclosure.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/hpilo.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/isl29003.ko 2023-08-25T09:33:15.280 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/isl29020.ko 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/lis3lv02d 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei/mei-me.ko 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei/mei-txe.ko 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei/mei.ko 2023-08-25T09:33:15.281 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-gru 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-gru/gru.ko 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp/xp.ko 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp/xpc.ko 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/tifm_7xx1.ko 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/tifm_core.ko 2023-08-25T09:33:15.282 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/tsl2550.ko 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/vmw_balloon.ko 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/vmw_vmci 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/bonding 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/bonding/bonding.ko 2023-08-25T09:33:15.283 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/b53 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/b53/b53_common.ko 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/bcm-sf2.ko 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/mv88e6060.ko 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/mv88e6xxx 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-08-25T09:33:15.284 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dummy.ko 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/eql.ko 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-08-25T09:33:15.285 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/8390.ko 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/adaptec 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/agere 2023-08-25T09:33:15.286 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/agere/et131x.ko 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/alteon 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/altera 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-08-25T09:33:15.287 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/alx 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1c 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1e 2023-08-25T09:33:15.288 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atlx 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-08-25T09:33:15.289 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/genet 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/brocade 2023-08-25T09:33:15.290 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/brocade/bna 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-08-25T09:33:15.291 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/cisco 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/cisco/enic 2023-08-25T09:33:15.292 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-08-25T09:33:15.293 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dlink 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dnet.ko 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/emulex 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/emulex/benet 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-08-25T09:33:15.294 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/ethoc.ko 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/fealnx.ko 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e100.ko 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000e 2023-08-25T09:33:15.295 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/fm10k 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/i40e 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/iavf 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-08-25T09:33:15.296 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igb 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igbvf 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbe 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbevf 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-08-25T09:33:15.297 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/jme.ko 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell/skge.ko 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx4 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-08-25T09:33:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx5 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/micrel 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/myricom 2023-08-25T09:33:15.299 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/myricom/myri10ge 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/natsemi 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/neterion 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-08-25T09:33:15.300 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/nvidia 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/packetengines 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/netxen 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-08-25T09:33:15.301 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rdc 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-08-25T09:33:15.302 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/atp.ko 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rocker 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sfc 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/silan 2023-08-25T09:33:15.303 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sis 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sis/sis190.ko 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sis/sis900.ko 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-08-25T09:33:15.304 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro/stmmac 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/cassini.ko 2023-08-25T09:33:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/niu.ko 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/sungem.ko 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/tehuti 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/ti 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/ti/tlan.ko 2023-08-25T09:33:15.306 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/via 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/wiznet 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/xircom 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-08-25T09:33:15.307 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/geneve.ko 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/hyperv 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ieee802154 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ieee802154/fakelb.ko 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ifb.ko 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ipvlan 2023-08-25T09:33:15.308 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ipvlan/ipvlan.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/macvlan.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/macvtap.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio/mdio-regmap.ko 2023-08-25T09:33:15.309 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mii.ko 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/net_failover.ko 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/netconsole.ko 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/nlmon.ko 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/pcs 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/pcs/pcs-lynx.ko 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy 2023-08-25T09:33:15.310 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/amd.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm7xxx.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm87xx.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/broadcom.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/cicada.ko 2023-08-25T09:33:15.311 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/davicom.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/dp83640.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/et1011c.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/icplus.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/lxt.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/marvell.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/micrel.ko 2023-08-25T09:33:15.312 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/national.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/phylink.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/qsemi.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/realtek.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/smsc.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/ste10Xp.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/vitesse.ko 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp 2023-08-25T09:33:15.313 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/bsd_comp.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_async.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_deflate.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_generic.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_mppe.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_synctty.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/pppoe.ko 2023-08-25T09:33:15.314 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/pppox.ko 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/pptp.ko 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/slip 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/slip/slhc.ko 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/slip/slip.ko 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/sungem_phy.ko 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/tap.ko 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team 2023-08-25T09:33:15.315 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_activebackup.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_broadcast.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_random.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/tun.ko 2023-08-25T09:33:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/veth.ko 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/virtio_net.ko 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vmxnet3 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vxlan 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vxlan/vxlan.ko 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/xen-netback 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/xen-netback/xen-netback.ko 2023-08-25T09:33:15.317 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/xen-netfront.ko 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ntb 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ntb/ntb.ko 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme/host 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme/host/nvme-core.ko 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme/host/nvme.ko 2023-08-25T09:33:15.318 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport.ko 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport_cs.ko 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport_pc.ko 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport_serial.ko 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/hotplug 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-08-25T09:33:15.319 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/pcie 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/pcie/aer_inject.ko 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/xen-pcifront.ko 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia/i82092.ko 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia/pd6729.ko 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia/yenta_socket.ko 2023-08-25T09:33:15.320 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pinctrl 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pinctrl/intel 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/acer-wmi.ko 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/acerhdf.ko 2023-08-25T09:33:15.321 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/apple-gmux.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/asus-laptop.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/asus-wmi.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/classmate-laptop.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-08-25T09:33:15.322 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/hdaps.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/intel 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/intel_ips.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/msi-wmi.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/mxm-wmi.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-08-25T09:33:15.323 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/samsung-laptop.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/samsung-q10.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/topstar-laptop.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/toshiba_haps.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/wmi-bmof.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/wmi.ko 2023-08-25T09:33:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/powercap 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/powercap/intel_rapl_common.ko 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/powercap/intel_rapl_msr.ko 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients/pps-gpio.ko 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients/pps-ldisc.ko 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients/pps_parport.ko 2023-08-25T09:33:15.325 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/pps_core.ko 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ptp 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ptp/ptp.ko 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ptp/ptp_kvm.ko 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-bq32k.ko 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-bq4802.ko 2023-08-25T09:33:15.326 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1286.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1307.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1374.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1511.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1553.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1672.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1742.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds2404.ko 2023-08-25T09:33:15.327 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds3232.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-em3027.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-fm3130.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-isl12022.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-isl1208.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-m41t80.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-m48t35.ko 2023-08-25T09:33:15.328 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-m48t59.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-max6900.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-msm6242.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf2127.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf85063.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf8523.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf8563.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf8583.ko 2023-08-25T09:33:15.329 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rp5c01.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rs5c372.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rx8025.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rx8581.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-x1205.ko 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi 2023-08-25T09:33:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/3w-9xxx.ko 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/3w-sas.ko 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/3w-xxxx.ko 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/BusLogic.ko 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/a100u2w.ko 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aacraid 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aacraid/aacraid.ko 2023-08-25T09:33:15.331 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/advansys.ko 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aic7xxx 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/am53c974.ko 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/arcmsr 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/atp870u.ko 2023-08-25T09:33:15.332 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/be2iscsi 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bfa 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bfa/bfa.ko 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2fc 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2i 2023-08-25T09:33:15.333 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ch.ko 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/csiostor 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/csiostor/csiostor.ko 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb3i 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb4i 2023-08-25T09:33:15.334 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/dc395x.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-08-25T09:33:15.335 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/dmx3191d.ko 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/esas2r 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/esas2r/esas2r.ko 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/esp_scsi.ko 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fcoe 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fcoe/fcoe.ko 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-08-25T09:33:15.336 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fnic 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fnic/fnic.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/hpsa.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/hptiop.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/hv_storvsc.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/initio.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ipr.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ips.ko 2023-08-25T09:33:15.337 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/isci 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/isci/isci.ko 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/iscsi_tcp.ko 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libfc 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libfc/libfc.ko 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libiscsi.ko 2023-08-25T09:33:15.338 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libiscsi_tcp.ko 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libsas 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libsas/libsas.ko 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/lpfc 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/lpfc/lpfc.ko 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid.ko 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-08-25T09:33:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mpt3sas 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mvsas 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mvsas/mvsas.ko 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mvumi.ko 2023-08-25T09:33:15.340 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/pm8001 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/pmcraid.ko 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla1280.ko 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla2xxx 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla4xxx 2023-08-25T09:33:15.341 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/raid_class.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_debug.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_fc.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_sas.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_spi.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_srp.ko 2023-08-25T09:33:15.342 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ses.ko 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/st.ko 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/stex.ko 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/sym53c8xx_2 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/virtio_scsi.ko 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/vmw_pvscsi.ko 2023-08-25T09:33:15.343 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/wd719x.ko 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ssb 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ssb/ssb.ko 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/iscsi 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/loopback 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/loopback/tcm_loop.ko 2023-08-25T09:33:15.344 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_file.ko 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_iblock.ko 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_mod.ko 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_pscsi.ko 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_user.ko 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/tcm_fc 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-08-25T09:33:15.345 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-08-25T09:33:15.346 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-08-25T09:33:15.347 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/ipwireless 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/n_gsm.ko 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/n_hdlc.ko 2023-08-25T09:33:15.348 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/nozomi.ko 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/8250 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/arc_uart.ko 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/jsm 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/jsm/jsm.ko 2023-08-25T09:33:15.349 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/synclink_gt.ko 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/core 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/core/ufshcd-core.ko 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/host 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio.ko 2023-08-25T09:33:15.350 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_aec.ko 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_cif.ko 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_pci_generic.ko 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_sercos3.ko 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/cdc-acm.ko 2023-08-25T09:33:15.351 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/cdc-wdm.ko 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/usblp.ko 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/usbtmc.ko 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/host 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/host/sl811-hcd.ko 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/image 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/image/mdc800.ko 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/image/microtek.ko 2023-08-25T09:33:15.352 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/adutux.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/appledisplay.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/emi26.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/emi62.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/ezusb.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/idmouse.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/iowarrior.ko 2023-08-25T09:33:15.353 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/isight_firmware.ko 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/ldusb.ko 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/legousbtower.ko 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/sisusbvga 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/trancevibrator.ko 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/usb3503.ko 2023-08-25T09:33:15.354 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/usblcd.ko 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/usbsevseg.ko 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/uss720.ko 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/yurex.ko 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/phy 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/phy/phy-generic.ko 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/aircable.ko 2023-08-25T09:33:15.355 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ark3116.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/belkin_sa.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ch341.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/cp210x.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/cyberjack.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/cypress_m8.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/digi_acceleport.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/empeg.ko 2023-08-25T09:33:15.356 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ftdi_sio.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/garmin_gps.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/io_edgeport.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/io_ti.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ipaq.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ipw.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ir-usb.ko 2023-08-25T09:33:15.357 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/keyspan.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/keyspan_pda.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/kl5kusb105.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/kobil_sct.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/mct_u232.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/mos7720.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/mos7840.ko 2023-08-25T09:33:15.358 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/navman.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/omninet.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/opticon.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/option.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/oti6858.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/pl2303.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/qcaux.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/qcserial.ko 2023-08-25T09:33:15.359 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/quatech2.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/safe_serial.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/sierra.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/spcp8x5.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ssu100.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/symbolserial.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-08-25T09:33:15.360 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/usb_debug.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/usb_wwan.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/visor.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/whiteheat.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/xsens_mt.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/uas.ko 2023-08-25T09:33:15.361 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-alauda.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-cypress.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-datafab.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-freecom.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-isd200.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-08-25T09:33:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-karma.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-onetouch.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-realtek.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-sddr09.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-sddr55.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-usbat.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/usb-storage.ko 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip 2023-08-25T09:33:15.363 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip/usbip-core.ko 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip/usbip-host.ko 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/pci 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/pci/vfio-pci.ko 2023-08-25T09:33:15.364 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/vfio.ko 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost.ko 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost_iotlb.ko 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost_net.ko 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost_scsi.ko 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video 2023-08-25T09:33:15.365 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight/apple_bl.ko 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight/lcd.ko 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight/platform_lcd.ko 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio.ko 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_balloon.ko 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_mmio.ko 2023-08-25T09:33:15.366 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_pci.ko 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_ring.ko 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/masters 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/masters/ds2482.ko 2023-08-25T09:33:15.367 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/masters/ds2490.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-08-25T09:33:15.368 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_smem.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_therm.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/wire.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/alim1535_wdt.ko 2023-08-25T09:33:15.369 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/alim7101_wdt.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/f71808e_wdt.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/hpwdt.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/i6300esb.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/iTCO_wdt.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/ib700wdt.ko 2023-08-25T09:33:15.370 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/ibmasr.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/it8712f_wdt.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/it87_wdt.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/machzwd.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/nv_tco.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/pcwd_pci.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/pcwd_usb.ko 2023-08-25T09:33:15.371 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/sch311x_wdt.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/softdog.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/sp5100_tco.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/via_wdt.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/w83877f_wdt.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/w83977f_wdt.ko 2023-08-25T09:33:15.372 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/wdt_pci.ko 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/xen_wdt.ko 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-acpi-processor.ko 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-evtchn.ko 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-gntalloc.ko 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-gntdev.ko 2023-08-25T09:33:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-pciback 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-privcmd.ko 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xenfs 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xenfs/xenfs.ko 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/9p 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/9p/9p.ko 2023-08-25T09:33:15.374 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/affs 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/affs/affs.ko 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/befs 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/befs/befs.ko 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/binfmt_misc.ko 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/btrfs 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/btrfs/btrfs.ko 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cachefiles 2023-08-25T09:33:15.375 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cachefiles/cachefiles.ko 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ceph 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ceph/ceph.ko 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/coda 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/coda/coda.ko 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cramfs 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cramfs/cramfs.ko 2023-08-25T09:33:15.376 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/dlm 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/dlm/dlm.ko 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ecryptfs 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ecryptfs/ecryptfs.ko 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/f2fs 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/f2fs/f2fs.ko 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat/fat.ko 2023-08-25T09:33:15.377 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat/msdos.ko 2023-08-25T09:33:15.378 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat/vfat.ko 2023-08-25T09:33:15.378 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fscache 2023-08-25T09:33:15.379 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fscache/fscache.ko 2023-08-25T09:33:15.379 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fuse 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fuse/cuse.ko 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fuse/fuse.ko 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/gfs2 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/gfs2/gfs2.ko 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfs 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfs/hfs.ko 2023-08-25T09:33:15.380 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfsplus 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfsplus/hfsplus.ko 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/isofs 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/isofs/isofs.ko 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/jfs 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/jfs/jfs.ko 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/lockd 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/lockd/lockd.ko 2023-08-25T09:33:15.381 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/minix 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/minix/minix.ko 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/netfs 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/netfs/netfs.ko 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/blocklayout 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/filelayout 2023-08-25T09:33:15.382 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/flexfilelayout 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/nfs.ko 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/nfsv3.ko 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/nfsv4.ko 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs_common 2023-08-25T09:33:15.383 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs_common/grace.ko 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs_common/nfs_acl.ko 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfsd 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfsd/nfsd.ko 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nilfs2 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nilfs2/nilfs2.ko 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-celtic.ko 2023-08-25T09:33:15.384 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-centeuro.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-croatian.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-cyrillic.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-gaelic.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-greek.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-iceland.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-inuit.ko 2023-08-25T09:33:15.385 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-roman.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-romanian.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-turkish.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp1250.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp1251.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp1255.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp737.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp775.ko 2023-08-25T09:33:15.386 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp850.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp852.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp855.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp857.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp860.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp861.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp862.ko 2023-08-25T09:33:15.387 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp863.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp864.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp865.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp866.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp869.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp874.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp932.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp936.ko 2023-08-25T09:33:15.388 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp949.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp950.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_euc-jp.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-1.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-13.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-14.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-15.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-2.ko 2023-08-25T09:33:15.389 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-3.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-4.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-5.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-6.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-7.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-9.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_koi8-r.ko 2023-08-25T09:33:15.390 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_koi8-ru.ko 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_koi8-u.ko 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_utf8.ko 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/cluster 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlm 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-08-25T09:33:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlmfs 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2.ko 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/overlayfs 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/overlayfs/overlay.ko 2023-08-25T09:33:15.392 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/pstore 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/pstore/ramoops.ko 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/reiserfs 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/reiserfs/reiserfs.ko 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/romfs 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/romfs/romfs.ko 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb 2023-08-25T09:33:15.393 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/client 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/client/cifs.ko 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/common 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/common/cifs_arc4.ko 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/common/cifs_md4.ko 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/squashfs 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/squashfs/squashfs.ko 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/sysv 2023-08-25T09:33:15.394 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/sysv/sysv.ko 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/udf 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/udf/udf.ko 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ufs 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ufs/ufs.ko 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/xfs 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/xfs/xfs.ko 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel 2023-08-25T09:33:15.395 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/rcu 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/rcu/rcutorture.ko 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/torture.ko 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/trace 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/asn1_encoder.ko 2023-08-25T09:33:15.396 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crc-itu-t.ko 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crc8.ko 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crypto 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crypto/libarc4.ko 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crypto/libdes.ko 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/libcrc32c.ko 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lru_cache.ko 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lz4 2023-08-25T09:33:15.397 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lz4/lz4_compress.ko 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lz4/lz4hc_compress.ko 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/math 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/math/cordic.ko 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/raid6 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/raid6/raid6_pq.ko 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/reed_solomon 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/reed_solomon/reed_solomon.ko 2023-08-25T09:33:15.398 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/ts_bm.ko 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/ts_fsm.ko 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/ts_kmp.ko 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/zstd 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/zstd/zstd_compress.ko 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/mm 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/mm/hwpoison-inject.ko 2023-08-25T09:33:15.399 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/6lowpan.ko 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_dest.ko 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_fragment.ko 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_hop.ko 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_ipv6.ko 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_mobility.ko 2023-08-25T09:33:15.400 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_routing.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_udp.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/garp.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/mrp.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/p8022.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/psnap.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/stp.ko 2023-08-25T09:33:15.401 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/8021q 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/8021q/8021q.ko 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet.ko 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet_fd.ko 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet_rdma.ko 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet_virtio.ko 2023-08-25T09:33:15.402 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/batman-adv 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/batman-adv/batman-adv.ko 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/br_netfilter.ko 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/bridge.ko 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_among.ko 2023-08-25T09:33:15.403 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_arp.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_ip.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_limit.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_log.ko 2023-08-25T09:33:15.404 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_mark.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_snat.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_stp.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-08-25T09:33:15.405 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtables.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ceph 2023-08-25T09:33:15.406 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ceph/libceph.ko 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/core 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/core/failover.ko 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/core/pktgen.ko 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp.ko 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp_diag.ko 2023-08-25T09:33:15.407 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp_ipv4.ko 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp_ipv6.ko 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dns_resolver 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dns_resolver/dns_resolver.ko 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/dsa_core.ko 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_brcm.ko 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_dsa.ko 2023-08-25T09:33:15.408 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_none.ko 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_trailer.ko 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/6lowpan 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/ieee802154.ko 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/ieee802154_socket.ko 2023-08-25T09:33:15.409 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ah4.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/esp4.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/fou.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/gre.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/inet_diag.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ip_gre.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ip_tunnel.ko 2023-08-25T09:33:15.410 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ip_vti.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ipcomp.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ipip.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/arp_tables.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-08-25T09:33:15.411 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_security.ko 2023-08-25T09:33:15.412 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-08-25T09:33:15.413 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_bic.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_dctcp.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_diag.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_highspeed.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_htcp.ko 2023-08-25T09:33:15.414 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_hybla.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_illinois.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_lp.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_scalable.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_vegas.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_veno.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_westwood.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_yeah.ko 2023-08-25T09:33:15.415 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tunnel4.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/udp_diag.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/udp_tunnel.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/xfrm4_tunnel.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ah6.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/esp6.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/fou6.ko 2023-08-25T09:33:15.416 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ip6_tunnel.ko 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ip6_vti.ko 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ipcomp6.ko 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-08-25T09:33:15.417 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-08-25T09:33:15.418 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-08-25T09:33:15.419 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/sit.ko 2023-08-25T09:33:15.420 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/tunnel6.ko 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/xfrm6_tunnel.ko 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/key 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/key/af_key.ko 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_core.ko 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_debugfs.ko 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_eth.ko 2023-08-25T09:33:15.421 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_ip.ko 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_ip6.ko 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_netlink.ko 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_ppp.ko 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/llc 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/llc/llc.ko 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mac802154 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mac802154/mac802154.ko 2023-08-25T09:33:15.422 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mpls 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mpls/mpls_gso.ko 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set.ko 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-08-25T09:33:15.423 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-08-25T09:33:15.424 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-08-25T09:33:15.425 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-08-25T09:33:15.426 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conncount.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-08-25T09:33:15.427 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_h323.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_irc.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_sane.ko 2023-08-25T09:33:15.428 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_sip.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_dup_netdev.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_flow_table.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_flow_table_inet.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_log_syslog.ko 2023-08-25T09:33:15.429 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_amanda.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_ftp.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_irc.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_sip.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_tftp.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_synproxy_core.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_tables.ko 2023-08-25T09:33:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_acct.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_log.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_osf.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_queue.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_chain_nat.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_compat.ko 2023-08-25T09:33:15.431 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_ct.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_dup_netdev.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_fib.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_fwd_netdev.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_hash.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_limit.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_log.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_masq.ko 2023-08-25T09:33:15.432 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_nat.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_numgen.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_queue.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_quota.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_redir.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_reject.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_reject_inet.ko 2023-08-25T09:33:15.433 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_socket.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_tunnel.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_AUDIT.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CHECKSUM.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CLASSIFY.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CT.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_DSCP.ko 2023-08-25T09:33:15.434 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_HL.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_HMARK.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_IDLETIMER.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_LED.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_LOG.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_MASQUERADE.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_NETMAP.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_NFLOG.ko 2023-08-25T09:33:15.435 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_NFQUEUE.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_RATEEST.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_REDIRECT.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_SECMARK.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TCPMSS.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TEE.ko 2023-08-25T09:33:15.436 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TPROXY.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TRACE.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_addrtype.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_bpf.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_cgroup.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_cluster.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_comment.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connbytes.ko 2023-08-25T09:33:15.437 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connlabel.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connlimit.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connmark.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_conntrack.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_cpu.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_dccp.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_devgroup.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_dscp.ko 2023-08-25T09:33:15.438 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_ecn.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_esp.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_hashlimit.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_helper.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_hl.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_ipcomp.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_iprange.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_ipvs.ko 2023-08-25T09:33:15.439 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_l2tp.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_length.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_limit.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_mac.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_mark.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_multiport.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_nat.ko 2023-08-25T09:33:15.440 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_nfacct.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_osf.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_owner.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_physdev.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_pkttype.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_policy.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_quota.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_rateest.ko 2023-08-25T09:33:15.441 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_realm.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_recent.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_sctp.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_set.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_socket.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_state.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_statistic.ko 2023-08-25T09:33:15.442 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_string.ko 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_tcpmss.ko 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_time.ko 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_u32.ko 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netlink 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netlink/netlink_diag.ko 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/nsh 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/nsh/nsh.ko 2023-08-25T09:33:15.443 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/openvswitch.ko 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/vport-geneve.ko 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/vport-gre.ko 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/vport-vxlan.ko 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/packet 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/packet/af_packet_diag.ko 2023-08-25T09:33:15.444 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds/rds.ko 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds/rds_rdma.ko 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds/rds_tcp.ko 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_csum.ko 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_gact.ko 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_ipt.ko 2023-08-25T09:33:15.445 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_mirred.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_nat.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_pedit.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_police.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_simple.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_skbedit.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_vlan.ko 2023-08-25T09:33:15.446 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_basic.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_bpf.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_flow.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_fw.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_route.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_u32.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_cmp.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_ipset.ko 2023-08-25T09:33:15.447 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_meta.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_nbyte.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_text.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_u32.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_choke.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_codel.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_drr.ko 2023-08-25T09:33:15.448 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_fq.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_gred.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_hfsc.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_hhf.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_htb.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_ingress.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_mqprio.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_mqprio_lib.ko 2023-08-25T09:33:15.449 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_multiq.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_netem.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_pie.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_plug.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_prio.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_qfq.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_red.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_sfb.ko 2023-08-25T09:33:15.450 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_sfq.ko 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_tbf.ko 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_teql.ko 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sctp 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sctp/sctp.ko 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sctp/sctp_diag.ko 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc 2023-08-25T09:33:15.451 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/auth_gss 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/sunrpc.ko 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/xprtrdma 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/tipc 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/tipc/diag.ko 2023-08-25T09:33:15.452 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/tipc/tipc.ko 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/unix 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/unix/unix_diag.ko 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vsock.ko 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vsock_diag.ko 2023-08-25T09:33:15.453 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vsock_loopback.ko 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/xfrm 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/xfrm/xfrm_ipcomp.ko 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/encrypted-keys 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-08-25T09:33:15.454 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/trusted-keys 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/trusted-keys/trusted.ko 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/virt 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/virt/lib 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/virt/lib/irqbypass.ko 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.alias 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.alias.bin 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin 2023-08-25T09:33:15.455 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin.alias.bin 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin.bin 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin.modinfo 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.dep 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.dep.bin 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.devname 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.order 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.softdep 2023-08-25T09:33:15.456 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.symbols 2023-08-25T09:33:15.457 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.symbols.bin 2023-08-25T09:33:15.458 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.458 DEBUG:teuthology.orchestra.run.smithi164:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-08-25T09:33:15.480 INFO:teuthology.orchestra.run.smithi164.stdout:postinstall scriptlet (using /bin/sh): 2023-08-25T09:33:15.480 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc7-ga6a696707610 -a -r /boot/System.map-6.5.0-rc7-ga6a696707610 ]; then 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:cp /boot/vmlinuz-6.5.0-rc7-ga6a696707610 /boot/.vmlinuz-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:cp /boot/System.map-6.5.0-rc7-ga6a696707610 /boot/.System.map-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:rm -f /boot/vmlinuz-6.5.0-rc7-ga6a696707610 /boot/System.map-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:/sbin/installkernel 6.5.0-rc7-ga6a696707610 /boot/.vmlinuz-6.5.0-rc7-ga6a696707610-rpm /boot/.System.map-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:rm -f /boot/.vmlinuz-6.5.0-rc7-ga6a696707610-rpm /boot/.System.map-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2023-08-25T09:33:15.481 INFO:teuthology.orchestra.run.smithi164.stdout:preuninstall scriptlet (using /bin/sh): 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:new-kernel-pkg --remove 6.5.0-rc7-ga6a696707610 --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc7-ga6a696707610.img 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-install remove 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:postuninstall scriptlet (using /bin/sh): 2023-08-25T09:33:15.482 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /sbin/update-bootloader ]; then 2023-08-25T09:33:15.483 INFO:teuthology.orchestra.run.smithi164.stdout:/sbin/update-bootloader --remove 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.483 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2023-08-25T09:33:15.483 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2023-08-25T09:33:15.559 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 53.el9 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Mon 30 Jan 2023 10:38:40 AM UTC 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2023-08-25T09:33:15.560 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8551923 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Wed 18 Jan 2023 04:40:07 PM UTC, Key ID 05b555b38483c65d 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-53.el9.src.rpm 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Fri 13 Jan 2023 09:07:28 PM UTC 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2023-08-25T09:33:15.561 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2023-08-25T09:33:15.562 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2023-08-25T09:33:15.563 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:33:15.563 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2023-08-25T09:33:15.564 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-08-25T09:33:15.564 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:15.564 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-08-25T09:33:16.195 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2023-08-25T09:33:17.538 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-08-25T09:33:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:done 2023-08-25T09:33:17.594 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2023-08-25T09:33:17.646 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.KFXxZurbFt 2023-08-25T09:33:17.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KFXxZurbFt 2023-08-25T09:33:17.681 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.KFXxZurbFt 2023-08-25T09:33:17.826 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.KFXxZurbFt is 6KB 2023-08-25T09:33:17.873 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.KFXxZurbFt 2023-08-25T09:33:17.887 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2023-08-25T09:33:17.953 INFO:teuthology.orchestra.run.smithi164.stdout:e13cba1d25f14c51a1d04da8db823a4e-0-rescue.conf 2023-08-25T09:33:17.953 INFO:teuthology.orchestra.run.smithi164.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-239.el9.x86_64.conf 2023-08-25T09:33:17.954 INFO:teuthology.orchestra.run.smithi164.stdout:e13cba1d25f14c51a1d04da8db823a4e-6.5.0-rc7-ga6a696707610.conf 2023-08-25T09:33:17.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default e13cba1d25f14c51a1d04da8db823a4e-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:18.142 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2023-08-25T09:33:29.535 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-08-25T09:33:29.535 DEBUG:teuthology.orchestra.run.smithi071:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-08-25T09:33:29.602 INFO:teuthology.orchestra.run.smithi071.stdout:/boot/System.map-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/boot/config-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/boot/vmlinuz-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86 2023-08-25T09:33:29.603 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-08-25T09:33:29.604 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/crc32c-intel.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/serpent-avx2.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-08-25T09:33:29.605 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-08-25T09:33:29.606 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-08-25T09:33:29.606 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-08-25T09:33:29.606 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-08-25T09:33:29.606 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-08-25T09:33:29.606 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-08-25T09:33:29.606 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel/cpu 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel/cpu/mce 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm/kvm-amd.ko 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm/kvm-intel.ko 2023-08-25T09:33:29.607 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/arch/x86/kvm/kvm.ko 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/ansi_cprng.ko 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/anubis.ko 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/arc4.ko 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_memcpy.ko 2023-08-25T09:33:29.608 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_pq.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_raid6_recov.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_tx.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/async_xor.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/async_tx/raid6test.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/authenc.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/authencesn.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/blake2b_generic.ko 2023-08-25T09:33:29.609 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/blowfish_common.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/blowfish_generic.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/camellia_generic.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cast5_generic.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cast6_generic.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cast_common.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/ccm.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cmac.ko 2023-08-25T09:33:29.610 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/crc32_generic.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/crypto_engine.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/crypto_user.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/cts.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/des_generic.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/dh_generic.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/echainiv.ko 2023-08-25T09:33:29.611 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/essiv.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/fcrypt.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/khazad.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/lz4.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/lz4hc.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/md4.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/michael_mic.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/pcbc.ko 2023-08-25T09:33:29.612 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/pcrypt.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/rmd160.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/seed.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/serpent_generic.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/tcrypt.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/tea.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/twofish_common.ko 2023-08-25T09:33:29.613 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/twofish_generic.ko 2023-08-25T09:33:29.614 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/vmac.ko 2023-08-25T09:33:29.614 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/wp512.ko 2023-08-25T09:33:29.614 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/xcbc.ko 2023-08-25T09:33:29.614 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/xor.ko 2023-08-25T09:33:29.615 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/crypto/xxhash_generic.ko 2023-08-25T09:33:29.615 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers 2023-08-25T09:33:29.615 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi 2023-08-25T09:33:29.615 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/acpi_ipmi.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/acpi_pad.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/custom_method.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/ec_sys.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/platform_profile.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/sbs.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/sbshc.ko 2023-08-25T09:33:29.616 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/acpi/video.ko 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/acard-ahci.ko 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/ahci_platform.ko 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/ata_generic.ko 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/libahci_platform.ko 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_acpi.ko 2023-08-25T09:33:29.617 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ali.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_amd.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_artop.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_atiixp.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_atp867x.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_cmd640.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_cmd64x.ko 2023-08-25T09:33:29.618 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_cypress.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_efar.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt366.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt37x.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt3x2n.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_hpt3x3.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_it8213.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_it821x.ko 2023-08-25T09:33:29.619 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_jmicron.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_marvell.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_mpiix.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_netcell.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ninja32.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ns87410.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_ns87415.ko 2023-08-25T09:33:29.620 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_oldpiix.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_opti.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_optidma.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_pcmcia.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_pdc2027x.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_piccolo.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_rdc.ko 2023-08-25T09:33:29.621 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sch.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_serverworks.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sil680.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sis.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_sl82c105.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_triflex.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pata_via.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/pdc_adma.ko 2023-08-25T09:33:29.622 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_inic162x.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_mv.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_nv.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_promise.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_qstor.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sil.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sil24.ko 2023-08-25T09:33:29.623 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sis.ko 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_svw.ko 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_sx4.ko 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_uli.ko 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_via.ko 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ata/sata_vsc.ko 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/auxdisplay 2023-08-25T09:33:29.624 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/auxdisplay/ks0108.ko 2023-08-25T09:33:29.625 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base 2023-08-25T09:33:29.625 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base/regmap 2023-08-25T09:33:29.625 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base/regmap/regmap-i2c.ko 2023-08-25T09:33:29.625 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/base/regmap/regmap-mmio.ko 2023-08-25T09:33:29.625 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/bcma 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/bcma/bcma.ko 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/aoe 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/aoe/aoe.ko 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/brd.ko 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/drbd 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/drbd/drbd.ko 2023-08-25T09:33:29.626 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/floppy.ko 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/loop.ko 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/mtip32xx 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/nbd.ko 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/null_blk 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/null_blk/null_blk.ko 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/pktcdvd.ko 2023-08-25T09:33:29.627 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/rbd.ko 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/virtio_blk.ko 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/xen-blkback 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/xen-blkfront.ko 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/zram 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/block/zram/zram.ko 2023-08-25T09:33:29.628 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hangcheck-timer.ko 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/amd-rng.ko 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/intel-rng.ko 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/via-rng.ko 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/hw_random/virtio-rng.ko 2023-08-25T09:33:29.629 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi 2023-08-25T09:33:29.630 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-08-25T09:33:29.630 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-08-25T09:33:29.630 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-08-25T09:33:29.630 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_si.ko 2023-08-25T09:33:29.630 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-08-25T09:33:29.630 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-08-25T09:33:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/lp.ko 2023-08-25T09:33:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/mwave 2023-08-25T09:33:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/mwave/mwave.ko 2023-08-25T09:33:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/ppdev.ko 2023-08-25T09:33:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tlclk.ko 2023-08-25T09:33:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm.ko 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_atmel.ko 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_infineon.ko 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_nsc.ko 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_tis.ko 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-08-25T09:33:29.632 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/uv_mmtimer.ko 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/char/virtio_console.ko 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/p4-clockmod.ko 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-08-25T09:33:29.633 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/powernow-k8.ko 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/cpufreq/speedstep-lib.ko 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/ccp 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/ccp/ccp.ko 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat 2023-08-25T09:33:29.634 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_common 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/padlock-aes.ko 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/padlock-sha.ko 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/virtio 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-08-25T09:33:29.635 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dca 2023-08-25T09:33:29.636 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dca/dca.ko 2023-08-25T09:33:29.636 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/devfreq 2023-08-25T09:33:29.636 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-08-25T09:33:29.636 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma 2023-08-25T09:33:29.636 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/dw 2023-08-25T09:33:29.636 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/dw/dw_dmac.ko 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/ioat 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/dma/ioat/ioatdma.ko 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/amd64_edac.ko 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/e752x_edac.ko 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/edac_mce_amd.ko 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i3000_edac.ko 2023-08-25T09:33:29.637 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i3200_edac.ko 2023-08-25T09:33:29.638 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i5100_edac.ko 2023-08-25T09:33:29.638 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i5400_edac.ko 2023-08-25T09:33:29.638 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i7300_edac.ko 2023-08-25T09:33:29.638 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i7core_edac.ko 2023-08-25T09:33:29.638 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/i82975x_edac.ko 2023-08-25T09:33:29.638 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/ie31200_edac.ko 2023-08-25T09:33:29.639 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/sb_edac.ko 2023-08-25T09:33:29.639 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/edac/x38_edac.ko 2023-08-25T09:33:29.639 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/firmware 2023-08-25T09:33:29.639 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/firmware/edd.ko 2023-08-25T09:33:29.639 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/firmware/iscsi_ibft.ko 2023-08-25T09:33:29.640 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/gpio 2023-08-25T09:33:29.640 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/gpio/gpio-ich.ko 2023-08-25T09:33:29.640 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/gpio/gpio-viperboard.ko 2023-08-25T09:33:29.640 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid 2023-08-25T09:33:29.640 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-appleir.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-aureal.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-axff.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-dr.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-elecom.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-elo.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-emsff.ko 2023-08-25T09:33:29.641 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-gaff.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-gt683r.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-gyration.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-holtek-kbd.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-holtek-mouse.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-holtekff.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-hyperv.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-icade.ko 2023-08-25T09:33:29.642 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-keytouch.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-kye.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-lcpower.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-led.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-lenovo.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-logitech-dj.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-multitouch.ko 2023-08-25T09:33:29.643 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-ortek.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-penmount.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-petalynx.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-picolcd.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-pl.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-plantronics.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-primax.ko 2023-08-25T09:33:29.644 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-rmi.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-arvo.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-common.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-isku.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-kone.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-konepure.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-08-25T09:33:29.645 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-lua.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-pyra.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-ryos.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat-savu.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-roccat.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-saitek.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-samsung.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sensor-hub.ko 2023-08-25T09:33:29.646 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sjoy.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sony.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-speedlink.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-steelseries.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-sunplus.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-thrustmaster.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-tivo.ko 2023-08-25T09:33:29.647 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-tmff.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-topseed.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-twinhan.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-uclogic.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-waltop.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-wiimote.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-xinmo.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-zpff.ko 2023-08-25T09:33:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/hid-zydacron.ko 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/uhid.ko 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hid/wacom.ko 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv/hv_balloon.ko 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv/hv_utils.ko 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hv/hv_vmbus.ko 2023-08-25T09:33:29.649 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/abituguru.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/abituguru3.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/acpi_power_meter.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ad7414.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ad7418.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adc128d818.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1025.ko 2023-08-25T09:33:29.650 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1026.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1029.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm1031.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adm9240.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ads7828.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7410.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7411.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7462.ko 2023-08-25T09:33:29.651 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7470.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7475.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/adt7x10.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/amc6821.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/applesmc.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/asb100.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/asc7621.ko 2023-08-25T09:33:29.652 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/asus_atk0110.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/atxp1.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/coretemp.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/dme1737.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ds1621.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ds620.ko 2023-08-25T09:33:29.653 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/emc1403.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/emc6w201.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/f71805f.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/f71882fg.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/f75375s.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/fam15h_power.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/fschmd.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/g760a.ko 2023-08-25T09:33:29.654 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/g762.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/gl518sm.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/gl520sm.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/hwmon-vid.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/i5500_temp.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/i5k_amb.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ibmaem.ko 2023-08-25T09:33:29.655 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ibmpex.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ina209.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ina2xx.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/it87.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/k10temp.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/k8temp.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lineage-pem.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm63.ko 2023-08-25T09:33:29.656 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm73.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm75.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm77.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm78.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm80.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm83.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm85.ko 2023-08-25T09:33:29.657 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm87.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm90.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm92.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm93.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm95234.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm95241.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/lm95245.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc2945.ko 2023-08-25T09:33:29.658 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4151.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4215.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4222.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4245.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4260.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/ltc4261.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max16065.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max1619.ko 2023-08-25T09:33:29.659 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max1668.ko 2023-08-25T09:33:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max197.ko 2023-08-25T09:33:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max6639.ko 2023-08-25T09:33:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max6650.ko 2023-08-25T09:33:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/max6697.ko 2023-08-25T09:33:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/mcp3021.ko 2023-08-25T09:33:29.660 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct6683.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct6775-core.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct6775.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/nct7802.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pc87360.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pc87427.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pcf8591.ko 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus 2023-08-25T09:33:29.661 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/max16064.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/max34440.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/max8688.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-08-25T09:33:29.662 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/powr1220.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sch5627.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sch5636.ko 2023-08-25T09:33:29.663 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sch56xx-common.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sht15.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sht21.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/shtc1.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/sis5595.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/smsc47b397.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/smsc47m1.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/smsc47m192.ko 2023-08-25T09:33:29.664 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/thmc50.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp102.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp103.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp401.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/tmp421.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/via-cputemp.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/via686a.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/vt1211.ko 2023-08-25T09:33:29.665 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/vt8231.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83627ehf.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83627hf.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83781d.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83791d.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83792d.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83793.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83795.ko 2023-08-25T09:33:29.666 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83l785ts.ko 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/hwmon/w83l786ng.ko 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/algos 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-08-25T09:33:29.667 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-08-25T09:33:29.668 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-08-25T09:33:29.668 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-08-25T09:33:29.668 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-08-25T09:33:29.668 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-08-25T09:33:29.668 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-08-25T09:33:29.668 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-08-25T09:33:29.669 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-i801.ko 2023-08-25T09:33:29.669 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-isch.ko 2023-08-25T09:33:29.669 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-08-25T09:33:29.669 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-08-25T09:33:29.669 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-parport.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-08-25T09:33:29.670 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-via.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-dev.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-mux.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-smbus.ko 2023-08-25T09:33:29.671 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/i2c/i2c-stub.ko 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_cm.ko 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_core.ko 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_umad.ko 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/iw_cm.ko 2023-08-25T09:33:29.672 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/rdma_cm.ko 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/cxgb4 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx4 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-08-25T09:33:29.673 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx5 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mthca 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/ipoib 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/iser 2023-08-25T09:33:29.674 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/isert 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srp 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srpt 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input 2023-08-25T09:33:29.675 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/fm801-gp.ko 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/gameport.ko 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/lightning.ko 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/gameport/ns558.ko 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/joydev.ko 2023-08-25T09:33:29.676 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/keyboard 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/keyboard/gpio_keys.ko 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/appletouch.ko 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/bcm5974.ko 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/cyapatp.ko 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/elan_i2c.ko 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/sermouse.ko 2023-08-25T09:33:29.677 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/synaptics_usb.ko 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/mouse/vsxxxaa.ko 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/rmi4 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/rmi4/rmi_core.ko 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/altera_ps2.ko 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/arc_ps2.ko 2023-08-25T09:33:29.678 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/serio/serio_raw.ko 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/input/sparse-keymap.ko 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/iommu 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/iommu/amd 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/iommu/amd/iommu_v2.ko 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds 2023-08-25T09:33:29.679 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-blinkm.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-lm3530.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-lp3944.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/leds-ss4200.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-08-25T09:33:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/bcache 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/bcache/bcache.ko 2023-08-25T09:33:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-bio-prison.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-cache-smq.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-cache.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-crypt.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-delay.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-flakey.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-log-userspace.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-multipath.ko 2023-08-25T09:33:29.682 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-queue-length.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-raid.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-round-robin.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-service-time.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-switch.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-thin-pool.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/dm-verity.ko 2023-08-25T09:33:29.683 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/faulty.ko 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/linear.ko 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/multipath.ko 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/persistent-data 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid0.ko 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid1.ko 2023-08-25T09:33:29.684 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid10.ko 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/md/raid456.ko 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptbase.ko 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptctl.ko 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptfc.ko 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptlan.ko 2023-08-25T09:33:29.685 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptsas.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptscsih.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/message/fusion/mptspi.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/lpc_ich.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/lpc_sch.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/mfd-core.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/sm501.ko 2023-08-25T09:33:29.686 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/viperboard.ko 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/vx855.ko 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/mfd/wl1273-core.ko 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/altera-stapl 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/apds9802als.ko 2023-08-25T09:33:29.687 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/apds990x.ko 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/bh1770glc.ko 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/cb710 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/cb710/cb710.ko 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/at24.ko 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/eeprom.ko 2023-08-25T09:33:29.688 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/eeprom/max6875.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/enclosure.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/hpilo.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/isl29003.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/isl29020.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/lis3lv02d 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-08-25T09:33:29.689 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei/mei-me.ko 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei/mei-txe.ko 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/mei/mei.ko 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-gru 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-gru/gru.ko 2023-08-25T09:33:29.690 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp/xp.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp/xpc.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/tifm_7xx1.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/tifm_core.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/tsl2550.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/vmw_balloon.ko 2023-08-25T09:33:29.691 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/vmw_vmci 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/bonding 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/bonding/bonding.ko 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/b53 2023-08-25T09:33:29.692 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/b53/b53_common.ko 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/bcm-sf2.ko 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/mv88e6060.ko 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/mv88e6xxx 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/dummy.ko 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/eql.ko 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet 2023-08-25T09:33:29.693 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/8390.ko 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-08-25T09:33:29.694 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/adaptec 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/agere 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/agere/et131x.ko 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/alteon 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-08-25T09:33:29.695 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/altera 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros 2023-08-25T09:33:29.696 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/alx 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1c 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1e 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atlx 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-08-25T09:33:29.697 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-08-25T09:33:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/genet 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/brocade 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/brocade/bna 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb 2023-08-25T09:33:29.699 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-08-25T09:33:29.700 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/cisco 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/cisco/enic 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-08-25T09:33:29.701 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dlink 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-08-25T09:33:29.702 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/dnet.ko 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/emulex 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/emulex/benet 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/ethoc.ko 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/fealnx.ko 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel 2023-08-25T09:33:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e100.ko 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000e 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/fm10k 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-08-25T09:33:29.704 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/i40e 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/iavf 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igb 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igbvf 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-08-25T09:33:29.705 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbe 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbevf 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/jme.ko 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell/skge.ko 2023-08-25T09:33:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx4 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx5 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-08-25T09:33:29.707 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/micrel 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/myricom 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/myricom/myri10ge 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/natsemi 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-08-25T09:33:29.708 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/neterion 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/nvidia 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/packetengines 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-08-25T09:33:29.709 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/netxen 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rdc 2023-08-25T09:33:29.710 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/atp.ko 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rocker 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-08-25T09:33:29.711 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sfc 2023-08-25T09:33:29.712 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-08-25T09:33:29.712 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/silan 2023-08-25T09:33:29.712 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-08-25T09:33:29.712 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sis 2023-08-25T09:33:29.712 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sis/sis190.ko 2023-08-25T09:33:29.714 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sis/sis900.ko 2023-08-25T09:33:29.714 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc 2023-08-25T09:33:29.714 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-08-25T09:33:29.714 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-08-25T09:33:29.714 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro/stmmac 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/cassini.ko 2023-08-25T09:33:29.715 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/niu.ko 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/sungem.ko 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/tehuti 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/ti 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/ti/tlan.ko 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/via 2023-08-25T09:33:29.716 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/wiznet 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/xircom 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-08-25T09:33:29.717 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/geneve.ko 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/hyperv 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ieee802154 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ieee802154/fakelb.ko 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ifb.ko 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ipvlan 2023-08-25T09:33:29.718 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ipvlan/ipvlan.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/macvlan.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/macvtap.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mdio/mdio-regmap.ko 2023-08-25T09:33:29.719 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/mii.ko 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/net_failover.ko 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/netconsole.ko 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/nlmon.ko 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/pcs 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/pcs/pcs-lynx.ko 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-08-25T09:33:29.720 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/amd.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm7xxx.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/bcm87xx.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/broadcom.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/cicada.ko 2023-08-25T09:33:29.721 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/davicom.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/dp83640.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/et1011c.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/icplus.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/lxt.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/marvell.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/micrel.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/national.ko 2023-08-25T09:33:29.722 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/phylink.ko 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/qsemi.ko 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/realtek.ko 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/smsc.ko 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/ste10Xp.ko 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/phy/vitesse.ko 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp 2023-08-25T09:33:29.723 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/bsd_comp.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_async.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_deflate.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_generic.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_mppe.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/ppp_synctty.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/pppoe.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/pppox.ko 2023-08-25T09:33:29.724 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/ppp/pptp.ko 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/slip 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/slip/slhc.ko 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/slip/slip.ko 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/sungem_phy.ko 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/tap.ko 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team.ko 2023-08-25T09:33:29.725 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_activebackup.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_broadcast.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_random.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/tun.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/veth.ko 2023-08-25T09:33:29.726 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/virtio_net.ko 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vmxnet3 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vxlan 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/vxlan/vxlan.ko 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/xen-netback 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/xen-netback/xen-netback.ko 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/net/xen-netfront.ko 2023-08-25T09:33:29.727 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ntb 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ntb/ntb.ko 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme/host 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme/host/nvme-core.ko 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/nvme/host/nvme.ko 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport 2023-08-25T09:33:29.728 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport.ko 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport_cs.ko 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport_pc.ko 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/parport/parport_serial.ko 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/hotplug 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-08-25T09:33:29.729 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/pcie 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/pcie/aer_inject.ko 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pci/xen-pcifront.ko 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia/i82092.ko 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia/pd6729.ko 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pcmcia/yenta_socket.ko 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pinctrl 2023-08-25T09:33:29.730 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pinctrl/intel 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/acer-wmi.ko 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/acerhdf.ko 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/apple-gmux.ko 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/asus-laptop.ko 2023-08-25T09:33:29.731 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/asus-wmi.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/classmate-laptop.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-08-25T09:33:29.732 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/hdaps.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/intel 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/intel_ips.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/msi-wmi.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/mxm-wmi.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/samsung-laptop.ko 2023-08-25T09:33:29.733 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/samsung-q10.ko 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/topstar-laptop.ko 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/toshiba_haps.ko 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/wmi-bmof.ko 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/platform/x86/wmi.ko 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/powercap 2023-08-25T09:33:29.734 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/powercap/intel_rapl_common.ko 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/powercap/intel_rapl_msr.ko 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients/pps-gpio.ko 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients/pps-ldisc.ko 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/clients/pps_parport.ko 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/pps/pps_core.ko 2023-08-25T09:33:29.735 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ptp 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ptp/ptp.ko 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ptp/ptp_kvm.ko 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-bq32k.ko 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-bq4802.ko 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1286.ko 2023-08-25T09:33:29.736 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1307.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1374.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1511.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1553.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1672.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds1742.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds2404.ko 2023-08-25T09:33:29.737 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-ds3232.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-em3027.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-fm3130.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-isl12022.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-isl1208.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-m41t80.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-m48t35.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-m48t59.ko 2023-08-25T09:33:29.738 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-max6900.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-msm6242.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf2127.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf85063.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf8523.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf8563.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-pcf8583.ko 2023-08-25T09:33:29.739 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rp5c01.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rs5c372.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rx8025.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-rx8581.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/rtc/rtc-x1205.ko 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi 2023-08-25T09:33:29.740 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/3w-9xxx.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/3w-sas.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/3w-xxxx.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/BusLogic.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/a100u2w.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aacraid 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aacraid/aacraid.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/advansys.ko 2023-08-25T09:33:29.741 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aic7xxx 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/am53c974.ko 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/arcmsr 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/atp870u.ko 2023-08-25T09:33:29.742 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/be2iscsi 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bfa 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bfa/bfa.ko 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2fc 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2i 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-08-25T09:33:29.743 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ch.ko 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/csiostor 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/csiostor/csiostor.ko 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb3i 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb4i 2023-08-25T09:33:29.744 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/dc395x.ko 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-08-25T09:33:29.745 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/dmx3191d.ko 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/esas2r 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/esas2r/esas2r.ko 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/esp_scsi.ko 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fcoe 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fcoe/fcoe.ko 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-08-25T09:33:29.746 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fnic 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/fnic/fnic.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/hpsa.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/hptiop.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/hv_storvsc.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/initio.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ipr.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ips.ko 2023-08-25T09:33:29.747 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/isci 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/isci/isci.ko 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/iscsi_tcp.ko 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libfc 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libfc/libfc.ko 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libiscsi.ko 2023-08-25T09:33:29.748 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libiscsi_tcp.ko 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libsas 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/libsas/libsas.ko 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/lpfc 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/lpfc/lpfc.ko 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid.ko 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-08-25T09:33:29.749 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mpt3sas 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mvsas 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mvsas/mvsas.ko 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/mvumi.ko 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/pm8001 2023-08-25T09:33:29.750 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/pmcraid.ko 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla1280.ko 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla2xxx 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla4xxx 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-08-25T09:33:29.751 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/raid_class.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_debug.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_fc.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_sas.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_spi.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/scsi_transport_srp.ko 2023-08-25T09:33:29.752 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/ses.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/st.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/stex.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/sym53c8xx_2 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/virtio_scsi.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/vmw_pvscsi.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/scsi/wd719x.ko 2023-08-25T09:33:29.753 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ssb 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ssb/ssb.ko 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/iscsi 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/loopback 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/loopback/tcm_loop.ko 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_file.ko 2023-08-25T09:33:29.754 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_iblock.ko 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_mod.ko 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_pscsi.ko 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/target_core_user.ko 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/tcm_fc 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal 2023-08-25T09:33:29.755 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-08-25T09:33:29.756 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-08-25T09:33:29.757 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/ipwireless 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/n_gsm.ko 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/n_hdlc.ko 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/nozomi.ko 2023-08-25T09:33:29.758 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/8250 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/arc_uart.ko 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/jsm 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/serial/jsm/jsm.ko 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/tty/synclink_gt.ko 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs 2023-08-25T09:33:29.759 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/core 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/core/ufshcd-core.ko 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/host 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio.ko 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_aec.ko 2023-08-25T09:33:29.760 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_cif.ko 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_pci_generic.ko 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/uio/uio_sercos3.ko 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/cdc-acm.ko 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/cdc-wdm.ko 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/usblp.ko 2023-08-25T09:33:29.761 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/class/usbtmc.ko 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/host 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/host/sl811-hcd.ko 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/image 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/image/mdc800.ko 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/image/microtek.ko 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc 2023-08-25T09:33:29.762 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/adutux.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/appledisplay.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/emi26.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/emi62.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/ezusb.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/idmouse.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/iowarrior.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/isight_firmware.ko 2023-08-25T09:33:29.763 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/ldusb.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/legousbtower.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/sisusbvga 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/trancevibrator.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/usb3503.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/usblcd.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/usbsevseg.ko 2023-08-25T09:33:29.764 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/uss720.ko 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/misc/yurex.ko 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/phy 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/phy/phy-generic.ko 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/aircable.ko 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ark3116.ko 2023-08-25T09:33:29.765 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/belkin_sa.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ch341.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/cp210x.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/cyberjack.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/cypress_m8.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/digi_acceleport.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/empeg.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ftdi_sio.ko 2023-08-25T09:33:29.766 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/garmin_gps.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/io_edgeport.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/io_ti.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ipaq.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ipw.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ir-usb.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/keyspan.ko 2023-08-25T09:33:29.767 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/keyspan_pda.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/kl5kusb105.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/kobil_sct.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/mct_u232.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/mos7720.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/mos7840.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/navman.ko 2023-08-25T09:33:29.768 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/omninet.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/opticon.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/option.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/oti6858.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/pl2303.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/qcaux.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/qcserial.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/quatech2.ko 2023-08-25T09:33:29.769 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/safe_serial.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/sierra.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/spcp8x5.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ssu100.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/symbolserial.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/usb_debug.ko 2023-08-25T09:33:29.770 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/usb_wwan.ko 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/visor.ko 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/whiteheat.ko 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/serial/xsens_mt.ko 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/uas.ko 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-alauda.ko 2023-08-25T09:33:29.771 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-cypress.ko 2023-08-25T09:33:29.772 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-datafab.ko 2023-08-25T09:33:29.773 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-08-25T09:33:29.773 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-freecom.ko 2023-08-25T09:33:29.773 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-isd200.ko 2023-08-25T09:33:29.773 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-08-25T09:33:29.773 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-karma.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-onetouch.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-realtek.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-sddr09.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-sddr55.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/ums-usbat.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/storage/usb-storage.ko 2023-08-25T09:33:29.774 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip/usbip-core.ko 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip/usbip-host.ko 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/pci 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-08-25T09:33:29.775 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/pci/vfio-pci.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/vfio.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost_iotlb.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost_net.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/vhost/vhost_scsi.ko 2023-08-25T09:33:29.776 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight/apple_bl.ko 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight/lcd.ko 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/video/backlight/platform_lcd.ko 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio.ko 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_balloon.ko 2023-08-25T09:33:29.777 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_mmio.ko 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_pci.ko 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/virtio/virtio_ring.ko 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/masters 2023-08-25T09:33:29.778 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/masters/ds2482.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/masters/ds2490.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-08-25T09:33:29.779 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_smem.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/slaves/w1_therm.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/w1/wire.ko 2023-08-25T09:33:29.780 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/alim1535_wdt.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/alim7101_wdt.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/f71808e_wdt.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/hpwdt.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/i6300esb.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/iTCO_wdt.ko 2023-08-25T09:33:29.781 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/ib700wdt.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/ibmasr.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/it8712f_wdt.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/it87_wdt.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/machzwd.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/nv_tco.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/pcwd_pci.ko 2023-08-25T09:33:29.782 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/pcwd_usb.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/sch311x_wdt.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/softdog.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/sp5100_tco.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/via_wdt.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/w83877f_wdt.ko 2023-08-25T09:33:29.783 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/w83977f_wdt.ko 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/wdt_pci.ko 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/watchdog/xen_wdt.ko 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-acpi-processor.ko 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-evtchn.ko 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-gntalloc.ko 2023-08-25T09:33:29.784 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-gntdev.ko 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-pciback 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xen-privcmd.ko 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xenfs 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/drivers/xen/xenfs/xenfs.ko 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/9p 2023-08-25T09:33:29.785 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/9p/9p.ko 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/affs 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/affs/affs.ko 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/befs 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/befs/befs.ko 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/binfmt_misc.ko 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/btrfs 2023-08-25T09:33:29.786 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/btrfs/btrfs.ko 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cachefiles 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cachefiles/cachefiles.ko 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ceph 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ceph/ceph.ko 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/coda 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/coda/coda.ko 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cramfs 2023-08-25T09:33:29.787 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/cramfs/cramfs.ko 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/dlm 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/dlm/dlm.ko 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ecryptfs 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ecryptfs/ecryptfs.ko 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/f2fs 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/f2fs/f2fs.ko 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat 2023-08-25T09:33:29.788 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat/fat.ko 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat/msdos.ko 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fat/vfat.ko 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fscache 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fscache/fscache.ko 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fuse 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fuse/cuse.ko 2023-08-25T09:33:29.789 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/fuse/fuse.ko 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/gfs2 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/gfs2/gfs2.ko 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfs 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfs/hfs.ko 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfsplus 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/hfsplus/hfsplus.ko 2023-08-25T09:33:29.790 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/isofs 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/isofs/isofs.ko 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/jfs 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/jfs/jfs.ko 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/lockd 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/lockd/lockd.ko 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/minix 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/minix/minix.ko 2023-08-25T09:33:29.791 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/netfs 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/netfs/netfs.ko 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/blocklayout 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/filelayout 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-08-25T09:33:29.792 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/flexfilelayout 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/nfs.ko 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/nfsv3.ko 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs/nfsv4.ko 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs_common 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs_common/grace.ko 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfs_common/nfs_acl.ko 2023-08-25T09:33:29.793 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfsd 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nfsd/nfsd.ko 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nilfs2 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nilfs2/nilfs2.ko 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-celtic.ko 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-centeuro.ko 2023-08-25T09:33:29.794 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-croatian.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-cyrillic.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-gaelic.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-greek.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-iceland.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-inuit.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-roman.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-romanian.ko 2023-08-25T09:33:29.795 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/mac-turkish.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp1250.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp1251.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp1255.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp737.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp775.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp850.ko 2023-08-25T09:33:29.796 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp852.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp855.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp857.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp860.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp861.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp862.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp863.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp864.ko 2023-08-25T09:33:29.797 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp865.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp866.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp869.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp874.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp932.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp936.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp949.ko 2023-08-25T09:33:29.798 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_cp950.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_euc-jp.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-1.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-13.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-14.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-15.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-2.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-3.ko 2023-08-25T09:33:29.799 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-4.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-5.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-6.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-7.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_iso8859-9.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_koi8-r.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_koi8-ru.ko 2023-08-25T09:33:29.800 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_koi8-u.ko 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/nls/nls_utf8.ko 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/cluster 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlm 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlmfs 2023-08-25T09:33:29.801 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2.ko 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/overlayfs 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/overlayfs/overlay.ko 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/pstore 2023-08-25T09:33:29.802 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/pstore/ramoops.ko 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/reiserfs 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/reiserfs/reiserfs.ko 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/romfs 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/romfs/romfs.ko 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/client 2023-08-25T09:33:29.803 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/client/cifs.ko 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/common 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/common/cifs_arc4.ko 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/smb/common/cifs_md4.ko 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/squashfs 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/squashfs/squashfs.ko 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/sysv 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/sysv/sysv.ko 2023-08-25T09:33:29.804 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/udf 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/udf/udf.ko 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ufs 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/ufs/ufs.ko 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/xfs 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/fs/xfs/xfs.ko 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel 2023-08-25T09:33:29.805 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/rcu 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/rcu/rcutorture.ko 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/torture.ko 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/trace 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/asn1_encoder.ko 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crc-itu-t.ko 2023-08-25T09:33:29.806 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crc8.ko 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crypto 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crypto/libarc4.ko 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/crypto/libdes.ko 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/libcrc32c.ko 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lru_cache.ko 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lz4 2023-08-25T09:33:29.807 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lz4/lz4_compress.ko 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/lz4/lz4hc_compress.ko 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/math 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/math/cordic.ko 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/raid6 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/raid6/raid6_pq.ko 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/reed_solomon 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/reed_solomon/reed_solomon.ko 2023-08-25T09:33:29.808 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/ts_bm.ko 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/ts_fsm.ko 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/ts_kmp.ko 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/zstd 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/lib/zstd/zstd_compress.ko 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/mm 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/mm/hwpoison-inject.ko 2023-08-25T09:33:29.809 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/6lowpan.ko 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_dest.ko 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_fragment.ko 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_hop.ko 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_ipv6.ko 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_mobility.ko 2023-08-25T09:33:29.810 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_routing.ko 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/6lowpan/nhc_udp.ko 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/garp.ko 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/mrp.ko 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/p8022.ko 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/psnap.ko 2023-08-25T09:33:29.811 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/802/stp.ko 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/8021q 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/8021q/8021q.ko 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet.ko 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet_fd.ko 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet_rdma.ko 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/9p/9pnet_virtio.ko 2023-08-25T09:33:29.812 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/batman-adv 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/batman-adv/batman-adv.ko 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/br_netfilter.ko 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/bridge.ko 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_among.ko 2023-08-25T09:33:29.813 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_arp.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_ip.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_limit.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_log.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_mark.ko 2023-08-25T09:33:29.814 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_snat.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_stp.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-08-25T09:33:29.815 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/ebtables.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ceph 2023-08-25T09:33:29.816 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ceph/libceph.ko 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/core 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/core/failover.ko 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/core/pktgen.ko 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp.ko 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp_diag.ko 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp_ipv4.ko 2023-08-25T09:33:29.817 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dccp/dccp_ipv6.ko 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dns_resolver 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dns_resolver/dns_resolver.ko 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/dsa_core.ko 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_brcm.ko 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_dsa.ko 2023-08-25T09:33:29.818 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_none.ko 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/dsa/tag_trailer.ko 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/6lowpan 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/ieee802154.ko 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ieee802154/ieee802154_socket.ko 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4 2023-08-25T09:33:29.819 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ah4.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/esp4.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/fou.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/gre.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/inet_diag.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ip_gre.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ip_tunnel.ko 2023-08-25T09:33:29.820 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ip_vti.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ipcomp.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/ipip.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/arp_tables.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-08-25T09:33:29.821 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/iptable_security.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-08-25T09:33:29.822 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-08-25T09:33:29.823 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_bic.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_dctcp.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_diag.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_highspeed.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_htcp.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_hybla.ko 2023-08-25T09:33:29.824 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_illinois.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_lp.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_scalable.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_vegas.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_veno.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_westwood.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tcp_yeah.ko 2023-08-25T09:33:29.825 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/tunnel4.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/udp_diag.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/udp_tunnel.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv4/xfrm4_tunnel.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ah6.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/esp6.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/fou6.ko 2023-08-25T09:33:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ip6_tunnel.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ip6_vti.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/ipcomp6.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-08-25T09:33:29.827 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-08-25T09:33:29.828 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-08-25T09:33:29.829 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/sit.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/tunnel6.ko 2023-08-25T09:33:29.830 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/ipv6/xfrm6_tunnel.ko 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/key 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/key/af_key.ko 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_core.ko 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_debugfs.ko 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_eth.ko 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_ip.ko 2023-08-25T09:33:29.831 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_ip6.ko 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_netlink.ko 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/l2tp/l2tp_ppp.ko 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/llc 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/llc/llc.ko 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mac802154 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mac802154/mac802154.ko 2023-08-25T09:33:29.832 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mpls 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/mpls/mpls_gso.ko 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set.ko 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-08-25T09:33:29.833 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-08-25T09:33:29.834 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-08-25T09:33:29.835 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-08-25T09:33:29.836 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conncount.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-08-25T09:33:29.837 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_h323.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_irc.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_sane.ko 2023-08-25T09:33:29.838 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_sip.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_dup_netdev.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_flow_table.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_flow_table_inet.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_log_syslog.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat.ko 2023-08-25T09:33:29.839 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_amanda.ko 2023-08-25T09:33:29.840 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_ftp.ko 2023-08-25T09:33:29.840 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_irc.ko 2023-08-25T09:33:29.840 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_sip.ko 2023-08-25T09:33:29.840 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_nat_tftp.ko 2023-08-25T09:33:29.840 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_synproxy_core.ko 2023-08-25T09:33:29.840 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nf_tables.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_acct.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_log.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_osf.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nfnetlink_queue.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_chain_nat.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_compat.ko 2023-08-25T09:33:29.841 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_ct.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_dup_netdev.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_fib.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_fwd_netdev.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_hash.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_limit.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_log.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_masq.ko 2023-08-25T09:33:29.842 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_nat.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_numgen.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_queue.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_quota.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_redir.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_reject.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_reject_inet.ko 2023-08-25T09:33:29.843 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_socket.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/nft_tunnel.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_AUDIT.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CHECKSUM.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CLASSIFY.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_CT.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_DSCP.ko 2023-08-25T09:33:29.844 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_HL.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_HMARK.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_IDLETIMER.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_LED.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_LOG.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_MASQUERADE.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_NETMAP.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_NFLOG.ko 2023-08-25T09:33:29.845 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_NFQUEUE.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_RATEEST.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_REDIRECT.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_SECMARK.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TCPMSS.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TEE.ko 2023-08-25T09:33:29.846 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TPROXY.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_TRACE.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_addrtype.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_bpf.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_cgroup.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_cluster.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_comment.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connbytes.ko 2023-08-25T09:33:29.847 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connlabel.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connlimit.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_connmark.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_conntrack.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_cpu.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_dccp.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_devgroup.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_dscp.ko 2023-08-25T09:33:29.848 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_ecn.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_esp.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_hashlimit.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_helper.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_hl.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_ipcomp.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_iprange.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_ipvs.ko 2023-08-25T09:33:29.849 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_l2tp.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_length.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_limit.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_mac.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_mark.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_multiport.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_nat.ko 2023-08-25T09:33:29.850 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_nfacct.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_osf.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_owner.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_physdev.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_pkttype.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_policy.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_quota.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_rateest.ko 2023-08-25T09:33:29.851 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_realm.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_recent.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_sctp.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_set.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_socket.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_state.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_statistic.ko 2023-08-25T09:33:29.852 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_string.ko 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_tcpmss.ko 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_time.ko 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netfilter/xt_u32.ko 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netlink 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/netlink/netlink_diag.ko 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/nsh 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/nsh/nsh.ko 2023-08-25T09:33:29.853 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/openvswitch.ko 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/vport-geneve.ko 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/vport-gre.ko 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/openvswitch/vport-vxlan.ko 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/packet 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/packet/af_packet_diag.ko 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds 2023-08-25T09:33:29.854 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds/rds.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds/rds_rdma.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/rds/rds_tcp.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_csum.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_gact.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_ipt.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_mirred.ko 2023-08-25T09:33:29.855 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_nat.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_pedit.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_police.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_simple.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_skbedit.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/act_vlan.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_basic.ko 2023-08-25T09:33:29.856 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_bpf.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_flow.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_fw.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_route.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/cls_u32.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_cmp.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_ipset.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_meta.ko 2023-08-25T09:33:29.857 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_nbyte.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_text.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/em_u32.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_choke.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_codel.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_drr.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_fq.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_gred.ko 2023-08-25T09:33:29.858 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_hfsc.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_hhf.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_htb.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_ingress.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_mqprio.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_mqprio_lib.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_multiq.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_netem.ko 2023-08-25T09:33:29.859 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_pie.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_plug.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_prio.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_qfq.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_red.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_sfb.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_sfq.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_tbf.ko 2023-08-25T09:33:29.860 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sched/sch_teql.ko 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sctp 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sctp/sctp.ko 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sctp/sctp_diag.ko 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/auth_gss 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-08-25T09:33:29.861 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/sunrpc.ko 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/xprtrdma 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/tipc 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/tipc/diag.ko 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/tipc/tipc.ko 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/unix 2023-08-25T09:33:29.862 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/unix/unix_diag.ko 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vsock.ko 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vsock_diag.ko 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/vmw_vsock/vsock_loopback.ko 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/xfrm 2023-08-25T09:33:29.863 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/net/xfrm/xfrm_ipcomp.ko 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/encrypted-keys 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/trusted-keys 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/security/keys/trusted-keys/trusted.ko 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/virt 2023-08-25T09:33:29.864 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/virt/lib 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/kernel/virt/lib/irqbypass.ko 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.alias 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.alias.bin 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin.alias.bin 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin.bin 2023-08-25T09:33:29.865 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.builtin.modinfo 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.dep 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.dep.bin 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.devname 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.order 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.softdep 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.symbols 2023-08-25T09:33:29.866 INFO:teuthology.orchestra.run.smithi071.stdout:/lib/modules/6.5.0-rc7-ga6a696707610/modules.symbols.bin 2023-08-25T09:33:29.867 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.868 DEBUG:teuthology.orchestra.run.smithi071:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-08-25T09:33:29.892 INFO:teuthology.orchestra.run.smithi071.stdout:postinstall scriptlet (using /bin/sh): 2023-08-25T09:33:29.892 INFO:teuthology.orchestra.run.smithi071.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc7-ga6a696707610 -a -r /boot/System.map-6.5.0-rc7-ga6a696707610 ]; then 2023-08-25T09:33:29.893 INFO:teuthology.orchestra.run.smithi071.stdout:cp /boot/vmlinuz-6.5.0-rc7-ga6a696707610 /boot/.vmlinuz-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:29.893 INFO:teuthology.orchestra.run.smithi071.stdout:cp /boot/System.map-6.5.0-rc7-ga6a696707610 /boot/.System.map-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:29.893 INFO:teuthology.orchestra.run.smithi071.stdout:rm -f /boot/vmlinuz-6.5.0-rc7-ga6a696707610 /boot/System.map-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.893 INFO:teuthology.orchestra.run.smithi071.stdout:/sbin/installkernel 6.5.0-rc7-ga6a696707610 /boot/.vmlinuz-6.5.0-rc7-ga6a696707610-rpm /boot/.System.map-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:29.893 INFO:teuthology.orchestra.run.smithi071.stdout:rm -f /boot/.vmlinuz-6.5.0-rc7-ga6a696707610-rpm /boot/.System.map-6.5.0-rc7-ga6a696707610-rpm 2023-08-25T09:33:29.893 INFO:teuthology.orchestra.run.smithi071.stdout:fi 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:preuninstall scriptlet (using /bin/sh): 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:new-kernel-pkg --remove 6.5.0-rc7-ga6a696707610 --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc7-ga6a696707610.img 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-install remove 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:fi 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:postuninstall scriptlet (using /bin/sh): 2023-08-25T09:33:29.894 INFO:teuthology.orchestra.run.smithi071.stdout:if [ -x /sbin/update-bootloader ]; then 2023-08-25T09:33:29.895 INFO:teuthology.orchestra.run.smithi071.stdout:/sbin/update-bootloader --remove 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.895 INFO:teuthology.orchestra.run.smithi071.stdout:fi 2023-08-25T09:33:29.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2023-08-25T09:33:29.975 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2023-08-25T09:33:29.976 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2023-08-25T09:33:29.976 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2023-08-25T09:33:29.976 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 53.el9 2023-08-25T09:33:29.976 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2023-08-25T09:33:29.976 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Mon 30 Jan 2023 10:38:40 AM UTC 2023-08-25T09:33:29.977 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2023-08-25T09:33:29.977 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8551923 2023-08-25T09:33:29.977 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2023-08-25T09:33:29.977 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Wed 18 Jan 2023 04:40:07 PM UTC, Key ID 05b555b38483c65d 2023-08-25T09:33:29.977 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-53.el9.src.rpm 2023-08-25T09:33:29.977 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Fri 13 Jan 2023 09:07:28 PM UTC 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2023-08-25T09:33:29.978 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:33:29.979 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-08-25T09:33:29.979 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2023-08-25T09:33:29.979 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2023-08-25T09:33:29.979 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2023-08-25T09:33:29.979 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:33:29.980 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2023-08-25T09:33:29.981 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-08-25T09:33:29.981 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 6.5.0-rc7-ga6a696707610 2023-08-25T09:33:29.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-08-25T09:33:30.663 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2023-08-25T09:33:32.114 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-08-25T09:33:32.140 INFO:teuthology.orchestra.run.smithi071.stderr:done 2023-08-25T09:33:32.143 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-08-25T09:33:32.185 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.7PWRXAt4Vq 2023-08-25T09:33:32.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7PWRXAt4Vq 2023-08-25T09:33:32.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.7PWRXAt4Vq 2023-08-25T09:33:32.409 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.7PWRXAt4Vq is 6KB 2023-08-25T09:33:32.431 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.7PWRXAt4Vq 2023-08-25T09:33:32.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2023-08-25T09:33:32.513 INFO:teuthology.orchestra.run.smithi071.stdout:e13cba1d25f14c51a1d04da8db823a4e-0-rescue.conf 2023-08-25T09:33:32.514 INFO:teuthology.orchestra.run.smithi071.stdout:e13cba1d25f14c51a1d04da8db823a4e-5.14.0-239.el9.x86_64.conf 2023-08-25T09:33:32.514 INFO:teuthology.orchestra.run.smithi071.stdout:e13cba1d25f14c51a1d04da8db823a4e-6.5.0-rc7-ga6a696707610.conf 2023-08-25T09:33:32.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default e13cba1d25f14c51a1d04da8db823a4e-6.5.0-rc7-ga6a696707610 2023-08-25T09:33:32.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2023-08-25T09:33:48.160 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-08-25T09:33:48.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-08-25T09:33:48.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:34:02.696 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-08-25T09:34:02.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-08-25T09:34:02.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:34:22.832 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-08-25T09:34:31.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-08-25T09:34:31.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:34:37.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-08-25T09:34:46.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-08-25T09:34:46.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:34:50.224 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-08-25T09:35:02.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-08-25T09:35:02.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:35:03.099 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-08-25T09:35:03.921 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-08-25T09:35:03.921 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.5.0-rc7-ga6a696707610"... 2023-08-25T09:35:03.922 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2023-08-25T09:35:03.962 INFO:teuthology.orchestra.run.smithi164.stdout:6.5.0-rc7-ga6a696707610 2023-08-25T09:35:03.962 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc7-ga6a696707610 vs 6.5.0-rc7-ga6a696707610 2023-08-25T09:35:03.962 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-08-25T09:35:03.963 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-08-25T09:35:04.564 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-08-25T09:35:04.964 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-08-25T09:35:04.964 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-25T09:35:05.217 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2023-08-25T09:35:05.273 DEBUG:teuthology.parallel:result is None 2023-08-25T09:35:16.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-08-25T09:35:16.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:35:17.904 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-08-25T09:35:18.674 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-08-25T09:35:18.675 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.5.0-rc7-ga6a696707610"... 2023-08-25T09:35:18.675 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2023-08-25T09:35:18.715 INFO:teuthology.orchestra.run.smithi071.stdout:6.5.0-rc7-ga6a696707610 2023-08-25T09:35:18.715 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc7-ga6a696707610 vs 6.5.0-rc7-ga6a696707610 2023-08-25T09:35:18.715 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-08-25T09:35:18.715 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-08-25T09:35:19.717 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-08-25T09:35:19.717 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-25T09:35:19.842 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2023-08-25T09:35:19.875 DEBUG:teuthology.parallel:result is None 2023-08-25T09:35:19.876 INFO:teuthology.run_tasks:Running task internal.base... 2023-08-25T09:35:19.885 INFO:teuthology.task.internal:Creating test directory... 2023-08-25T09:35:19.886 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-25T09:35:19.892 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-25T09:35:19.987 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-08-25T09:35:19.997 INFO:teuthology.run_tasks:Running task internal.archive... 2023-08-25T09:35:20.019 INFO:teuthology.task.internal:Creating archive directory... 2023-08-25T09:35:20.020 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-25T09:35:20.026 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-25T09:35:20.135 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-08-25T09:35:20.141 INFO:teuthology.task.internal:Enabling coredump saving... 2023-08-25T09:35:20.142 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-25T09:35:20.146 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-25T09:35:20.263 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T09:35:20.303 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T09:35:20.314 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T09:35:20.387 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-25T09:35:20.395 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-08-25T09:35:20.403 INFO:teuthology.task.internal:Configuring sudo... 2023-08-25T09:35:20.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-25T09:35:20.410 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-25T09:35:20.480 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-08-25T09:35:20.488 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-08-25T09:35:20.489 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-25T09:35:20.497 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-25T09:35:20.538 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T09:35:20.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T09:35:20.735 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T09:35:20.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T09:35:20.900 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:35:20.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-25T09:35:20.969 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T09:35:21.051 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-25T09:35:21.160 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T09:35:21.232 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-25T09:35:21.308 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:35:21.308 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-25T09:35:21.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2023-08-25T09:35:21.390 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2023-08-25T09:35:21.515 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T09:35:21.545 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T09:35:21.837 INFO:teuthology.run_tasks:Running task internal.timer... 2023-08-25T09:35:21.850 INFO:teuthology.task.internal:Starting timer... 2023-08-25T09:35:21.850 INFO:teuthology.run_tasks:Running task pcp... 2023-08-25T09:35:21.889 INFO:teuthology.run_tasks:Running task selinux... 2023-08-25T09:35:21.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2023-08-25T09:35:22.026 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2023-08-25T09:35:22.029 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2023-08-25T09:35:22.298 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2023-08-25T09:35:22.307 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-08-25T09:35:22.308 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2023-08-25T09:35:22.365 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2023-08-25T09:35:22.365 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2023-08-25T09:35:22.502 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2023-08-25T09:35:22.503 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2023-08-25T09:35:22.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T09:35:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T09:35:22.590 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T09:35:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T09:35:22.670 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-08-25T09:35:22.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2023-08-25T09:35:22.779 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2023-08-25T09:35:22.874 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-08-25T09:35:22.886 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-08-25T09:35:22.887 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-08-25T09:35:22.908 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-08-25T09:35:22.911 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-08-25T09:46:22.647 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2023-08-25T09:46:23.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-08-25T09:46:23.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:46:23.843 DEBUG:teuthology.orchestra.run.smithi071:> true 2023-08-25T09:46:24.000 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2023-08-25T09:46:24.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-08-25T09:46:24.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-25T09:46:24.144 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-08-25T09:46:24.311 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-08-25T09:46:24.312 INFO:teuthology.run_tasks:Running task clock... 2023-08-25T09:46:24.325 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-08-25T09:46:24.326 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-25T09:46:24.326 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T09:46:24.336 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-25T09:46:24.337 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T09:46:24.436 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-25T09:46:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-25T09:46:24.489 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-25T09:46:24.498 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-25T09:46:24.588 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2023-08-25T09:46:24.593 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2023-08-25T09:46:24.631 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2023-08-25T09:46:24.635 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2023-08-25T09:46:24.683 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-25T09:46:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-25T09:46:24.741 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-25T09:46:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-25T09:46:24.867 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2023-08-25T09:46:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2023-08-25T09:46:24.877 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T09:46:24.878 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-08-25T09:46:24.878 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.878 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.878 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.878 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.882 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T09:46:24.882 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2023-08-25T09:46:24.882 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.882 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.883 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.883 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T09:46:24.884 INFO:teuthology.run_tasks:Running task install... 2023-08-25T09:46:24.892 DEBUG:teuthology.task.install:project ceph 2023-08-25T09:46:24.892 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'}} 2023-08-25T09:46:24.893 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'} 2023-08-25T09:46:24.893 INFO:teuthology.task.install:Using flavor: default 2023-08-25T09:46:24.899 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-08-25T09:46:24.900 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-08-25T09:46:24.900 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'tag': None, 'wait_for_package': False} 2023-08-25T09:46:24.901 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T09:46:24.903 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'tag': None, 'wait_for_package': False} 2023-08-25T09:46:24.903 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T09:46:25.042 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/ 2023-08-25T09:46:25.043 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5703.gba6b39e4 2023-08-25T09:46:25.116 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/ 2023-08-25T09:46:25.116 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5703.gba6b39e4 2023-08-25T09:46:25.222 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-25T09:46:25.223 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:46:25.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-25T09:46:25.228 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-rishabh-2023aug3-b5/ba6b39e4f4f77fb95c747099b53db0467d435f56/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-25T09:46:25.228 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:46:25.228 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-25T09:46:25.334 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-08-25T09:46:25.334 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba6b39e4f4f77fb95c747099b53db0467d435f56/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-25T09:46:25.335 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-08-25T09:46:25.336 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba6b39e4f4f77fb95c747099b53db0467d435f56/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-25T09:46:25.479 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-25T09:46:25.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-25T09:46:25.619 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-25T09:46:25.621 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-25T09:46:25.715 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2023-08-25T09:46:25.716 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2023-08-25T09:46:25.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2023-08-25T09:46:25.721 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2023-08-25T09:46:26.091 INFO:teuthology.orchestra.run.smithi071.stdout:72 files removed 2023-08-25T09:46:26.097 INFO:teuthology.orchestra.run.smithi164.stdout:72 files removed 2023-08-25T09:46:26.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-08-25T09:46:26.136 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-08-25T09:46:27.018 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 173 kB/s | 78 kB 00:00 2023-08-25T09:46:27.045 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 163 kB/s | 78 kB 00:00 2023-08-25T09:46:27.476 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 25 kB/s | 9.4 kB 00:00 2023-08-25T09:46:27.611 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 19 kB/s | 9.4 kB 00:00 2023-08-25T09:46:28.010 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2023-08-25T09:46:28.162 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2023-08-25T09:46:28.598 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 12 MB/s | 6.2 MB 00:00 2023-08-25T09:46:28.783 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 11 MB/s | 6.2 MB 00:00 2023-08-25T09:46:30.157 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2023-08-25T09:46:30.324 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2023-08-25T09:46:31.287 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 7.8 MB 00:00 2023-08-25T09:46:32.027 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 7.8 MB 00:01 2023-08-25T09:46:33.845 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 25 MB/s | 18 MB 00:00 2023-08-25T09:46:34.310 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 18 MB 00:01 2023-08-25T09:46:39.038 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 5.7 MB 00:01 2023-08-25T09:46:39.118 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.7 MB 00:01 2023-08-25T09:46:40.481 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 14 kB 00:00 2023-08-25T09:46:41.505 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 14 kB 00:01 2023-08-25T09:46:42.141 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 12 MB/s | 18 MB 00:01 2023-08-25T09:46:43.765 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 18 MB 00:02 2023-08-25T09:46:49.155 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-08-25T09:46:49.157 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-08-25T09:46:49.176 INFO:teuthology.orchestra.run.smithi071.stdout:Package NetworkManager-1:1.41.8-1.el9.x86_64 is already installed. 2023-08-25T09:46:49.177 INFO:teuthology.orchestra.run.smithi071.stdout:Package iproute-6.0.0-2.el9.x86_64 is already installed. 2023-08-25T09:46:49.178 INFO:teuthology.orchestra.run.smithi071.stdout:Package util-linux-2.37.4-9.el9.x86_64 is already installed. 2023-08-25T09:46:49.180 INFO:teuthology.orchestra.run.smithi071.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-08-25T09:46:49.182 INFO:teuthology.orchestra.run.smithi071.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-08-25T09:46:49.186 INFO:teuthology.orchestra.run.smithi071.stdout:Package xfsdump-3.1.10-1.el9.x86_64 is already installed. 2023-08-25T09:46:49.187 INFO:teuthology.orchestra.run.smithi071.stdout:Package xfsprogs-5.14.2-1.el9.x86_64 is already installed. 2023-08-25T09:46:49.269 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2023-08-25T09:46:49.288 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2023-08-25T09:46:49.289 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2023-08-25T09:46:49.289 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2023-08-25T09:46:49.289 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2023-08-25T09:46:49.289 INFO:teuthology.orchestra.run.smithi071.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-08-25T09:46:49.289 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 6.5 k 2023-08-25T09:46:49.289 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 5.2 M 2023-08-25T09:46:49.290 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 850 k 2023-08-25T09:46:49.290 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 141 k 2023-08-25T09:46:49.290 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 1.5 M 2023-08-25T09:46:49.290 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 132 k 2023-08-25T09:46:49.290 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 1.7 M 2023-08-25T09:46:49.290 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 7.4 M 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 47 k 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 8.2 M 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 40 M 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 261 k 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 225 k 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-mirror x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 218 k 2023-08-25T09:46:49.291 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-top noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 18 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 CentOS-AppStream 21 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 31 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 667 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 126 k 2023-08-25T09:46:49.292 INFO:teuthology.orchestra.run.smithi071.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-08-25T09:46:49.293 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-devel x86_64 2.37.4-14.el9 CentOS-AppStream 24 k 2023-08-25T09:46:49.293 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-devel x86_64 1:3.0.7-24.el9 CentOS-AppStream 3.0 M 2023-08-25T09:46:49.293 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-08-25T09:46:49.293 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-08-25T09:46:49.294 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-08-25T09:46:49.294 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 161 k 2023-08-25T09:46:49.294 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 320 k 2023-08-25T09:46:49.295 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 297 k 2023-08-25T09:46:49.295 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 99 k 2023-08-25T09:46:49.295 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 85 k 2023-08-25T09:46:49.296 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 3.0 M 2023-08-25T09:46:49.297 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 169 k 2023-08-25T09:46:49.297 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-08-25T09:46:49.297 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2023-08-25T09:46:49.297 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager x86_64 1:1.44.0-2.el9 CentOS-BaseOS 2.2 M 2023-08-25T09:46:49.298 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-libnm x86_64 1:1.44.0-2.el9 CentOS-BaseOS 1.8 M 2023-08-25T09:46:49.298 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-team x86_64 1:1.44.0-2.el9 CentOS-BaseOS 41 k 2023-08-25T09:46:49.298 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-tui x86_64 1:1.44.0-2.el9 CentOS-BaseOS 245 k 2023-08-25T09:46:49.299 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-debuginfod-client x86_64 0.189-3.el9 CentOS-BaseOS 36 k 2023-08-25T09:46:49.299 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf x86_64 0.189-3.el9 CentOS-BaseOS 192 k 2023-08-25T09:46:49.300 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libs x86_64 0.189-3.el9 CentOS-BaseOS 254 k 2023-08-25T09:46:49.300 INFO:teuthology.orchestra.run.smithi071.stdout: iproute x86_64 6.2.0-5.el9 CentOS-BaseOS 785 k 2023-08-25T09:46:49.301 INFO:teuthology.orchestra.run.smithi071.stdout: iproute-tc x86_64 6.2.0-5.el9 CentOS-BaseOS 448 k 2023-08-25T09:46:49.301 INFO:teuthology.orchestra.run.smithi071.stdout: libblkid x86_64 2.37.4-14.el9 CentOS-BaseOS 107 k 2023-08-25T09:46:49.301 INFO:teuthology.orchestra.run.smithi071.stdout: libfdisk x86_64 2.37.4-14.el9 CentOS-BaseOS 154 k 2023-08-25T09:46:49.301 INFO:teuthology.orchestra.run.smithi071.stdout: libmount x86_64 2.37.4-14.el9 CentOS-BaseOS 133 k 2023-08-25T09:46:49.302 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 3.3 M 2023-08-25T09:46:49.302 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 3.0 M 2023-08-25T09:46:49.302 INFO:teuthology.orchestra.run.smithi071.stdout: libsmartcols x86_64 2.37.4-14.el9 CentOS-BaseOS 63 k 2023-08-25T09:46:49.302 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid x86_64 2.37.4-14.el9 CentOS-BaseOS 28 k 2023-08-25T09:46:49.303 INFO:teuthology.orchestra.run.smithi071.stdout: openssl x86_64 1:3.0.7-24.el9 CentOS-BaseOS 1.2 M 2023-08-25T09:46:49.303 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-libs x86_64 1:3.0.7-24.el9 CentOS-BaseOS 2.1 M 2023-08-25T09:46:49.303 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux x86_64 2.37.4-14.el9 CentOS-BaseOS 2.2 M 2023-08-25T09:46:49.303 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-core x86_64 2.37.4-14.el9 CentOS-BaseOS 433 k 2023-08-25T09:46:49.304 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-user x86_64 2.37.4-14.el9 CentOS-BaseOS 30 k 2023-08-25T09:46:49.304 INFO:teuthology.orchestra.run.smithi071.stdout: xfsdump x86_64 3.1.12-3.el9 CentOS-BaseOS 321 k 2023-08-25T09:46:49.304 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs x86_64 5.19.0-4.el9 CentOS-BaseOS 1.0 M 2023-08-25T09:46:49.305 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2023-08-25T09:46:49.305 INFO:teuthology.orchestra.run.smithi071.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-08-25T09:46:49.305 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-08-25T09:46:49.305 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 19 M 2023-08-25T09:46:49.306 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 23 k 2023-08-25T09:46:49.306 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 2.1 M 2023-08-25T09:46:49.306 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 242 k 2023-08-25T09:46:49.307 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 4.4 M 2023-08-25T09:46:49.307 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 18 M 2023-08-25T09:46:49.307 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 15 k 2023-08-25T09:46:49.307 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 24 k 2023-08-25T09:46:49.308 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-08-25T09:46:49.308 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-08-25T09:46:49.308 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-08-25T09:46:49.308 INFO:teuthology.orchestra.run.smithi071.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-08-25T09:46:49.309 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-08-25T09:46:49.309 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-08-25T09:46:49.309 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-15.el9 CentOS-AppStream 186 k 2023-08-25T09:46:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-08-25T09:46:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-08-25T09:46:49.310 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 164 k 2023-08-25T09:46:49.311 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-2.1.el9 CentOS-BaseOS 807 k 2023-08-25T09:46:49.311 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-08-25T09:46:49.311 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-08-25T09:46:49.311 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-2.1.el9 CentOS-BaseOS 197 k 2023-08-25T09:46:49.312 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-08-25T09:46:49.312 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 473 k 2023-08-25T09:46:49.312 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-08-25T09:46:49.313 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 4.7 M 2023-08-25T09:46:49.313 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-08-25T09:46:49.313 INFO:teuthology.orchestra.run.smithi071.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-08-25T09:46:49.313 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-08-25T09:46:49.314 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-08-25T09:46:49.314 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-08-25T09:46:49.314 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-08-25T09:46:49.315 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-08-25T09:46:49.315 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-08-25T09:46:49.315 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-08-25T09:46:49.315 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-08-25T09:46:49.316 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-08-25T09:46:49.316 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-08-25T09:46:49.316 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 45 k 2023-08-25T09:46:49.317 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 127 k 2023-08-25T09:46:49.317 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-08-25T09:46:49.317 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-08-25T09:46:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-08-25T09:46:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-08-25T09:46:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-08-25T09:46:49.318 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.16-1.el9 appstream 247 k 2023-08-25T09:46:49.319 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-08-25T09:46:49.319 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-08-25T09:46:49.319 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-08-25T09:46:49.320 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-08-25T09:46:49.320 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-08-25T09:46:49.320 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-08-25T09:46:49.320 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-08-25T09:46:49.321 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-08-25T09:46:49.321 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-08-25T09:46:49.321 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-08-25T09:46:49.322 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-08-25T09:46:49.322 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-08-25T09:46:49.322 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-08-25T09:46:49.322 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-08-25T09:46:49.323 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-08-25T09:46:49.323 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-08-25T09:46:49.323 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-08-25T09:46:49.323 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-08-25T09:46:49.324 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-08-25T09:46:49.324 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-08-25T09:46:49.324 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-08-25T09:46:49.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-08-25T09:46:49.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-08-25T09:46:49.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-08-25T09:46:49.325 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-08-25T09:46:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-08-25T09:46:49.326 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-08-25T09:46:49.327 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-08-25T09:46:49.327 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-08-25T09:46:49.327 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-08-25T09:46:49.327 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-08-25T09:46:49.328 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-08-25T09:46:49.328 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-08-25T09:46:49.328 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-08-25T09:46:49.329 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-08-25T09:46:49.329 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-08-25T09:46:49.329 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-08-25T09:46:49.329 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-08-25T09:46:49.330 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-08-25T09:46:49.330 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-08-25T09:46:49.330 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-08-25T09:46:49.331 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-08-25T09:46:49.331 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-08-25T09:46:49.331 INFO:teuthology.orchestra.run.smithi071.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-08-25T09:46:49.331 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-08-25T09:46:49.332 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel x86_64 1.2.11-36.el9 appstream 45 k 2023-08-25T09:46:49.332 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2023-08-25T09:46:49.332 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-08-25T09:46:49.333 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:46:49.333 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2023-08-25T09:46:49.333 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2023-08-25T09:46:49.333 INFO:teuthology.orchestra.run.smithi071.stdout:Install 131 Packages 2023-08-25T09:46:49.334 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 23 Packages 2023-08-25T09:46:49.334 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:46:49.335 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 206 M 2023-08-25T09:46:49.335 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2023-08-25T09:46:50.017 INFO:teuthology.orchestra.run.smithi071.stdout:(1/154): ceph-18.0.0-5703.gba6b39e4.el9.x86_64. 21 kB/s | 6.5 kB 00:00 2023-08-25T09:46:50.317 INFO:teuthology.orchestra.run.smithi071.stdout:(2/154): ceph-fuse-18.0.0-5703.gba6b39e4.el9.x8 2.8 MB/s | 850 kB 00:00 2023-08-25T09:46:50.409 INFO:teuthology.orchestra.run.smithi071.stdout:(3/154): ceph-immutable-object-cache-18.0.0-570 1.5 MB/s | 141 kB 00:00 2023-08-25T09:46:50.542 INFO:teuthology.orchestra.run.smithi071.stdout:(4/154): ceph-base-18.0.0-5703.gba6b39e4.el9.x8 6.2 MB/s | 5.2 MB 00:00 2023-08-25T09:46:50.742 INFO:teuthology.orchestra.run.smithi071.stdout:(5/154): ceph-mgr-18.0.0-5703.gba6b39e4.el9.x86 7.3 MB/s | 1.5 MB 00:00 2023-08-25T09:46:50.787 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-08-25T09:46:50.789 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-08-25T09:46:50.810 INFO:teuthology.orchestra.run.smithi164.stdout:Package NetworkManager-1:1.41.8-1.el9.x86_64 is already installed. 2023-08-25T09:46:50.812 INFO:teuthology.orchestra.run.smithi164.stdout:Package iproute-6.0.0-2.el9.x86_64 is already installed. 2023-08-25T09:46:50.813 INFO:teuthology.orchestra.run.smithi164.stdout:Package util-linux-2.37.4-9.el9.x86_64 is already installed. 2023-08-25T09:46:50.815 INFO:teuthology.orchestra.run.smithi164.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-08-25T09:46:50.817 INFO:teuthology.orchestra.run.smithi164.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-08-25T09:46:50.821 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsdump-3.1.10-1.el9.x86_64 is already installed. 2023-08-25T09:46:50.822 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsprogs-5.14.2-1.el9.x86_64 is already installed. 2023-08-25T09:46:50.867 INFO:teuthology.orchestra.run.smithi071.stdout:(6/154): ceph-mds-18.0.0-5703.gba6b39e4.el9.x86 4.6 MB/s | 2.1 MB 00:00 2023-08-25T09:46:50.906 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2023-08-25T09:46:50.936 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2023-08-25T09:46:50.937 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2023-08-25T09:46:50.937 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2023-08-25T09:46:50.937 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2023-08-25T09:46:50.937 INFO:teuthology.orchestra.run.smithi164.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-08-25T09:46:50.937 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 6.5 k 2023-08-25T09:46:50.937 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 5.2 M 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 850 k 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 141 k 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 1.5 M 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 132 k 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 1.7 M 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 7.4 M 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 47 k 2023-08-25T09:46:50.938 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 8.2 M 2023-08-25T09:46:50.939 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 40 M 2023-08-25T09:46:50.939 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 261 k 2023-08-25T09:46:50.939 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 225 k 2023-08-25T09:46:50.939 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 218 k 2023-08-25T09:46:50.939 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 18 k 2023-08-25T09:46:50.939 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 CentOS-AppStream 21 k 2023-08-25T09:46:50.940 INFO:teuthology.orchestra.run.smithi164.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-08-25T09:46:50.940 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-08-25T09:46:50.940 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 31 k 2023-08-25T09:46:50.940 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 667 k 2023-08-25T09:46:50.940 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 126 k 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel x86_64 2.37.4-14.el9 CentOS-AppStream 24 k 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel x86_64 1:3.0.7-24.el9 CentOS-AppStream 3.0 M 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 161 k 2023-08-25T09:46:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 320 k 2023-08-25T09:46:50.942 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 297 k 2023-08-25T09:46:50.942 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 99 k 2023-08-25T09:46:50.942 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 85 k 2023-08-25T09:46:50.942 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 3.0 M 2023-08-25T09:46:50.942 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 169 k 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager x86_64 1:1.44.0-2.el9 CentOS-BaseOS 2.2 M 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm x86_64 1:1.44.0-2.el9 CentOS-BaseOS 1.8 M 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team x86_64 1:1.44.0-2.el9 CentOS-BaseOS 41 k 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui x86_64 1:1.44.0-2.el9 CentOS-BaseOS 245 k 2023-08-25T09:46:50.943 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client x86_64 0.189-3.el9 CentOS-BaseOS 36 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf x86_64 0.189-3.el9 CentOS-BaseOS 192 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs x86_64 0.189-3.el9 CentOS-BaseOS 254 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: iproute x86_64 6.2.0-5.el9 CentOS-BaseOS 785 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: iproute-tc x86_64 6.2.0-5.el9 CentOS-BaseOS 448 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: libblkid x86_64 2.37.4-14.el9 CentOS-BaseOS 107 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: libfdisk x86_64 2.37.4-14.el9 CentOS-BaseOS 154 k 2023-08-25T09:46:50.944 INFO:teuthology.orchestra.run.smithi164.stdout: libmount x86_64 2.37.4-14.el9 CentOS-BaseOS 133 k 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 3.3 M 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 3.0 M 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: libsmartcols x86_64 2.37.4-14.el9 CentOS-BaseOS 63 k 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid x86_64 2.37.4-14.el9 CentOS-BaseOS 28 k 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: openssl x86_64 1:3.0.7-24.el9 CentOS-BaseOS 1.2 M 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs x86_64 1:3.0.7-24.el9 CentOS-BaseOS 2.1 M 2023-08-25T09:46:50.945 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux x86_64 2.37.4-14.el9 CentOS-BaseOS 2.2 M 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-core x86_64 2.37.4-14.el9 CentOS-BaseOS 433 k 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-user x86_64 2.37.4-14.el9 CentOS-BaseOS 30 k 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: xfsdump x86_64 3.1.12-3.el9 CentOS-BaseOS 321 k 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs x86_64 5.19.0-4.el9 CentOS-BaseOS 1.0 M 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-08-25T09:46:50.946 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 19 M 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 23 k 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 2.1 M 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 242 k 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 4.4 M 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 18 M 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5703.gba6b39e4.el9 ceph-noarch 15 k 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 24 k 2023-08-25T09:46:50.947 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_64 1.6-15.el9 CentOS-AppStream 186 k 2023-08-25T09:46:50.948 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 164 k 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-2.1.el9 CentOS-BaseOS 807 k 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-2.1.el9 CentOS-BaseOS 197 k 2023-08-25T09:46:50.949 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-08-25T09:46:50.950 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 473 k 2023-08-25T09:46:50.950 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-08-25T09:46:50.950 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 4.7 M 2023-08-25T09:46:50.950 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-08-25T09:46:50.950 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-08-25T09:46:50.950 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-08-25T09:46:50.951 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-08-25T09:46:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-08-25T09:46:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-08-25T09:46:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 45 k 2023-08-25T09:46:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:18.0.0-5703.gba6b39e4.el9 ceph 127 k 2023-08-25T09:46:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-08-25T09:46:50.952 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-08-25T09:46:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-08-25T09:46:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-08-25T09:46:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-08-25T09:46:50.953 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.16-1.el9 appstream 247 k 2023-08-25T09:46:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-08-25T09:46:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-08-25T09:46:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-08-25T09:46:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-08-25T09:46:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-08-25T09:46:50.954 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-08-25T09:46:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-08-25T09:46:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-08-25T09:46:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-08-25T09:46:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-08-25T09:46:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-08-25T09:46:50.955 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-08-25T09:46:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-08-25T09:46:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-08-25T09:46:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-08-25T09:46:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-08-25T09:46:50.956 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-08-25T09:46:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-08-25T09:46:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-08-25T09:46:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-08-25T09:46:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-08-25T09:46:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-08-25T09:46:50.957 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-08-25T09:46:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-08-25T09:46:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-08-25T09:46:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-08-25T09:46:50.958 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-08-25T09:46:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-08-25T09:46:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-08-25T09:46:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-08-25T09:46:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-08-25T09:46:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-08-25T09:46:50.959 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-08-25T09:46:50.960 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-08-25T09:46:50.960 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-08-25T09:46:50.960 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-08-25T09:46:50.960 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-08-25T09:46:50.960 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-08-25T09:46:50.961 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-08-25T09:46:50.961 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-08-25T09:46:50.961 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-08-25T09:46:50.961 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-08-25T09:46:50.961 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-08-25T09:46:50.962 INFO:teuthology.orchestra.run.smithi164.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-08-25T09:46:50.962 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-08-25T09:46:50.962 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel x86_64 1.2.11-36.el9 appstream 45 k 2023-08-25T09:46:50.962 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2023-08-25T09:46:50.962 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-08-25T09:46:50.963 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:46:50.963 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2023-08-25T09:46:50.963 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================================ 2023-08-25T09:46:50.963 INFO:teuthology.orchestra.run.smithi164.stdout:Install 131 Packages 2023-08-25T09:46:50.964 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 23 Packages 2023-08-25T09:46:50.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:46:50.964 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 206 M 2023-08-25T09:46:50.965 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2023-08-25T09:46:51.259 INFO:teuthology.orchestra.run.smithi071.stdout:(7/154): ceph-mon-18.0.0-5703.gba6b39e4.el9.x86 8.5 MB/s | 4.4 MB 00:00 2023-08-25T09:46:51.491 INFO:teuthology.orchestra.run.smithi164.stdout:(1/154): ceph-18.0.0-5703.gba6b39e4.el9.x86_64. 23 kB/s | 6.5 kB 00:00 2023-08-25T09:46:51.535 INFO:teuthology.orchestra.run.smithi071.stdout:(8/154): ceph-common-18.0.0-5703.gba6b39e4.el9. 10 MB/s | 19 MB 00:01 2023-08-25T09:46:51.623 INFO:teuthology.orchestra.run.smithi071.stdout:(9/154): ceph-selinux-18.0.0-5703.gba6b39e4.el9 279 kB/s | 24 kB 00:00 2023-08-25T09:46:51.966 INFO:teuthology.orchestra.run.smithi164.stdout:(2/154): ceph-fuse-18.0.0-5703.gba6b39e4.el9.x8 1.8 MB/s | 850 kB 00:00 2023-08-25T09:46:52.049 INFO:teuthology.orchestra.run.smithi164.stdout:(3/154): ceph-immutable-object-cache-18.0.0-570 1.7 MB/s | 141 kB 00:00 2023-08-25T09:46:52.127 INFO:teuthology.orchestra.run.smithi071.stdout:(10/154): ceph-radosgw-18.0.0-5703.gba6b39e4.el 9.4 MB/s | 8.2 MB 00:00 2023-08-25T09:46:52.210 INFO:teuthology.orchestra.run.smithi071.stdout:(11/154): cephfs-mirror-18.0.0-5703.gba6b39e4.e 2.6 MB/s | 218 kB 00:00 2023-08-25T09:46:52.302 INFO:teuthology.orchestra.run.smithi071.stdout:(12/154): libcephfs-devel-18.0.0-5703.gba6b39e4 346 kB/s | 31 kB 00:00 2023-08-25T09:46:52.525 INFO:teuthology.orchestra.run.smithi164.stdout:(4/154): ceph-base-18.0.0-5703.gba6b39e4.el9.x8 3.9 MB/s | 5.2 MB 00:01 2023-08-25T09:46:52.717 INFO:teuthology.orchestra.run.smithi164.stdout:(5/154): ceph-mds-18.0.0-5703.gba6b39e4.el9.x86 3.2 MB/s | 2.1 MB 00:00 2023-08-25T09:46:52.792 INFO:teuthology.orchestra.run.smithi164.stdout:(6/154): ceph-mgr-18.0.0-5703.gba6b39e4.el9.x86 5.5 MB/s | 1.5 MB 00:00 2023-08-25T09:46:52.819 INFO:teuthology.orchestra.run.smithi071.stdout:(13/154): libcephfs2-18.0.0-5703.gba6b39e4.el9. 1.3 MB/s | 667 kB 00:00 2023-08-25T09:46:52.919 INFO:teuthology.orchestra.run.smithi071.stdout:(14/154): libcephsqlite-18.0.0-5703.gba6b39e4.e 1.6 MB/s | 164 kB 00:00 2023-08-25T09:46:53.028 INFO:teuthology.orchestra.run.smithi071.stdout:(15/154): librados-devel-18.0.0-5703.gba6b39e4. 1.2 MB/s | 126 kB 00:00 2023-08-25T09:46:53.445 INFO:teuthology.orchestra.run.smithi071.stdout:(16/154): ceph-osd-18.0.0-5703.gba6b39e4.el9.x8 6.8 MB/s | 18 MB 00:02 2023-08-25T09:46:53.478 INFO:teuthology.orchestra.run.smithi071.stdout:(17/154): libradosstriper1-18.0.0-5703.gba6b39e 1.0 MB/s | 473 kB 00:00 2023-08-25T09:46:53.562 INFO:teuthology.orchestra.run.smithi071.stdout:(18/154): python3-ceph-argparse-18.0.0-5703.gba 545 kB/s | 45 kB 00:00 2023-08-25T09:46:53.629 INFO:teuthology.orchestra.run.smithi071.stdout:(19/154): python3-ceph-common-18.0.0-5703.gba6b 1.9 MB/s | 127 kB 00:00 2023-08-25T09:46:53.661 INFO:teuthology.orchestra.run.smithi164.stdout:(7/154): ceph-mon-18.0.0-5703.gba6b39e4.el9.x86 4.6 MB/s | 4.4 MB 00:00 2023-08-25T09:46:53.696 INFO:teuthology.orchestra.run.smithi071.stdout:(20/154): python3-cephfs-18.0.0-5703.gba6b39e4. 2.4 MB/s | 161 kB 00:00 2023-08-25T09:46:53.813 INFO:teuthology.orchestra.run.smithi071.stdout:(21/154): python3-rados-18.0.0-5703.gba6b39e4.e 2.7 MB/s | 320 kB 00:00 2023-08-25T09:46:53.905 INFO:teuthology.orchestra.run.smithi071.stdout:(22/154): python3-rbd-18.0.0-5703.gba6b39e4.el9 3.2 MB/s | 297 kB 00:00 2023-08-25T09:46:53.989 INFO:teuthology.orchestra.run.smithi071.stdout:(23/154): python3-rgw-18.0.0-5703.gba6b39e4.el9 1.2 MB/s | 99 kB 00:00 2023-08-25T09:46:54.064 INFO:teuthology.orchestra.run.smithi071.stdout:(24/154): rbd-fuse-18.0.0-5703.gba6b39e4.el9.x8 1.1 MB/s | 85 kB 00:00 2023-08-25T09:46:54.798 INFO:teuthology.orchestra.run.smithi071.stdout:(25/154): rbd-mirror-18.0.0-5703.gba6b39e4.el9. 4.1 MB/s | 3.0 MB 00:00 2023-08-25T09:46:54.914 INFO:teuthology.orchestra.run.smithi071.stdout:(26/154): librgw2-18.0.0-5703.gba6b39e4.el9.x86 3.2 MB/s | 4.7 MB 00:01 2023-08-25T09:46:54.944 INFO:teuthology.orchestra.run.smithi164.stdout:(8/154): ceph-osd-18.0.0-5703.gba6b39e4.el9.x86 8.2 MB/s | 18 MB 00:02 2023-08-25T09:46:54.948 INFO:teuthology.orchestra.run.smithi071.stdout:(27/154): rbd-nbd-18.0.0-5703.gba6b39e4.el9.x86 1.1 MB/s | 169 kB 00:00 2023-08-25T09:46:54.982 INFO:teuthology.orchestra.run.smithi071.stdout:(28/154): ceph-grafana-dashboards-18.0.0-5703.g 352 kB/s | 23 kB 00:00 2023-08-25T09:46:55.003 INFO:teuthology.orchestra.run.smithi164.stdout:(9/154): ceph-selinux-18.0.0-5703.gba6b39e4.el9 421 kB/s | 24 kB 00:00 2023-08-25T09:46:55.024 INFO:teuthology.orchestra.run.smithi071.stdout:(29/154): ceph-mgr-cephadm-18.0.0-5703.gba6b39e 1.7 MB/s | 132 kB 00:00 2023-08-25T09:46:55.465 INFO:teuthology.orchestra.run.smithi071.stdout:(30/154): ceph-mgr-dashboard-18.0.0-5703.gba6b3 3.6 MB/s | 1.7 MB 00:00 2023-08-25T09:46:55.566 INFO:teuthology.orchestra.run.smithi071.stdout:(31/154): ceph-mgr-modules-core-18.0.0-5703.gba 2.4 MB/s | 242 kB 00:00 2023-08-25T09:46:55.649 INFO:teuthology.orchestra.run.smithi071.stdout:(32/154): ceph-mgr-rook-18.0.0-5703.gba6b39e4.e 576 kB/s | 47 kB 00:00 2023-08-25T09:46:55.724 INFO:teuthology.orchestra.run.smithi071.stdout:(33/154): ceph-prometheus-alerts-18.0.0-5703.gb 198 kB/s | 15 kB 00:00 2023-08-25T09:46:55.737 INFO:teuthology.orchestra.run.smithi164.stdout:(10/154): ceph-radosgw-18.0.0-5703.gba6b39e4.el 3.9 MB/s | 8.2 MB 00:02 2023-08-25T09:46:55.791 INFO:teuthology.orchestra.run.smithi071.stdout:(34/154): ceph-volume-18.0.0-5703.gba6b39e4.el9 3.9 MB/s | 261 kB 00:00 2023-08-25T09:46:55.867 INFO:teuthology.orchestra.run.smithi071.stdout:(35/154): cephadm-18.0.0-5703.gba6b39e4.el9.noa 3.0 MB/s | 225 kB 00:00 2023-08-25T09:46:55.934 INFO:teuthology.orchestra.run.smithi071.stdout:(36/154): cephfs-top-18.0.0-5703.gba6b39e4.el9. 275 kB/s | 18 kB 00:00 2023-08-25T09:46:56.037 INFO:teuthology.orchestra.run.smithi164.stdout:(11/154): ceph-common-18.0.0-5703.gba6b39e4.el9 3.9 MB/s | 19 MB 00:04 2023-08-25T09:46:56.087 INFO:teuthology.orchestra.run.smithi164.stdout:(12/154): cephfs-mirror-18.0.0-5703.gba6b39e4.e 624 kB/s | 218 kB 00:00 2023-08-25T09:46:56.100 INFO:teuthology.orchestra.run.smithi071.stdout:(37/154): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2023-08-25T09:46:56.112 INFO:teuthology.orchestra.run.smithi164.stdout:(13/154): libcephfs-devel-18.0.0-5703.gba6b39e4 422 kB/s | 31 kB 00:00 2023-08-25T09:46:56.159 INFO:teuthology.orchestra.run.smithi071.stdout:(38/154): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2023-08-25T09:46:56.201 INFO:teuthology.orchestra.run.smithi071.stdout:(39/154): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2023-08-25T09:46:56.212 INFO:teuthology.orchestra.run.smithi164.stdout:(14/154): libcephsqlite-18.0.0-5703.gba6b39e4.e 1.6 MB/s | 164 kB 00:00 2023-08-25T09:46:56.235 INFO:teuthology.orchestra.run.smithi071.stdout:(40/154): elfutils-libelf-devel-0.189-3.el9.x86 653 kB/s | 21 kB 00:00 2023-08-25T09:46:56.263 INFO:teuthology.orchestra.run.smithi164.stdout:(15/154): libcephfs2-18.0.0-5703.gba6b39e4.el9. 3.7 MB/s | 667 kB 00:00 2023-08-25T09:46:56.276 INFO:teuthology.orchestra.run.smithi071.stdout:(41/154): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 307 kB 00:00 2023-08-25T09:46:56.313 INFO:teuthology.orchestra.run.smithi164.stdout:(16/154): librados-devel-18.0.0-5703.gba6b39e4. 1.2 MB/s | 126 kB 00:00 2023-08-25T09:46:56.430 INFO:teuthology.orchestra.run.smithi164.stdout:(17/154): libradosstriper1-18.0.0-5703.gba6b39e 2.8 MB/s | 473 kB 00:00 2023-08-25T09:46:56.443 INFO:teuthology.orchestra.run.smithi071.stdout:(42/154): ceph-mgr-diskprediction-local-18.0.0- 5.2 MB/s | 7.4 MB 00:01 2023-08-25T09:46:56.461 INFO:teuthology.orchestra.run.smithi071.stdout:(43/154): flexiblas-3.0.4-8.el9.x86_64.rpm 173 kB/s | 32 kB 00:00 2023-08-25T09:46:56.489 INFO:teuthology.orchestra.run.smithi164.stdout:(18/154): python3-ceph-argparse-18.0.0-5703.gba 788 kB/s | 45 kB 00:00 2023-08-25T09:46:56.561 INFO:teuthology.orchestra.run.smithi071.stdout:(44/154): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2023-08-25T09:46:56.572 INFO:teuthology.orchestra.run.smithi164.stdout:(19/154): python3-ceph-common-18.0.0-5703.gba6b 1.5 MB/s | 127 kB 00:00 2023-08-25T09:46:56.577 INFO:teuthology.orchestra.run.smithi071.stdout:(45/154): flexiblas-openblas-openmp-3.0.4-8.el9 174 kB/s | 17 kB 00:00 2023-08-25T09:46:56.621 INFO:teuthology.orchestra.run.smithi071.stdout:(46/154): jq-1.6-15.el9.x86_64.rpm 4.5 MB/s | 186 kB 00:00 2023-08-25T09:46:56.636 INFO:teuthology.orchestra.run.smithi071.stdout:(47/154): libaio-devel-0.3.111-13.el9.x86_64.rp 176 kB/s | 10 kB 00:00 2023-08-25T09:46:56.656 INFO:teuthology.orchestra.run.smithi164.stdout:(20/154): python3-cephfs-18.0.0-5703.gba6b39e4. 1.9 MB/s | 161 kB 00:00 2023-08-25T09:46:56.678 INFO:teuthology.orchestra.run.smithi071.stdout:(48/154): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-08-25T09:46:56.712 INFO:teuthology.orchestra.run.smithi071.stdout:(49/154): librabbitmq-0.11.0-7.el9.x86_64.rpm 598 kB/s | 44 kB 00:00 2023-08-25T09:46:56.772 INFO:teuthology.orchestra.run.smithi071.stdout:(50/154): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-08-25T09:46:56.782 INFO:teuthology.orchestra.run.smithi164.stdout:(21/154): python3-rados-18.0.0-5703.gba6b39e4.e 2.5 MB/s | 320 kB 00:00 2023-08-25T09:46:56.812 INFO:teuthology.orchestra.run.smithi071.stdout:(51/154): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 578 kB 00:00 2023-08-25T09:46:56.846 INFO:teuthology.orchestra.run.smithi071.stdout:(52/154): libuuid-devel-2.37.4-14.el9.x86_64.rp 762 kB/s | 24 kB 00:00 2023-08-25T09:46:56.871 INFO:teuthology.orchestra.run.smithi071.stdout:(53/154): libzstd-devel-1.5.1-2.el9.x86_64.rpm 811 kB/s | 46 kB 00:00 2023-08-25T09:46:56.913 INFO:teuthology.orchestra.run.smithi071.stdout:(54/154): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.3 MB/s | 217 kB 00:00 2023-08-25T09:46:56.915 INFO:teuthology.orchestra.run.smithi164.stdout:(22/154): librgw2-18.0.0-5703.gba6b39e4.el9.x86 7.8 MB/s | 4.7 MB 00:00 2023-08-25T09:46:56.947 INFO:teuthology.orchestra.run.smithi071.stdout:(55/154): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-08-25T09:46:56.965 INFO:teuthology.orchestra.run.smithi164.stdout:(23/154): python3-rbd-18.0.0-5703.gba6b39e4.el9 1.6 MB/s | 297 kB 00:00 2023-08-25T09:46:56.972 INFO:teuthology.orchestra.run.smithi071.stdout:(56/154): openblas-0.3.21-2.el9.x86_64.rpm 622 kB/s | 34 kB 00:00 2023-08-25T09:46:57.016 INFO:teuthology.orchestra.run.smithi164.stdout:(24/154): python3-rgw-18.0.0-5703.gba6b39e4.el9 997 kB/s | 99 kB 00:00 2023-08-25T09:46:57.049 INFO:teuthology.orchestra.run.smithi164.stdout:(25/154): rbd-fuse-18.0.0-5703.gba6b39e4.el9.x8 1.0 MB/s | 85 kB 00:00 2023-08-25T09:46:57.114 INFO:teuthology.orchestra.run.smithi071.stdout:(57/154): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2023-08-25T09:46:57.116 INFO:teuthology.orchestra.run.smithi164.stdout:(26/154): rbd-nbd-18.0.0-5703.gba6b39e4.el9.x86 2.5 MB/s | 169 kB 00:00 2023-08-25T09:46:57.183 INFO:teuthology.orchestra.run.smithi164.stdout:(27/154): ceph-grafana-dashboards-18.0.0-5703.g 354 kB/s | 23 kB 00:00 2023-08-25T09:46:57.215 INFO:teuthology.orchestra.run.smithi071.stdout:(58/154): openssl-devel-3.0.7-24.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2023-08-25T09:46:57.273 INFO:teuthology.orchestra.run.smithi071.stdout:(59/154): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2023-08-25T09:46:57.284 INFO:teuthology.orchestra.run.smithi164.stdout:(28/154): ceph-mgr-cephadm-18.0.0-5703.gba6b39e 1.3 MB/s | 132 kB 00:00 2023-08-25T09:46:57.315 INFO:teuthology.orchestra.run.smithi071.stdout:(60/154): postgresql-contrib-13.11-1.el9.x86_64 9.3 MB/s | 811 kB 00:00 2023-08-25T09:46:57.373 INFO:teuthology.orchestra.run.smithi071.stdout:(61/154): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2023-08-25T09:46:57.392 INFO:teuthology.orchestra.run.smithi164.stdout:(29/154): rbd-mirror-18.0.0-5703.gba6b39e4.el9. 8.1 MB/s | 3.0 MB 00:00 2023-08-25T09:46:57.524 INFO:teuthology.orchestra.run.smithi071.stdout:(62/154): postgresql-server-13.11-1.el9.x86_64. 28 MB/s | 5.7 MB 00:00 2023-08-25T09:46:57.657 INFO:teuthology.orchestra.run.smithi071.stdout:(63/154): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-08-25T09:46:57.674 INFO:teuthology.orchestra.run.smithi071.stdout:(64/154): python3-jinja2-2.11.3-4.el9.noarch.rp 1.7 MB/s | 228 kB 00:00 2023-08-25T09:46:57.716 INFO:teuthology.orchestra.run.smithi071.stdout:(65/154): python3-jsonpointer-2.0-4.el9.noarch. 450 kB/s | 18 kB 00:00 2023-08-25T09:46:57.717 INFO:teuthology.orchestra.run.smithi164.stdout:(30/154): ceph-mgr-dashboard-18.0.0-5703.gba6b3 4.1 MB/s | 1.7 MB 00:00 2023-08-25T09:46:57.741 INFO:teuthology.orchestra.run.smithi071.stdout:(66/154): python3-jsonpatch-1.21-16.el9.noarch. 372 kB/s | 24 kB 00:00 2023-08-25T09:46:57.758 INFO:teuthology.orchestra.run.smithi071.stdout:(67/154): python3-mako-1.1.4-6.el9.noarch.rpm 3.5 MB/s | 144 kB 00:00 2023-08-25T09:46:57.783 INFO:teuthology.orchestra.run.smithi071.stdout:(68/154): python3-markupsafe-1.1.1-12.el9.x86_6 799 kB/s | 32 kB 00:00 2023-08-25T09:46:57.842 INFO:teuthology.orchestra.run.smithi164.stdout:(31/154): ceph-mgr-modules-core-18.0.0-5703.gba 1.9 MB/s | 242 kB 00:00 2023-08-25T09:46:57.901 INFO:teuthology.orchestra.run.smithi164.stdout:(32/154): ceph-mgr-rook-18.0.0-5703.gba6b39e4.e 829 kB/s | 47 kB 00:00 2023-08-25T09:46:57.925 INFO:teuthology.orchestra.run.smithi071.stdout:(69/154): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2023-08-25T09:46:57.950 INFO:teuthology.orchestra.run.smithi071.stdout:(70/154): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2023-08-25T09:46:57.959 INFO:teuthology.orchestra.run.smithi164.stdout:(33/154): ceph-prometheus-alerts-18.0.0-5703.gb 256 kB/s | 15 kB 00:00 2023-08-25T09:46:57.992 INFO:teuthology.orchestra.run.smithi071.stdout:(71/154): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2023-08-25T09:46:58.017 INFO:teuthology.orchestra.run.smithi071.stdout:(72/154): python3-prettytable-0.7.2-27.el9.noar 631 kB/s | 41 kB 00:00 2023-08-25T09:46:58.059 INFO:teuthology.orchestra.run.smithi071.stdout:(73/154): python3-pyasn1-modules-0.4.8-6.el9.no 5.1 MB/s | 211 kB 00:00 2023-08-25T09:46:58.076 INFO:teuthology.orchestra.run.smithi164.stdout:(34/154): ceph-volume-18.0.0-5703.gba6b39e4.el9 2.2 MB/s | 261 kB 00:00 2023-08-25T09:46:58.085 INFO:teuthology.orchestra.run.smithi071.stdout:(74/154): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2023-08-25T09:46:58.110 INFO:teuthology.orchestra.run.smithi071.stdout:(75/154): python3-pytz-2021.1-5.el9.noarch.rpm 987 kB/s | 47 kB 00:00 2023-08-25T09:46:58.135 INFO:teuthology.orchestra.run.smithi071.stdout:(76/154): python3-requests-oauthlib-1.3.0-12.el 878 kB/s | 43 kB 00:00 2023-08-25T09:46:58.219 INFO:teuthology.orchestra.run.smithi164.stdout:(35/154): ceph-mgr-diskprediction-local-18.0.0- 9.0 MB/s | 7.4 MB 00:00 2023-08-25T09:46:58.260 INFO:teuthology.orchestra.run.smithi164.stdout:(36/154): cephadm-18.0.0-5703.gba6b39e4.el9.noa 1.2 MB/s | 225 kB 00:00 2023-08-25T09:46:58.286 INFO:teuthology.orchestra.run.smithi164.stdout:(37/154): cephfs-top-18.0.0-5703.gba6b39e4.el9. 275 kB/s | 18 kB 00:00 2023-08-25T09:46:58.477 INFO:teuthology.orchestra.run.smithi071.stdout:(77/154): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2023-08-25T09:46:58.502 INFO:teuthology.orchestra.run.smithi071.stdout:(78/154): python3-toml-0.10.2-6.el9.noarch.rpm 111 kB/s | 39 kB 00:00 2023-08-25T09:46:58.511 INFO:teuthology.orchestra.run.smithi164.stdout:(38/154): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2023-08-25T09:46:58.544 INFO:teuthology.orchestra.run.smithi071.stdout:(79/154): socat-1.7.4.1-5.el9.x86_64.rpm 7.3 MB/s | 300 kB 00:00 2023-08-25T09:46:58.570 INFO:teuthology.orchestra.run.smithi071.stdout:(80/154): uuid-1.6.2-55.el9.x86_64.rpm 861 kB/s | 56 kB 00:00 2023-08-25T09:46:58.586 INFO:teuthology.orchestra.run.smithi164.stdout:(39/154): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 662 kB 00:00 2023-08-25T09:46:58.595 INFO:teuthology.orchestra.run.smithi071.stdout:(81/154): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.7 MB/s | 85 kB 00:00 2023-08-25T09:46:58.621 INFO:teuthology.orchestra.run.smithi164.stdout:(40/154): boost-program-options-1.75.0-8.el9.x8 986 kB/s | 106 kB 00:00 2023-08-25T09:46:58.621 INFO:teuthology.orchestra.run.smithi071.stdout:(82/154): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2023-08-25T09:46:58.637 INFO:teuthology.orchestra.run.smithi164.stdout:(41/154): elfutils-libelf-devel-0.189-3.el9.x86 434 kB/s | 21 kB 00:00 2023-08-25T09:46:58.648 INFO:teuthology.orchestra.run.smithi071.stdout:(83/154): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2023-08-25T09:46:58.679 INFO:teuthology.orchestra.run.smithi164.stdout:(42/154): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2023-08-25T09:46:58.689 INFO:teuthology.orchestra.run.smithi071.stdout:(84/154): libgfortran-11.4.1-2.1.el9.x86_64.rpm 12 MB/s | 807 kB 00:00 2023-08-25T09:46:58.704 INFO:teuthology.orchestra.run.smithi164.stdout:(43/154): flexiblas-3.0.4-8.el9.x86_64.rpm 503 kB/s | 32 kB 00:00 2023-08-25T09:46:58.715 INFO:teuthology.orchestra.run.smithi071.stdout:(85/154): libquadmath-11.4.1-2.1.el9.x86_64.rpm 3.3 MB/s | 197 kB 00:00 2023-08-25T09:46:58.748 INFO:teuthology.orchestra.run.smithi071.stdout:(86/154): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 241 kB 00:00 2023-08-25T09:46:58.799 INFO:teuthology.orchestra.run.smithi071.stdout:(87/154): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2023-08-25T09:46:58.813 INFO:teuthology.orchestra.run.smithi164.stdout:(44/154): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2023-08-25T09:46:58.828 INFO:teuthology.orchestra.run.smithi071.stdout:(88/154): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-08-25T09:46:58.839 INFO:teuthology.orchestra.run.smithi164.stdout:(45/154): flexiblas-openblas-openmp-3.0.4-8.el9 149 kB/s | 17 kB 00:00 2023-08-25T09:46:58.858 INFO:teuthology.orchestra.run.smithi071.stdout:(89/154): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2023-08-25T09:46:58.872 INFO:teuthology.orchestra.run.smithi164.stdout:(46/154): libaio-devel-0.3.111-13.el9.x86_64.rp 315 kB/s | 10 kB 00:00 2023-08-25T09:46:58.914 INFO:teuthology.orchestra.run.smithi164.stdout:(47/154): jq-1.6-15.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2023-08-25T09:46:58.955 INFO:teuthology.orchestra.run.smithi164.stdout:(48/154): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2023-08-25T09:46:58.981 INFO:teuthology.orchestra.run.smithi164.stdout:(49/154): librabbitmq-0.11.0-7.el9.x86_64.rpm 687 kB/s | 44 kB 00:00 2023-08-25T09:46:59.039 INFO:teuthology.orchestra.run.smithi164.stdout:(50/154): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2023-08-25T09:46:59.090 INFO:teuthology.orchestra.run.smithi164.stdout:(51/154): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2023-08-25T09:46:59.115 INFO:teuthology.orchestra.run.smithi164.stdout:(52/154): libuuid-devel-2.37.4-14.el9.x86_64.rp 357 kB/s | 24 kB 00:00 2023-08-25T09:46:59.140 INFO:teuthology.orchestra.run.smithi164.stdout:(53/154): libzstd-devel-1.5.1-2.el9.x86_64.rpm 939 kB/s | 46 kB 00:00 2023-08-25T09:46:59.191 INFO:teuthology.orchestra.run.smithi164.stdout:(54/154): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-08-25T09:46:59.241 INFO:teuthology.orchestra.run.smithi164.stdout:(55/154): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.2 MB/s | 217 kB 00:00 2023-08-25T09:46:59.266 INFO:teuthology.orchestra.run.smithi164.stdout:(56/154): openblas-0.3.21-2.el9.x86_64.rpm 473 kB/s | 34 kB 00:00 2023-08-25T09:46:59.342 INFO:teuthology.orchestra.run.smithi071.stdout:(90/154): ceph-test-18.0.0-5703.gba6b39e4.el9.x 5.2 MB/s | 40 MB 00:07 2023-08-25T09:46:59.425 INFO:teuthology.orchestra.run.smithi164.stdout:(57/154): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2023-08-25T09:46:59.442 INFO:teuthology.orchestra.run.smithi071.stdout:(91/154): zlib-devel-1.2.11-36.el9.x86_64.rpm 77 kB/s | 45 kB 00:00 2023-08-25T09:46:59.525 INFO:teuthology.orchestra.run.smithi071.stdout:(92/154): python3-devel-3.9.16-1.el9.x86_64.rpm 356 kB/s | 247 kB 00:00 2023-08-25T09:46:59.526 INFO:teuthology.orchestra.run.smithi164.stdout:(58/154): openssl-devel-3.0.7-24.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-08-25T09:46:59.609 INFO:teuthology.orchestra.run.smithi164.stdout:(59/154): postgresql-13.11-1.el9.x86_64.rpm 9.0 MB/s | 1.5 MB 00:00 2023-08-25T09:46:59.668 INFO:teuthology.orchestra.run.smithi164.stdout:(60/154): postgresql-contrib-13.11-1.el9.x86_64 6.3 MB/s | 811 kB 00:00 2023-08-25T09:46:59.701 INFO:teuthology.orchestra.run.smithi164.stdout:(61/154): postgresql-private-libs-13.11-1.el9.x 1.6 MB/s | 134 kB 00:00 2023-08-25T09:46:59.767 INFO:teuthology.orchestra.run.smithi071.stdout:(93/154): fmt-8.1.1-5.el9.x86_64.rpm 261 kB/s | 111 kB 00:00 2023-08-25T09:46:59.859 INFO:teuthology.orchestra.run.smithi071.stdout:(94/154): libarrow-doc-9.0.0-9.el9.noarch.rpm 268 kB/s | 24 kB 00:00 2023-08-25T09:46:59.893 INFO:teuthology.orchestra.run.smithi164.stdout:(62/154): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2023-08-25T09:46:59.951 INFO:teuthology.orchestra.run.smithi071.stdout:(95/154): liboath-2.6.7-2.el9.x86_64.rpm 535 kB/s | 49 kB 00:00 2023-08-25T09:46:59.985 INFO:teuthology.orchestra.run.smithi071.stdout:(96/154): gperftools-libs-2.9.1-2.el9.x86_64.rp 570 kB/s | 309 kB 00:00 2023-08-25T09:47:00.043 INFO:teuthology.orchestra.run.smithi071.stdout:(97/154): libunwind-1.6.2-1.el9.x86_64.rpm 738 kB/s | 67 kB 00:00 2023-08-25T09:47:00.052 INFO:teuthology.orchestra.run.smithi164.stdout:(63/154): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-08-25T09:47:00.085 INFO:teuthology.orchestra.run.smithi164.stdout:(64/154): python3-jinja2-2.11.3-4.el9.noarch.rp 1.3 MB/s | 228 kB 00:00 2023-08-25T09:47:00.119 INFO:teuthology.orchestra.run.smithi164.stdout:(65/154): python3-jsonpatch-1.21-16.el9.noarch. 741 kB/s | 24 kB 00:00 2023-08-25T09:47:00.145 INFO:teuthology.orchestra.run.smithi164.stdout:(66/154): python3-jsonpointer-2.0-4.el9.noarch. 312 kB/s | 18 kB 00:00 2023-08-25T09:47:00.169 INFO:teuthology.orchestra.run.smithi071.stdout:(98/154): parquet-libs-9.0.0-9.el9.x86_64.rpm 4.5 MB/s | 837 kB 00:00 2023-08-25T09:47:00.186 INFO:teuthology.orchestra.run.smithi164.stdout:(67/154): python3-markupsafe-1.1.1-12.el9.x86_6 784 kB/s | 32 kB 00:00 2023-08-25T09:47:00.228 INFO:teuthology.orchestra.run.smithi164.stdout:(68/154): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2023-08-25T09:47:00.277 INFO:teuthology.orchestra.run.smithi071.stdout:(99/154): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 404 kB/s | 43 kB 00:00 2023-08-25T09:47:00.327 INFO:teuthology.orchestra.run.smithi071.stdout:(100/154): python3-asyncssh-2.13.2-3.el9.noarch 1.9 MB/s | 548 kB 00:00 2023-08-25T09:47:00.378 INFO:teuthology.orchestra.run.smithi071.stdout:(101/154): python3-cachetools-4.2.4-1.el9.noarc 324 kB/s | 32 kB 00:00 2023-08-25T09:47:00.378 INFO:teuthology.orchestra.run.smithi164.stdout:(69/154): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-08-25T09:47:00.430 INFO:teuthology.orchestra.run.smithi164.stdout:(70/154): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2023-08-25T09:47:00.470 INFO:teuthology.orchestra.run.smithi164.stdout:(71/154): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 174 kB 00:00 2023-08-25T09:47:00.478 INFO:teuthology.orchestra.run.smithi071.stdout:(102/154): libarrow-9.0.0-9.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2023-08-25T09:47:00.503 INFO:teuthology.orchestra.run.smithi071.stdout:(103/154): python3-certifi-2021.10.8-2.el9.noar 86 kB/s | 15 kB 00:00 2023-08-25T09:47:00.504 INFO:teuthology.orchestra.run.smithi164.stdout:(72/154): python3-prettytable-0.7.2-27.el9.noar 567 kB/s | 41 kB 00:00 2023-08-25T09:47:00.537 INFO:teuthology.orchestra.run.smithi071.stdout:(104/154): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2023-08-25T09:47:00.571 INFO:teuthology.orchestra.run.smithi164.stdout:(73/154): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2023-08-25T09:47:00.587 INFO:teuthology.orchestra.run.smithi071.stdout:(105/154): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2023-08-25T09:47:00.605 INFO:teuthology.orchestra.run.smithi164.stdout:(74/154): python3-pyasn1-0.4.8-6.el9.noarch.rpm 994 kB/s | 132 kB 00:00 2023-08-25T09:47:00.620 INFO:teuthology.orchestra.run.smithi071.stdout:(106/154): python3-google-auth-2.22.0-1.el9.noa 1.6 MB/s | 197 kB 00:00 2023-08-25T09:47:00.630 INFO:teuthology.orchestra.run.smithi164.stdout:(75/154): python3-pytz-2021.1-5.el9.noarch.rpm 840 kB/s | 47 kB 00:00 2023-08-25T09:47:00.646 INFO:teuthology.orchestra.run.smithi071.stdout:(107/154): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2023-08-25T09:47:00.664 INFO:teuthology.orchestra.run.smithi164.stdout:(76/154): python3-requests-oauthlib-1.3.0-12.el 739 kB/s | 43 kB 00:00 2023-08-25T09:47:00.688 INFO:teuthology.orchestra.run.smithi071.stdout:(108/154): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2023-08-25T09:47:00.705 INFO:teuthology.orchestra.run.smithi164.stdout:(77/154): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 39 kB 00:00 2023-08-25T09:47:00.721 INFO:teuthology.orchestra.run.smithi071.stdout:(109/154): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2023-08-25T09:47:00.763 INFO:teuthology.orchestra.run.smithi071.stdout:(110/154): python3-jaraco-functools-3.5.0-2.el9 167 kB/s | 19 kB 00:00 2023-08-25T09:47:00.805 INFO:teuthology.orchestra.run.smithi071.stdout:(111/154): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2023-08-25T09:47:00.847 INFO:teuthology.orchestra.run.smithi071.stdout:(112/154): python3-jwt+crypto-2.4.0-1.el9.noarc 72 kB/s | 9.0 kB 00:00 2023-08-25T09:47:00.873 INFO:teuthology.orchestra.run.smithi071.stdout:(113/154): python3-jwt-2.4.0-1.el9.noarch.rpm 377 kB/s | 41 kB 00:00 2023-08-25T09:47:00.948 INFO:teuthology.orchestra.run.smithi071.stdout:(114/154): python3-logutils-0.3.5-21.el9.noarch 466 kB/s | 46 kB 00:00 2023-08-25T09:47:00.982 INFO:teuthology.orchestra.run.smithi071.stdout:(115/154): python3-more-itertools-8.12.0-2.el9. 790 kB/s | 79 kB 00:00 2023-08-25T09:47:01.024 INFO:teuthology.orchestra.run.smithi071.stdout:(116/154): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2023-08-25T09:47:01.049 INFO:teuthology.orchestra.run.smithi071.stdout:(117/154): python3-natsort-7.1.1-5.el9.noarch.r 579 kB/s | 58 kB 00:00 2023-08-25T09:47:01.082 INFO:teuthology.orchestra.run.smithi071.stdout:(118/154): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2023-08-25T09:47:01.125 INFO:teuthology.orchestra.run.smithi071.stdout:(119/154): python3-portend-3.1.0-2.el9.noarch.r 165 kB/s | 16 kB 00:00 2023-08-25T09:47:01.158 INFO:teuthology.orchestra.run.smithi071.stdout:(120/154): python3-pyOpenSSL-21.0.0-1.el9.noarc 831 kB/s | 90 kB 00:00 2023-08-25T09:47:01.191 INFO:teuthology.orchestra.run.smithi071.stdout:(121/154): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2023-08-25T09:47:01.223 INFO:teuthology.orchestra.run.smithi164.stdout:(78/154): ceph-test-18.0.0-5703.gba6b39e4.el9.x 6.5 MB/s | 40 MB 00:06 2023-08-25T09:47:01.234 INFO:teuthology.orchestra.run.smithi071.stdout:(122/154): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2023-08-25T09:47:01.365 INFO:teuthology.orchestra.run.smithi164.stdout:(79/154): socat-1.7.4.1-5.el9.x86_64.rpm 468 kB/s | 300 kB 00:00 2023-08-25T09:47:01.397 INFO:teuthology.orchestra.run.smithi071.stdout:(123/154): python3-rsa-4.9-2.el9.noarch.rpm 247 kB/s | 59 kB 00:00 2023-08-25T09:47:01.426 INFO:teuthology.orchestra.run.smithi071.stdout:(124/154): python3-tempora-5.0.0-2.el9.noarch.r 153 kB/s | 36 kB 00:00 2023-08-25T09:47:01.451 INFO:teuthology.orchestra.run.smithi071.stdout:(125/154): python3-typing-extensions-4.4.0-2.el 235 kB/s | 51 kB 00:00 2023-08-25T09:47:01.501 INFO:teuthology.orchestra.run.smithi071.stdout:(126/154): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-08-25T09:47:01.518 INFO:teuthology.orchestra.run.smithi071.stdout:(127/154): python3-websocket-client-1.2.3-2.el9 980 kB/s | 90 kB 00:00 2023-08-25T09:47:01.569 INFO:teuthology.orchestra.run.smithi071.stdout:(128/154): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2023-08-25T09:47:01.611 INFO:teuthology.orchestra.run.smithi071.stdout:(129/154): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2023-08-25T09:47:01.636 INFO:teuthology.orchestra.run.smithi071.stdout:(130/154): re2-20211101-2.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2023-08-25T09:47:01.674 INFO:teuthology.orchestra.run.smithi164.stdout:(80/154): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2023-08-25T09:47:01.699 INFO:teuthology.orchestra.run.smithi164.stdout:(81/154): uuid-1.6.2-55.el9.x86_64.rpm 123 kB/s | 56 kB 00:00 2023-08-25T09:47:01.728 INFO:teuthology.orchestra.run.smithi071.stdout:(131/154): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2023-08-25T09:47:01.732 INFO:teuthology.orchestra.run.smithi164.stdout:(82/154): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 244 kB/s | 85 kB 00:00 2023-08-25T09:47:01.757 INFO:teuthology.orchestra.run.smithi164.stdout:(83/154): xmlstarlet-1.6.1-20.el9.x86_64.rpm 765 kB/s | 63 kB 00:00 2023-08-25T09:47:01.825 INFO:teuthology.orchestra.run.smithi164.stdout:(84/154): fuse-2.9.9-15.el9.x86_64.rpm 872 kB/s | 79 kB 00:00 2023-08-25T09:47:01.862 INFO:teuthology.orchestra.run.smithi071.stdout:(132/154): NetworkManager-1.44.0-2.el9.x86_64.r 17 MB/s | 2.2 MB 00:00 2023-08-25T09:47:01.883 INFO:teuthology.orchestra.run.smithi164.stdout:(85/154): libgfortran-11.4.1-2.1.el9.x86_64.rpm 5.3 MB/s | 807 kB 00:00 2023-08-25T09:47:01.933 INFO:teuthology.orchestra.run.smithi164.stdout:(86/154): libquadmath-11.4.1-2.1.el9.x86_64.rpm 1.1 MB/s | 197 kB 00:00 2023-08-25T09:47:01.967 INFO:teuthology.orchestra.run.smithi164.stdout:(87/154): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 241 kB 00:00 2023-08-25T09:47:01.970 INFO:teuthology.orchestra.run.smithi071.stdout:(133/154): NetworkManager-libnm-1.44.0-2.el9.x8 17 MB/s | 1.8 MB 00:00 2023-08-25T09:47:02.012 INFO:teuthology.orchestra.run.smithi071.stdout:(134/154): NetworkManager-team-1.44.0-2.el9.x86 1.0 MB/s | 41 kB 00:00 2023-08-25T09:47:02.042 INFO:teuthology.orchestra.run.smithi164.stdout:(88/154): python3-cryptography-36.0.1-4.el9.x86 7.3 MB/s | 1.1 MB 00:00 2023-08-25T09:47:02.084 INFO:teuthology.orchestra.run.smithi164.stdout:(89/154): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2023-08-25T09:47:02.126 INFO:teuthology.orchestra.run.smithi164.stdout:(90/154): python3-ply-3.11-14.el9.noarch.rpm 663 kB/s | 103 kB 00:00 2023-08-25T09:47:02.129 INFO:teuthology.orchestra.run.smithi071.stdout:(135/154): librbd1-18.0.0-5703.gba6b39e4.el9.x8 6.2 MB/s | 3.0 MB 00:00 2023-08-25T09:47:02.162 INFO:teuthology.orchestra.run.smithi071.stdout:(136/154): NetworkManager-tui-1.44.0-2.el9.x86_ 1.6 MB/s | 245 kB 00:00 2023-08-25T09:47:02.179 INFO:teuthology.orchestra.run.smithi071.stdout:(137/154): elfutils-debuginfod-client-0.189-3.e 766 kB/s | 36 kB 00:00 2023-08-25T09:47:02.247 INFO:teuthology.orchestra.run.smithi071.stdout:(138/154): elfutils-libelf-0.189-3.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2023-08-25T09:47:02.280 INFO:teuthology.orchestra.run.smithi071.stdout:(139/154): elfutils-libs-0.189-3.el9.x86_64.rpm 2.6 MB/s | 254 kB 00:00 2023-08-25T09:47:02.321 INFO:teuthology.orchestra.run.smithi071.stdout:(140/154): iproute-6.2.0-5.el9.x86_64.rpm 11 MB/s | 785 kB 00:00 2023-08-25T09:47:02.356 INFO:teuthology.orchestra.run.smithi071.stdout:(141/154): iproute-tc-6.2.0-5.el9.x86_64.rpm 6.6 MB/s | 448 kB 00:00 2023-08-25T09:47:02.360 INFO:teuthology.orchestra.run.smithi164.stdout:(91/154): zlib-devel-1.2.11-36.el9.x86_64.rpm 165 kB/s | 45 kB 00:00 2023-08-25T09:47:02.381 INFO:teuthology.orchestra.run.smithi071.stdout:(142/154): libblkid-2.37.4-14.el9.x86_64.rpm 1.9 MB/s | 107 kB 00:00 2023-08-25T09:47:02.401 INFO:teuthology.orchestra.run.smithi164.stdout:(92/154): python3-devel-3.9.16-1.el9.x86_64.rpm 692 kB/s | 247 kB 00:00 2023-08-25T09:47:02.414 INFO:teuthology.orchestra.run.smithi071.stdout:(143/154): libfdisk-2.37.4-14.el9.x86_64.rpm 2.7 MB/s | 154 kB 00:00 2023-08-25T09:47:02.439 INFO:teuthology.orchestra.run.smithi071.stdout:(144/154): libmount-2.37.4-14.el9.x86_64.rpm 2.3 MB/s | 133 kB 00:00 2023-08-25T09:47:02.473 INFO:teuthology.orchestra.run.smithi071.stdout:(145/154): libsmartcols-2.37.4-14.el9.x86_64.rp 1.1 MB/s | 63 kB 00:00 2023-08-25T09:47:02.498 INFO:teuthology.orchestra.run.smithi071.stdout:(146/154): libuuid-2.37.4-14.el9.x86_64.rpm 480 kB/s | 28 kB 00:00 2023-08-25T09:47:02.557 INFO:teuthology.orchestra.run.smithi071.stdout:(147/154): openssl-3.0.7-24.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2023-08-25T09:47:02.632 INFO:teuthology.orchestra.run.smithi071.stdout:(148/154): openssl-libs-3.0.7-24.el9.x86_64.rpm 16 MB/s | 2.1 MB 00:00 2023-08-25T09:47:02.716 INFO:teuthology.orchestra.run.smithi071.stdout:(149/154): util-linux-2.37.4-14.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2023-08-25T09:47:02.750 INFO:teuthology.orchestra.run.smithi071.stdout:(150/154): util-linux-core-2.37.4-14.el9.x86_64 4.3 MB/s | 433 kB 00:00 2023-08-25T09:47:02.760 INFO:teuthology.orchestra.run.smithi164.stdout:(93/154): fmt-8.1.1-5.el9.x86_64.rpm 214 kB/s | 111 kB 00:00 2023-08-25T09:47:02.775 INFO:teuthology.orchestra.run.smithi071.stdout:(151/154): util-linux-user-2.37.4-14.el9.x86_64 562 kB/s | 30 kB 00:00 2023-08-25T09:47:02.808 INFO:teuthology.orchestra.run.smithi071.stdout:(152/154): xfsdump-3.1.12-3.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2023-08-25T09:47:02.867 INFO:teuthology.orchestra.run.smithi071.stdout:(153/154): xfsprogs-5.19.0-4.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2023-08-25T09:47:02.877 INFO:teuthology.orchestra.run.smithi164.stdout:(94/154): libarrow-doc-9.0.0-9.el9.noarch.rpm 210 kB/s | 24 kB 00:00 2023-08-25T09:47:02.959 INFO:teuthology.orchestra.run.smithi071.stdout:(154/154): librados2-18.0.0-5703.gba6b39e4.el9. 2.4 MB/s | 3.3 MB 00:01 2023-08-25T09:47:02.967 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2023-08-25T09:47:02.967 INFO:teuthology.orchestra.run.smithi071.stdout:Total 15 MB/s | 206 MB 00:13 2023-08-25T09:47:02.986 INFO:teuthology.orchestra.run.smithi164.stdout:(95/154): liboath-2.6.7-2.el9.x86_64.rpm 451 kB/s | 49 kB 00:00 2023-08-25T09:47:03.028 INFO:teuthology.orchestra.run.smithi164.stdout:(96/154): gperftools-libs-2.9.1-2.el9.x86_64.rp 463 kB/s | 309 kB 00:00 2023-08-25T09:47:03.063 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2023-08-25T09:47:03.103 INFO:teuthology.orchestra.run.smithi164.stdout:(97/154): libunwind-1.6.2-1.el9.x86_64.rpm 580 kB/s | 67 kB 00:00 2023-08-25T09:47:03.236 INFO:teuthology.orchestra.run.smithi164.stdout:(98/154): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.9 MB/s | 837 kB 00:00 2023-08-25T09:47:03.362 INFO:teuthology.orchestra.run.smithi164.stdout:(99/154): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 349 kB/s | 43 kB 00:00 2023-08-25T09:47:03.376 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2023-08-25T09:47:03.377 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2023-08-25T09:47:03.471 INFO:teuthology.orchestra.run.smithi164.stdout:(100/154): libarrow-9.0.0-9.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2023-08-25T09:47:03.521 INFO:teuthology.orchestra.run.smithi164.stdout:(101/154): python3-asyncssh-2.13.2-3.el9.noarch 1.3 MB/s | 548 kB 00:00 2023-08-25T09:47:03.546 INFO:teuthology.orchestra.run.smithi164.stdout:(102/154): python3-cachetools-4.2.4-1.el9.noarc 176 kB/s | 32 kB 00:00 2023-08-25T09:47:03.579 INFO:teuthology.orchestra.run.smithi164.stdout:(103/154): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2023-08-25T09:47:03.646 INFO:teuthology.orchestra.run.smithi164.stdout:(104/154): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2023-08-25T09:47:03.688 INFO:teuthology.orchestra.run.smithi164.stdout:(105/154): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2023-08-25T09:47:03.756 INFO:teuthology.orchestra.run.smithi164.stdout:(106/154): python3-google-auth-2.22.0-1.el9.noa 1.1 MB/s | 197 kB 00:00 2023-08-25T09:47:03.780 INFO:teuthology.orchestra.run.smithi164.stdout:(107/154): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2023-08-25T09:47:03.806 INFO:teuthology.orchestra.run.smithi164.stdout:(108/154): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2023-08-25T09:47:03.864 INFO:teuthology.orchestra.run.smithi164.stdout:(109/154): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2023-08-25T09:47:03.890 INFO:teuthology.orchestra.run.smithi164.stdout:(110/154): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2023-08-25T09:47:03.907 INFO:teuthology.orchestra.run.smithi164.stdout:(111/154): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2023-08-25T09:47:03.967 INFO:teuthology.orchestra.run.smithi164.stdout:(112/154): python3-jwt+crypto-2.4.0-1.el9.noarc 91 kB/s | 9.0 kB 00:00 2023-08-25T09:47:03.991 INFO:teuthology.orchestra.run.smithi164.stdout:(113/154): python3-jwt-2.4.0-1.el9.noarch.rpm 409 kB/s | 41 kB 00:00 2023-08-25T09:47:04.058 INFO:teuthology.orchestra.run.smithi164.stdout:(114/154): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2023-08-25T09:47:04.074 INFO:teuthology.orchestra.run.smithi164.stdout:(115/154): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2023-08-25T09:47:04.108 INFO:teuthology.orchestra.run.smithi164.stdout:(116/154): python3-more-itertools-8.12.0-2.el9. 676 kB/s | 79 kB 00:00 2023-08-25T09:47:04.167 INFO:teuthology.orchestra.run.smithi164.stdout:(117/154): python3-natsort-7.1.1-5.el9.noarch.r 534 kB/s | 58 kB 00:00 2023-08-25T09:47:04.217 INFO:teuthology.orchestra.run.smithi164.stdout:(118/154): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-08-25T09:47:04.250 INFO:teuthology.orchestra.run.smithi164.stdout:(119/154): python3-portend-3.1.0-2.el9.noarch.r 116 kB/s | 16 kB 00:00 2023-08-25T09:47:04.292 INFO:teuthology.orchestra.run.smithi164.stdout:(120/154): python3-pyOpenSSL-21.0.0-1.el9.noarc 721 kB/s | 90 kB 00:00 2023-08-25T09:47:04.326 INFO:teuthology.orchestra.run.smithi164.stdout:(121/154): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2023-08-25T09:47:04.376 INFO:teuthology.orchestra.run.smithi164.stdout:(122/154): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2023-08-25T09:47:04.402 INFO:teuthology.orchestra.run.smithi164.stdout:(123/154): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2023-08-25T09:47:04.452 INFO:teuthology.orchestra.run.smithi164.stdout:(124/154): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2023-08-25T09:47:04.485 INFO:teuthology.orchestra.run.smithi164.stdout:(125/154): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2023-08-25T09:47:04.527 INFO:teuthology.orchestra.run.smithi164.stdout:(126/154): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2023-08-25T09:47:04.570 INFO:teuthology.orchestra.run.smithi164.stdout:(127/154): python3-websocket-client-1.2.3-2.el9 769 kB/s | 90 kB 00:00 2023-08-25T09:47:04.629 INFO:teuthology.orchestra.run.smithi164.stdout:(128/154): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2023-08-25T09:47:04.653 INFO:teuthology.orchestra.run.smithi164.stdout:(129/154): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2023-08-25T09:47:04.695 INFO:teuthology.orchestra.run.smithi164.stdout:(130/154): re2-20211101-2.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2023-08-25T09:47:05.054 INFO:teuthology.orchestra.run.smithi164.stdout:(131/154): librbd1-18.0.0-5703.gba6b39e4.el9.x8 8.5 MB/s | 3.0 MB 00:00 2023-08-25T09:47:05.197 INFO:teuthology.orchestra.run.smithi164.stdout:(132/154): librados2-18.0.0-5703.gba6b39e4.el9. 6.1 MB/s | 3.3 MB 00:00 2023-08-25T09:47:05.281 INFO:teuthology.orchestra.run.smithi164.stdout:(133/154): NetworkManager-1.44.0-2.el9.x86_64.r 10 MB/s | 2.2 MB 00:00 2023-08-25T09:47:05.382 INFO:teuthology.orchestra.run.smithi164.stdout:(134/154): NetworkManager-libnm-1.44.0-2.el9.x8 11 MB/s | 1.8 MB 00:00 2023-08-25T09:47:05.406 INFO:teuthology.orchestra.run.smithi164.stdout:(135/154): NetworkManager-team-1.44.0-2.el9.x86 380 kB/s | 41 kB 00:00 2023-08-25T09:47:05.472 INFO:teuthology.orchestra.run.smithi164.stdout:(136/154): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2023-08-25T09:47:05.515 INFO:teuthology.orchestra.run.smithi164.stdout:(137/154): NetworkManager-tui-1.44.0-2.el9.x86_ 1.8 MB/s | 245 kB 00:00 2023-08-25T09:47:05.516 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2023-08-25T09:47:05.516 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2023-08-25T09:47:05.540 INFO:teuthology.orchestra.run.smithi164.stdout:(138/154): elfutils-debuginfod-client-0.189-3.e 269 kB/s | 36 kB 00:00 2023-08-25T09:47:05.582 INFO:teuthology.orchestra.run.smithi164.stdout:(139/154): elfutils-libelf-0.189-3.el9.x86_64.r 1.8 MB/s | 192 kB 00:00 2023-08-25T09:47:05.632 INFO:teuthology.orchestra.run.smithi164.stdout:(140/154): elfutils-libs-0.189-3.el9.x86_64.rpm 2.8 MB/s | 254 kB 00:00 2023-08-25T09:47:05.699 INFO:teuthology.orchestra.run.smithi164.stdout:(141/154): iproute-6.2.0-5.el9.x86_64.rpm 4.9 MB/s | 785 kB 00:00 2023-08-25T09:47:05.741 INFO:teuthology.orchestra.run.smithi164.stdout:(142/154): iproute-tc-6.2.0-5.el9.x86_64.rpm 2.8 MB/s | 448 kB 00:00 2023-08-25T09:47:05.774 INFO:teuthology.orchestra.run.smithi164.stdout:(143/154): libblkid-2.37.4-14.el9.x86_64.rpm 822 kB/s | 107 kB 00:00 2023-08-25T09:47:05.816 INFO:teuthology.orchestra.run.smithi164.stdout:(144/154): libfdisk-2.37.4-14.el9.x86_64.rpm 1.3 MB/s | 154 kB 00:00 2023-08-25T09:47:05.841 INFO:teuthology.orchestra.run.smithi164.stdout:(145/154): libsmartcols-2.37.4-14.el9.x86_64.rp 979 kB/s | 63 kB 00:00 2023-08-25T09:47:05.875 INFO:teuthology.orchestra.run.smithi164.stdout:(146/154): libmount-2.37.4-14.el9.x86_64.rpm 1.3 MB/s | 133 kB 00:00 2023-08-25T09:47:05.900 INFO:teuthology.orchestra.run.smithi164.stdout:(147/154): libuuid-2.37.4-14.el9.x86_64.rpm 335 kB/s | 28 kB 00:00 2023-08-25T09:47:05.976 INFO:teuthology.orchestra.run.smithi164.stdout:(148/154): openssl-3.0.7-24.el9.x86_64.rpm 8.7 MB/s | 1.2 MB 00:00 2023-08-25T09:47:06.109 INFO:teuthology.orchestra.run.smithi164.stdout:(149/154): openssl-libs-3.0.7-24.el9.x86_64.rpm 9.2 MB/s | 2.1 MB 00:00 2023-08-25T09:47:06.209 INFO:teuthology.orchestra.run.smithi164.stdout:(150/154): util-linux-2.37.4-14.el9.x86_64.rpm 7.3 MB/s | 2.2 MB 00:00 2023-08-25T09:47:06.251 INFO:teuthology.orchestra.run.smithi164.stdout:(151/154): util-linux-core-2.37.4-14.el9.x86_64 1.6 MB/s | 433 kB 00:00 2023-08-25T09:47:06.268 INFO:teuthology.orchestra.run.smithi164.stdout:(152/154): util-linux-user-2.37.4-14.el9.x86_64 211 kB/s | 30 kB 00:00 2023-08-25T09:47:06.326 INFO:teuthology.orchestra.run.smithi164.stdout:(153/154): xfsdump-3.1.12-3.el9.x86_64.rpm 2.8 MB/s | 321 kB 00:00 2023-08-25T09:47:06.393 INFO:teuthology.orchestra.run.smithi164.stdout:(154/154): xfsprogs-5.19.0-4.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2023-08-25T09:47:06.404 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2023-08-25T09:47:06.405 INFO:teuthology.orchestra.run.smithi164.stdout:Total 13 MB/s | 206 MB 00:15 2023-08-25T09:47:06.492 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2023-08-25T09:47:06.809 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2023-08-25T09:47:06.810 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2023-08-25T09:47:08.890 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2023-08-25T09:47:08.891 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2023-08-25T09:47:12.256 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2023-08-25T09:47:12.388 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : openssl-libs-1:3.0.7-24.el9.x86_64 1/177 2023-08-25T09:47:12.422 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libuuid-2.37.4-14.el9.x86_64 2/177 2023-08-25T09:47:12.460 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libblkid-2.37.4-14.el9.x86_64 3/177 2023-08-25T09:47:12.578 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libblkid-2.37.4-14.el9.x86_64 3/177 2023-08-25T09:47:12.642 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/177 2023-08-25T09:47:12.719 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : elfutils-libelf-0.189-3.el9.x86_64 5/177 2023-08-25T09:47:12.801 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/177 2023-08-25T09:47:12.857 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 7/177 2023-08-25T09:47:12.894 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/177 2023-08-25T09:47:13.014 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/177 2023-08-25T09:47:13.453 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/177 2023-08-25T09:47:13.491 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 11/177 2023-08-25T09:47:13.626 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 11/177 2023-08-25T09:47:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 12/177 2023-08-25T09:47:13.798 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 12/177 2023-08-25T09:47:13.841 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:18.0.0-5703.gba6b39e4.el9.x86_64 13/177 2023-08-25T09:47:14.120 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/177 2023-08-25T09:47:14.188 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/177 2023-08-25T09:47:14.213 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 16/177 2023-08-25T09:47:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 16/177 2023-08-25T09:47:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86 17/177 2023-08-25T09:47:14.479 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86 17/177 2023-08-25T09:47:14.526 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libmount-2.37.4-14.el9.x86_64 18/177 2023-08-25T09:47:14.856 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libsmartcols-2.37.4-14.el9.x86_64 19/177 2023-08-25T09:47:15.044 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 20/177 2023-08-25T09:47:15.132 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 20/177 2023-08-25T09:47:15.335 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 21/177 2023-08-25T09:47:15.388 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-1:1.44.0-2.el9.x86_64 21/177 2023-08-25T09:47:15.427 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2023-08-25T09:47:15.533 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-libs-1:3.0.7-24.el9.x86_64 1/177 2023-08-25T09:47:15.590 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libuuid-2.37.4-14.el9.x86_64 2/177 2023-08-25T09:47:15.636 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libblkid-2.37.4-14.el9.x86_64 3/177 2023-08-25T09:47:15.754 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libblkid-2.37.4-14.el9.x86_64 3/177 2023-08-25T09:47:15.805 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 21/177 2023-08-25T09:47:15.839 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/177 2023-08-25T09:47:15.851 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/177 2023-08-25T09:47:15.888 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/177 2023-08-25T09:47:15.922 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libelf-0.189-3.el9.x86_64 5/177 2023-08-25T09:47:15.949 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 24/177 2023-08-25T09:47:16.000 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 25/177 2023-08-25T09:47:16.012 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/177 2023-08-25T09:47:16.100 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 7/177 2023-08-25T09:47:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/177 2023-08-25T09:47:16.145 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/177 2023-08-25T09:47:16.156 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/177 2023-08-25T09:47:16.189 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/177 2023-08-25T09:47:16.253 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/177 2023-08-25T09:47:16.289 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/177 2023-08-25T09:47:16.328 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el9. 30/177 2023-08-25T09:47:16.377 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 31/177 2023-08-25T09:47:16.481 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:18.0.0-5703.gba6b39e4.el9.x86_6 32/177 2023-08-25T09:47:16.657 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 33/177 2023-08-25T09:47:16.680 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/177 2023-08-25T09:47:16.722 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 34/177 2023-08-25T09:47:16.759 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 11/177 2023-08-25T09:47:16.908 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 11/177 2023-08-25T09:47:16.952 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 12/177 2023-08-25T09:47:17.086 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 12/177 2023-08-25T09:47:17.142 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:18.0.0-5703.gba6b39e4.el9.x86_64 13/177 2023-08-25T09:47:17.414 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/177 2023-08-25T09:47:17.455 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 35/177 2023-08-25T09:47:17.481 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/177 2023-08-25T09:47:17.514 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 16/177 2023-08-25T09:47:17.560 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 36/177 2023-08-25T09:47:17.648 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 16/177 2023-08-25T09:47:17.690 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86 17/177 2023-08-25T09:47:17.700 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 37/177 2023-08-25T09:47:17.772 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86 17/177 2023-08-25T09:47:17.875 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libmount-2.37.4-14.el9.x86_64 18/177 2023-08-25T09:47:17.928 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : util-linux-core-2.37.4-14.el9.x86_64 38/177 2023-08-25T09:47:18.002 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: util-linux-core-2.37.4-14.el9.x86_64 38/177 2023-08-25T09:47:18.062 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:18.0.0-5703.gba6b39e4.el9.x86_6 39/177 2023-08-25T09:47:18.119 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 40/177 2023-08-25T09:47:18.216 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libsmartcols-2.37.4-14.el9.x86_64 19/177 2023-08-25T09:47:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : elfutils-libs-0.189-3.el9.x86_64 41/177 2023-08-25T09:47:18.356 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : iproute-6.2.0-5.el9.x86_64 42/177 2023-08-25T09:47:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 20/177 2023-08-25T09:47:18.500 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 20/177 2023-08-25T09:47:18.720 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 21/177 2023-08-25T09:47:18.729 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : libfdisk-2.37.4-14.el9.x86_64 43/177 2023-08-25T09:47:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-1:1.44.0-2.el9.x86_64 21/177 2023-08-25T09:47:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : util-linux-2.37.4-14.el9.x86_64 44/177 2023-08-25T09:47:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : xfsprogs-5.19.0-4.el9.x86_64 45/177 2023-08-25T09:47:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libuuid-devel-2.37.4-14.el9.x86_64 46/177 2023-08-25T09:47:19.243 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 21/177 2023-08-25T09:47:19.295 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/177 2023-08-25T09:47:19.351 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/177 2023-08-25T09:47:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/177 2023-08-25T09:47:19.418 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : openssl-1:3.0.7-24.el9.x86_64 48/177 2023-08-25T09:47:19.581 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-2.el9.x86_64 49/177 2023-08-25T09:47:19.653 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 50/177 2023-08-25T09:47:19.735 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/177 2023-08-25T09:47:19.783 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 24/177 2023-08-25T09:47:19.843 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/177 2023-08-25T09:47:19.864 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 25/177 2023-08-25T09:47:19.879 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/177 2023-08-25T09:47:19.918 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/177 2023-08-25T09:47:19.984 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/177 2023-08-25T09:47:19.995 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/177 2023-08-25T09:47:20.044 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/177 2023-08-25T09:47:20.059 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/177 2023-08-25T09:47:20.112 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/177 2023-08-25T09:47:20.113 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/177 2023-08-25T09:47:20.244 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/177 2023-08-25T09:47:20.303 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/177 2023-08-25T09:47:20.305 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el9. 30/177 2023-08-25T09:47:20.352 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/177 2023-08-25T09:47:20.384 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 31/177 2023-08-25T09:47:20.396 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/177 2023-08-25T09:47:20.510 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:18.0.0-5703.gba6b39e4.el9.x86_6 32/177 2023-08-25T09:47:20.526 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/177 2023-08-25T09:47:20.567 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 62/177 2023-08-25T09:47:20.637 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/177 2023-08-25T09:47:20.675 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/177 2023-08-25T09:47:20.703 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 33/177 2023-08-25T09:47:20.715 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 65/177 2023-08-25T09:47:20.749 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fmt-8.1.1-5.el9.x86_64 66/177 2023-08-25T09:47:20.774 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 34/177 2023-08-25T09:47:20.907 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : zlib-devel-1.2.11-36.el9.x86_64 67/177 2023-08-25T09:47:20.965 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.16-1.el9.x86_64 68/177 2023-08-25T09:47:21.032 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 69/177 2023-08-25T09:47:21.106 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 70/177 2023-08-25T09:47:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 71/177 2023-08-25T09:47:21.420 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 72/177 2023-08-25T09:47:21.499 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 35/177 2023-08-25T09:47:21.589 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 73/177 2023-08-25T09:47:21.597 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 36/177 2023-08-25T09:47:21.646 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 74/177 2023-08-25T09:47:21.679 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 75/177 2023-08-25T09:47:21.718 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 76/177 2023-08-25T09:47:21.748 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 37/177 2023-08-25T09:47:21.761 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 77/177 2023-08-25T09:47:21.805 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 78/177 2023-08-25T09:47:21.822 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-core-2.37.4-14.el9.x86_64 38/177 2023-08-25T09:47:21.850 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 79/177 2023-08-25T09:47:21.869 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: util-linux-core-2.37.4-14.el9.x86_64 38/177 2023-08-25T09:47:21.909 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : uuid-1.6.2-55.el9.x86_64 80/177 2023-08-25T09:47:21.933 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:18.0.0-5703.gba6b39e4.el9.x86_6 39/177 2023-08-25T09:47:21.956 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 81/177 2023-08-25T09:47:22.007 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 40/177 2023-08-25T09:47:22.081 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 82/177 2023-08-25T09:47:22.126 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 83/177 2023-08-25T09:47:22.161 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 84/177 2023-08-25T09:47:22.202 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libs-0.189-3.el9.x86_64 41/177 2023-08-25T09:47:22.215 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 85/177 2023-08-25T09:47:22.256 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 86/177 2023-08-25T09:47:22.266 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : iproute-6.2.0-5.el9.x86_64 42/177 2023-08-25T09:47:22.396 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 87/177 2023-08-25T09:47:22.438 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 88/177 2023-08-25T09:47:22.578 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 89/177 2023-08-25T09:47:22.658 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 90/177 2023-08-25T09:47:22.672 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : libfdisk-2.37.4-14.el9.x86_64 43/177 2023-08-25T09:47:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-2.37.4-14.el9.x86_64 44/177 2023-08-25T09:47:23.035 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsprogs-5.19.0-4.el9.x86_64 45/177 2023-08-25T09:47:23.112 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libuuid-devel-2.37.4-14.el9.x86_64 46/177 2023-08-25T09:47:23.332 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/177 2023-08-25T09:47:23.394 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-1:3.0.7-24.el9.x86_64 48/177 2023-08-25T09:47:23.557 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-2.el9.x86_64 49/177 2023-08-25T09:47:23.606 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 50/177 2023-08-25T09:47:23.627 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/177 2023-08-25T09:47:23.676 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 92/177 2023-08-25T09:47:23.688 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/177 2023-08-25T09:47:23.709 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 93/177 2023-08-25T09:47:23.742 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 94/177 2023-08-25T09:47:23.783 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/177 2023-08-25T09:47:23.840 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/177 2023-08-25T09:47:23.896 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/177 2023-08-25T09:47:23.984 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/177 2023-08-25T09:47:24.008 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 95/177 2023-08-25T09:47:24.039 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 96/177 2023-08-25T09:47:24.064 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/177 2023-08-25T09:47:24.133 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/177 2023-08-25T09:47:24.134 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 97/177 2023-08-25T09:47:24.336 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/177 2023-08-25T09:47:24.396 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/177 2023-08-25T09:47:24.449 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/177 2023-08-25T09:47:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/177 2023-08-25T09:47:24.653 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 62/177 2023-08-25T09:47:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/177 2023-08-25T09:47:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/177 2023-08-25T09:47:24.801 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 65/177 2023-08-25T09:47:24.853 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fmt-8.1.1-5.el9.x86_64 66/177 2023-08-25T09:47:24.944 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 98/177 2023-08-25T09:47:25.038 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : zlib-devel-1.2.11-36.el9.x86_64 67/177 2023-08-25T09:47:25.083 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 99/177 2023-08-25T09:47:25.111 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.16-1.el9.x86_64 68/177 2023-08-25T09:47:25.212 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 69/177 2023-08-25T09:47:25.313 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 70/177 2023-08-25T09:47:25.598 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 71/177 2023-08-25T09:47:25.671 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 72/177 2023-08-25T09:47:25.855 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 73/177 2023-08-25T09:47:25.916 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 74/177 2023-08-25T09:47:25.956 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 75/177 2023-08-25T09:47:26.017 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 76/177 2023-08-25T09:47:26.082 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 77/177 2023-08-25T09:47:26.152 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-15.el9.x86_64 78/177 2023-08-25T09:47:26.215 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 79/177 2023-08-25T09:47:26.292 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : uuid-1.6.2-55.el9.x86_64 80/177 2023-08-25T09:47:26.353 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 81/177 2023-08-25T09:47:26.504 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 82/177 2023-08-25T09:47:26.563 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 83/177 2023-08-25T09:47:26.624 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 84/177 2023-08-25T09:47:26.640 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 100/177 2023-08-25T09:47:26.707 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 85/177 2023-08-25T09:47:26.743 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 101/177 2023-08-25T09:47:26.769 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 86/177 2023-08-25T09:47:26.781 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 102/177 2023-08-25T09:47:26.812 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-15.el9.x86_64 103/177 2023-08-25T09:47:26.852 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 104/177 2023-08-25T09:47:26.941 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 87/177 2023-08-25T09:47:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 88/177 2023-08-25T09:47:27.154 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 89/177 2023-08-25T09:47:27.218 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 105/177 2023-08-25T09:47:27.238 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 90/177 2023-08-25T09:47:27.249 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 106/177 2023-08-25T09:47:27.404 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 106/177 2023-08-25T09:47:27.439 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 107/177 2023-08-25T09:47:27.497 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 108/177 2023-08-25T09:47:27.976 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 109/177 2023-08-25T09:47:28.010 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 110/177 2023-08-25T09:47:28.100 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 110/177 2023-08-25T09:47:28.128 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 111/177 2023-08-25T09:47:28.261 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/177 2023-08-25T09:47:28.329 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 92/177 2023-08-25T09:47:28.379 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 93/177 2023-08-25T09:47:28.429 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 94/177 2023-08-25T09:47:28.716 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 95/177 2023-08-25T09:47:28.758 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 96/177 2023-08-25T09:47:28.863 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 97/177 2023-08-25T09:47:29.692 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 98/177 2023-08-25T09:47:29.845 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 99/177 2023-08-25T09:47:30.641 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 112/177 2023-08-25T09:47:30.683 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 112/177 2023-08-25T09:47:31.316 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 112/177 2023-08-25T09:47:31.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 113/177 2023-08-25T09:47:31.421 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 100/177 2023-08-25T09:47:31.529 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 101/177 2023-08-25T09:47:31.531 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 113/177 2023-08-25T09:47:31.532 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-08-25T09:47:31.532 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-08-25T09:47:31.532 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:31.575 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 102/177 2023-08-25T09:47:31.579 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 114/177 2023-08-25T09:47:31.614 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.6-15.el9.x86_64 103/177 2023-08-25T09:47:31.664 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 104/177 2023-08-25T09:47:32.055 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 105/177 2023-08-25T09:47:32.102 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 106/177 2023-08-25T09:47:32.205 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 106/177 2023-08-25T09:47:32.258 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 107/177 2023-08-25T09:47:32.335 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 108/177 2023-08-25T09:47:32.835 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 109/177 2023-08-25T09:47:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 110/177 2023-08-25T09:47:32.962 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 110/177 2023-08-25T09:47:32.998 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 111/177 2023-08-25T09:47:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 112/177 2023-08-25T09:47:35.553 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 112/177 2023-08-25T09:47:36.191 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 112/177 2023-08-25T09:47:36.247 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 113/177 2023-08-25T09:47:36.426 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 113/177 2023-08-25T09:47:36.427 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-08-25T09:47:36.427 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-08-25T09:47:36.427 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:36.490 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 114/177 2023-08-25T09:47:44.413 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 114/177 2023-08-25T09:47:44.414 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2023-08-25T09:47:44.414 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2023-08-25T09:47:44.414 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2023-08-25T09:47:44.414 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2023-08-25T09:47:44.414 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2023-08-25T09:47:44.415 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2023-08-25T09:47:44.415 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2023-08-25T09:47:44.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:44.697 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 115/177 2023-08-25T09:47:44.766 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 115/177 2023-08-25T09:47:44.767 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:44.767 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-08-25T09:47:44.767 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:46.489 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 116/177 2023-08-25T09:47:46.605 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 116/177 2023-08-25T09:47:46.606 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:46.606 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-08-25T09:47:46.606 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-25T09:47:46.606 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-25T09:47:46.607 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:46.831 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 117/177 2023-08-25T09:47:46.929 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 117/177 2023-08-25T09:47:46.930 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:46.930 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-08-25T09:47:46.930 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-25T09:47:46.931 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-25T09:47:46.931 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:47.415 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 118/177 2023-08-25T09:47:47.516 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 118/177 2023-08-25T09:47:47.517 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:47.517 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-08-25T09:47:47.517 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-25T09:47:47.517 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-25T09:47:47.517 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:47.633 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : automake-1.16.2-8.el9.noarch 119/177 2023-08-25T09:47:48.243 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 120/177 2023-08-25T09:47:48.285 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 120/177 2023-08-25T09:47:48.330 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 121/177 2023-08-25T09:47:48.444 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 122/177 2023-08-25T09:47:48.471 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noa 123/177 2023-08-25T09:47:48.798 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noa 123/177 2023-08-25T09:47:48.830 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.n 124/177 2023-08-25T09:47:49.261 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 114/177 2023-08-25T09:47:49.261 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2023-08-25T09:47:49.261 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2023-08-25T09:47:49.261 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2023-08-25T09:47:49.262 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2023-08-25T09:47:49.262 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2023-08-25T09:47:49.262 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2023-08-25T09:47:49.262 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2023-08-25T09:47:49.262 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:49.492 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 115/177 2023-08-25T09:47:49.544 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 115/177 2023-08-25T09:47:49.545 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:49.545 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-08-25T09:47:49.545 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:49.982 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.n 124/177 2023-08-25T09:47:50.006 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 125/177 2023-08-25T09:47:50.147 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 125/177 2023-08-25T09:47:50.314 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 126/177 2023-08-25T09:47:50.350 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 127/177 2023-08-25T09:47:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 127/177 2023-08-25T09:47:50.455 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:50.455 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-08-25T09:47:50.456 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-25T09:47:50.456 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-25T09:47:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 128/177 2023-08-25T09:47:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 128/177 2023-08-25T09:47:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:18.0.0-5703.gba6b39e4.el9.x86_64 129/177 2023-08-25T09:47:51.251 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 116/177 2023-08-25T09:47:51.337 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 116/177 2023-08-25T09:47:51.338 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:51.338 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-08-25T09:47:51.338 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-25T09:47:51.339 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-08-25T09:47:51.339 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:51.497 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/177 2023-08-25T09:47:51.535 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 131/177 2023-08-25T09:47:51.585 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 117/177 2023-08-25T09:47:51.623 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 131/177 2023-08-25T09:47:51.623 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:51.623 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-08-25T09:47:51.624 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-25T09:47:51.624 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-25T09:47:51.624 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:51.663 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 117/177 2023-08-25T09:47:51.663 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:51.663 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-08-25T09:47:51.663 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-25T09:47:51.664 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-08-25T09:47:51.664 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:51.680 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 132/177 2023-08-25T09:47:51.755 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 132/177 2023-08-25T09:47:51.756 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:51.756 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-08-25T09:47:51.757 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:51.837 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 133/177 2023-08-25T09:47:51.920 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 133/177 2023-08-25T09:47:51.921 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:51.921 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-08-25T09:47:51.921 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-08-25T09:47:51.921 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-08-25T09:47:51.922 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:52.168 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 118/177 2023-08-25T09:47:52.242 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 118/177 2023-08-25T09:47:52.243 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:52.243 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-08-25T09:47:52.243 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-25T09:47:52.243 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-08-25T09:47:52.244 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:52.288 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 134/177 2023-08-25T09:47:52.378 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : automake-1.16.2-8.el9.noarch 119/177 2023-08-25T09:47:52.387 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 134/177 2023-08-25T09:47:52.388 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:52.388 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-08-25T09:47:52.388 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-25T09:47:52.388 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-25T09:47:52.388 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:47:52.965 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 120/177 2023-08-25T09:47:53.021 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 120/177 2023-08-25T09:47:53.085 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 121/177 2023-08-25T09:47:53.211 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 122/177 2023-08-25T09:47:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noa 123/177 2023-08-25T09:47:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noa 123/177 2023-08-25T09:47:53.626 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.n 124/177 2023-08-25T09:47:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.n 124/177 2023-08-25T09:47:54.826 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 125/177 2023-08-25T09:47:54.969 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 125/177 2023-08-25T09:47:55.152 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 126/177 2023-08-25T09:47:55.203 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 127/177 2023-08-25T09:47:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 127/177 2023-08-25T09:47:55.280 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:55.281 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-08-25T09:47:55.281 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-25T09:47:55.281 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-08-25T09:47:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:55.353 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 128/177 2023-08-25T09:47:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 128/177 2023-08-25T09:47:55.498 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:18.0.0-5703.gba6b39e4.el9.x86_64 129/177 2023-08-25T09:47:56.347 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/177 2023-08-25T09:47:56.388 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 131/177 2023-08-25T09:47:56.479 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 131/177 2023-08-25T09:47:56.479 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:56.479 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-08-25T09:47:56.480 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-25T09:47:56.480 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-08-25T09:47:56.480 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:56.509 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:18.0.0-5703.gba6b39e4.el9.x86_64 135/177 2023-08-25T09:47:56.539 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 132/177 2023-08-25T09:47:56.563 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 136/177 2023-08-25T09:47:56.598 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 137/177 2023-08-25T09:47:56.617 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 132/177 2023-08-25T09:47:56.617 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:56.617 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-08-25T09:47:56.618 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:56.715 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 133/177 2023-08-25T09:47:56.795 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 133/177 2023-08-25T09:47:56.795 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:56.795 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-08-25T09:47:56.795 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-08-25T09:47:56.795 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-08-25T09:47:56.796 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:56.888 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 138/177 2023-08-25T09:47:57.006 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 139/177 2023-08-25T09:47:57.056 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 140/177 2023-08-25T09:47:57.106 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 141/177 2023-08-25T09:47:57.166 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 134/177 2023-08-25T09:47:57.205 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : xfsdump-3.1.12-3.el9.x86_64 142/177 2023-08-25T09:47:57.238 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 134/177 2023-08-25T09:47:57.238 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-08-25T09:47:57.238 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-08-25T09:47:57.238 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-25T09:47:57.238 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-08-25T09:47:57.239 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:47:57.331 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : util-linux-user-2.37.4-14.el9.x86_64 143/177 2023-08-25T09:47:57.374 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : iproute-tc-6.2.0-5.el9.x86_64 144/177 2023-08-25T09:47:57.428 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : elfutils-debuginfod-client-0.189-3.el9.x86_64 145/177 2023-08-25T09:47:57.455 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el9.x86_ 146/177 2023-08-25T09:47:57.507 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/177 2023-08-25T09:47:57.572 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : NetworkManager-tui-1:1.44.0-2.el9.x86_64 148/177 2023-08-25T09:47:58.421 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/177 2023-08-25T09:47:58.475 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 149/177 2023-08-25T09:47:58.548 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/177 2023-08-25T09:47:59.863 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-top-2:18.0.0-5703.gba6b39e4.el9.noarch 150/177 2023-08-25T09:47:59.944 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 151/177 2023-08-25T09:47:59.983 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 152/177 2023-08-25T09:48:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flex-2.6.4-9.el9.x86_64 153/177 2023-08-25T09:48:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : bison-3.7.4-5.el9.x86_64 154/177 2023-08-25T09:48:00.220 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-tui-1:1.41.8-1.el9.x86_64 155/177 2023-08-25T09:48:00.256 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : elfutils-debuginfod-client-0.188-3.el9.x86_64 156/177 2023-08-25T09:48:00.297 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : elfutils-libs-0.188-3.el9.x86_64 157/177 2023-08-25T09:48:00.297 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/177 2023-08-25T09:48:00.435 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/177 2023-08-25T09:48:00.485 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : openssl-1:3.0.7-2.el9.x86_64 159/177 2023-08-25T09:48:00.485 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/177 2023-08-25T09:48:00.623 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/177 2023-08-25T09:48:00.686 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : xfsdump-3.1.10-1.el9.x86_64 161/177 2023-08-25T09:48:00.732 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : xfsprogs-5.14.2-1.el9.x86_64 162/177 2023-08-25T09:48:00.779 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : iproute-tc-6.0.0-2.el9.x86_64 163/177 2023-08-25T09:48:00.822 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : iproute-6.0.0-2.el9.x86_64 164/177 2023-08-25T09:48:00.890 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : util-linux-user-2.37.4-9.el9.x86_64 165/177 2023-08-25T09:48:00.966 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : util-linux-2.37.4-9.el9.x86_64 166/177 2023-08-25T09:48:01.017 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : util-linux-core-2.37.4-9.el9.x86_64 167/177 2023-08-25T09:48:01.050 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libfdisk-2.37.4-9.el9.x86_64 168/177 2023-08-25T09:48:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libmount-2.37.4-9.el9.x86_64 169/177 2023-08-25T09:48:01.118 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libblkid-2.37.4-9.el9.x86_64 170/177 2023-08-25T09:48:01.151 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-team-1:1.41.8-1.el9.x86_64 171/177 2023-08-25T09:48:01.189 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.41.8-1.el9.x86_64 172/177 2023-08-25T09:48:01.190 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-1:1.41.8-1.el9.x86_64 172/177 2023-08-25T09:48:01.379 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:18.0.0-5703.gba6b39e4.el9.x86_64 135/177 2023-08-25T09:48:01.442 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 136/177 2023-08-25T09:48:01.485 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 137/177 2023-08-25T09:48:01.540 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-1:1.41.8-1.el9.x86_64 172/177 2023-08-25T09:48:01.540 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : NetworkManager-libnm-1:1.41.8-1.el9.x86_64 173/177 2023-08-25T09:48:01.699 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: NetworkManager-libnm-1:1.41.8-1.el9.x86_64 173/177 2023-08-25T09:48:01.729 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libuuid-2.37.4-9.el9.x86_64 174/177 2023-08-25T09:48:01.763 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : libsmartcols-2.37.4-9.el9.x86_64 175/177 2023-08-25T09:48:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 138/177 2023-08-25T09:48:01.799 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : elfutils-libelf-0.188-3.el9.x86_64 176/177 2023-08-25T09:48:01.910 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 139/177 2023-08-25T09:48:01.922 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : openssl-libs-1:3.0.7-2.el9.x86_64 177/177 2023-08-25T09:48:01.977 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 140/177 2023-08-25T09:48:02.049 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 141/177 2023-08-25T09:48:02.119 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsdump-3.1.12-3.el9.x86_64 142/177 2023-08-25T09:48:02.239 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : util-linux-user-2.37.4-14.el9.x86_64 143/177 2023-08-25T09:48:02.295 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : iproute-tc-6.2.0-5.el9.x86_64 144/177 2023-08-25T09:48:02.349 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-debuginfod-client-0.189-3.el9.x86_64 145/177 2023-08-25T09:48:02.375 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openssl-libs-1:3.0.7-2.el9.x86_64 177/177 2023-08-25T09:48:02.375 INFO:teuthology.orchestra.run.smithi071.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-08-25T09:48:02.375 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:02.393 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el9.x86_ 146/177 2023-08-25T09:48:02.456 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/177 2023-08-25T09:48:02.518 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-tui-1:1.44.0-2.el9.x86_64 148/177 2023-08-25T09:48:03.382 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/177 2023-08-25T09:48:03.439 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 149/177 2023-08-25T09:48:03.496 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/177 2023-08-25T09:48:04.866 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-top-2:18.0.0-5703.gba6b39e4.el9.noarch 150/177 2023-08-25T09:48:04.972 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 151/177 2023-08-25T09:48:05.039 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 152/177 2023-08-25T09:48:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flex-2.6.4-9.el9.x86_64 153/177 2023-08-25T09:48:05.250 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : bison-3.7.4-5.el9.x86_64 154/177 2023-08-25T09:48:05.301 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-tui-1:1.41.8-1.el9.x86_64 155/177 2023-08-25T09:48:05.343 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-debuginfod-client-0.188-3.el9.x86_64 156/177 2023-08-25T09:48:05.393 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libs-0.188-3.el9.x86_64 157/177 2023-08-25T09:48:05.393 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/177 2023-08-25T09:48:05.556 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/177 2023-08-25T09:48:05.639 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-1:3.0.7-2.el9.x86_64 159/177 2023-08-25T09:48:05.640 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/177 2023-08-25T09:48:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/177 2023-08-25T09:48:05.829 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsdump-3.1.10-1.el9.x86_64 161/177 2023-08-25T09:48:05.885 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsprogs-5.14.2-1.el9.x86_64 162/177 2023-08-25T09:48:05.951 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : iproute-tc-6.0.0-2.el9.x86_64 163/177 2023-08-25T09:48:06.017 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : iproute-6.0.0-2.el9.x86_64 164/177 2023-08-25T09:48:06.102 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-user-2.37.4-9.el9.x86_64 165/177 2023-08-25T09:48:06.187 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-2.37.4-9.el9.x86_64 166/177 2023-08-25T09:48:06.254 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : util-linux-core-2.37.4-9.el9.x86_64 167/177 2023-08-25T09:48:06.304 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libfdisk-2.37.4-9.el9.x86_64 168/177 2023-08-25T09:48:06.346 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libmount-2.37.4-9.el9.x86_64 169/177 2023-08-25T09:48:06.388 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libblkid-2.37.4-9.el9.x86_64 170/177 2023-08-25T09:48:06.429 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-team-1:1.41.8-1.el9.x86_64 171/177 2023-08-25T09:48:06.468 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.41.8-1.el9.x86_64 172/177 2023-08-25T09:48:06.468 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-1:1.41.8-1.el9.x86_64 172/177 2023-08-25T09:48:06.812 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.41.8-1.el9.x86_64 172/177 2023-08-25T09:48:06.812 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-libnm-1:1.41.8-1.el9.x86_64 173/177 2023-08-25T09:48:06.984 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.41.8-1.el9.x86_64 173/177 2023-08-25T09:48:07.032 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libuuid-2.37.4-9.el9.x86_64 174/177 2023-08-25T09:48:07.074 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : libsmartcols-2.37.4-9.el9.x86_64 175/177 2023-08-25T09:48:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libelf-0.188-3.el9.x86_64 176/177 2023-08-25T09:48:07.253 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-libs-1:3.0.7-2.el9.x86_64 177/177 2023-08-25T09:48:07.332 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:18.0.0-5703.gba6b39e4.el9.x86_64 1/177 2023-08-25T09:48:07.332 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 2/177 2023-08-25T09:48:07.332 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 3/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 4/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 5/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 6/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 7/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 8/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 9/177 2023-08-25T09:48:07.333 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 10/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 11/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:18.0.0-5703.gba6b39e4.el9.x86_64 12/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 13/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el9.x86_ 14/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 15/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 16/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:18.0.0-5703.gba6b39e4.el9.x86_6 17/177 2023-08-25T09:48:07.334 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86 18/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 19/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 20/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el9. 21/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:18.0.0-5703.gba6b39e4.el9.x86_6 22/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:18.0.0-5703.gba6b39e4.el9.x86_64 23/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 24/177 2023-08-25T09:48:07.335 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 25/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 26/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 27/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 28/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 29/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noa 30/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.n 31/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 32/177 2023-08-25T09:48:07.336 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 33/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 34/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 35/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 36/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 37/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-top-2:18.0.0-5703.gba6b39e4.el9.noarch 38/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/177 2023-08-25T09:48:07.337 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 42/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-15.el9.x86_64 47/177 2023-08-25T09:48:07.338 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-devel-2.37.4-14.el9.x86_64 53/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/177 2023-08-25T09:48:07.339 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 56/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 59/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/177 2023-08-25T09:48:07.340 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/177 2023-08-25T09:48:07.341 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/177 2023-08-25T09:48:07.342 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 85/177 2023-08-25T09:48:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 86/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.16-1.el9.x86_64 91/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : zlib-devel-1.2.11-36.el9.x86_64 92/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 93/177 2023-08-25T09:48:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 94/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 95/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 96/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 97/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 98/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 99/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 100/177 2023-08-25T09:48:07.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 101/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 102/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 103/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 104/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 105/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 106/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 107/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 108/177 2023-08-25T09:48:07.346 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 109/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 110/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 111/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 112/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 113/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 114/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 115/177 2023-08-25T09:48:07.347 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 116/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 117/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 118/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 119/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 121/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 122/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 123/177 2023-08-25T09:48:07.348 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 124/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 125/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 126/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 127/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 128/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 129/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-2.el9.x86_64 130/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 131/177 2023-08-25T09:48:07.349 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 132/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 133/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 134/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 135/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 136/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-1:1.41.8-1.el9.x86_64 137/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 138/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-libnm-1:1.41.8-1.el9.x86_64 139/177 2023-08-25T09:48:07.350 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 140/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-team-1:1.41.8-1.el9.x86_64 141/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 142/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : NetworkManager-tui-1:1.41.8-1.el9.x86_64 143/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 144/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-debuginfod-client-0.188-3.el9.x86_64 145/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 146/177 2023-08-25T09:48:07.351 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-0.188-3.el9.x86_64 147/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 148/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libs-0.188-3.el9.x86_64 149/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 150/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : iproute-6.0.0-2.el9.x86_64 151/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 152/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : iproute-tc-6.0.0-2.el9.x86_64 153/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libblkid-2.37.4-14.el9.x86_64 154/177 2023-08-25T09:48:07.352 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libblkid-2.37.4-9.el9.x86_64 155/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libfdisk-2.37.4-14.el9.x86_64 156/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libfdisk-2.37.4-9.el9.x86_64 157/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libmount-2.37.4-14.el9.x86_64 158/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libmount-2.37.4-9.el9.x86_64 159/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libsmartcols-2.37.4-14.el9.x86_64 160/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libsmartcols-2.37.4-9.el9.x86_64 161/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-2.37.4-14.el9.x86_64 162/177 2023-08-25T09:48:07.353 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-2.37.4-9.el9.x86_64 163/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 164/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-1:3.0.7-2.el9.x86_64 165/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 166/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-libs-1:3.0.7-2.el9.x86_64 167/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-2.37.4-14.el9.x86_64 168/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-2.37.4-9.el9.x86_64 169/177 2023-08-25T09:48:07.354 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-core-2.37.4-14.el9.x86_64 170/177 2023-08-25T09:48:07.355 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-core-2.37.4-9.el9.x86_64 171/177 2023-08-25T09:48:07.355 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-user-2.37.4-14.el9.x86_64 172/177 2023-08-25T09:48:07.355 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : util-linux-user-2.37.4-9.el9.x86_64 173/177 2023-08-25T09:48:07.355 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 174/177 2023-08-25T09:48:07.355 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsdump-3.1.10-1.el9.x86_64 175/177 2023-08-25T09:48:07.355 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 176/177 2023-08-25T09:48:07.723 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openssl-libs-1:3.0.7-2.el9.x86_64 177/177 2023-08-25T09:48:07.724 INFO:teuthology.orchestra.run.smithi164.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-08-25T09:48:07.724 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:09.688 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-5.14.2-1.el9.x86_64 177/177 2023-08-25T09:48:09.688 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-team-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout: NetworkManager-tui-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-debuginfod-client-0.189-3.el9.x86_64 2023-08-25T09:48:09.689 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-0.189-3.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libs-0.189-3.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: iproute-6.2.0-5.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: iproute-tc-6.2.0-5.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: libblkid-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: libfdisk-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: libmount-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.690 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: libsmartcols-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-1:3.0.7-24.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-libs-1:3.0.7-24.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-core-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.691 INFO:teuthology.orchestra.run.smithi071.stdout: util-linux-user-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout: xfsdump-3.1.12-3.el9.x86_64 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-5.19.0-4.el9.x86_64 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout: automake-1.16.2-8.el9.noarch 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout: bison-3.7.4-5.el9.x86_64 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-08-25T09:48:09.692 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.693 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.694 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-top-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:09.695 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: flex-2.6.4-9.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: fmt-8.1.1-5.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: fuse-2.9.9-15.el9.x86_64 2023-08-25T09:48:09.696 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-15.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-9.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.697 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-08-25T09:48:09.698 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: libtool-2.4.6-45.el9.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-devel-2.37.4-14.el9.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-08-25T09:48:09.699 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-13.11-1.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-server-13.11-1.el9.x86_64 2023-08-25T09:48:09.700 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-08-25T09:48:09.701 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.16-1.el9.x86_64 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-08-25T09:48:09.702 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-08-25T09:48:09.703 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-08-25T09:48:09.704 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-08-25T09:48:09.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-08-25T09:48:09.706 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-08-25T09:48:09.707 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:09.708 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-2.el9.x86_64 2023-08-25T09:48:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2023-08-25T09:48:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2023-08-25T09:48:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: uuid-1.6.2-55.el9.x86_64 2023-08-25T09:48:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-08-25T09:48:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-08-25T09:48:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel-1.2.11-36.el9.x86_64 2023-08-25T09:48:09.710 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:09.710 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2023-08-25T09:48:10.337 DEBUG:teuthology.parallel:result is None 2023-08-25T09:48:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:18.0.0-5703.gba6b39e4.el9.x86_64 1/177 2023-08-25T09:48:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 2/177 2023-08-25T09:48:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 3/177 2023-08-25T09:48:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 4/177 2023-08-25T09:48:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5703.gba6b3 5/177 2023-08-25T09:48:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 6/177 2023-08-25T09:48:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 7/177 2023-08-25T09:48:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 8/177 2023-08-25T09:48:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 9/177 2023-08-25T09:48:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 10/177 2023-08-25T09:48:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 11/177 2023-08-25T09:48:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:18.0.0-5703.gba6b39e4.el9.x86_64 12/177 2023-08-25T09:48:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 13/177 2023-08-25T09:48:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:18.0.0-5703.gba6b39e4.el9.x86_ 14/177 2023-08-25T09:48:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 15/177 2023-08-25T09:48:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 16/177 2023-08-25T09:48:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:18.0.0-5703.gba6b39e4.el9.x86_6 17/177 2023-08-25T09:48:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86 18/177 2023-08-25T09:48:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 19/177 2023-08-25T09:48:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el 20/177 2023-08-25T09:48:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:18.0.0-5703.gba6b39e4.el9. 21/177 2023-08-25T09:48:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:18.0.0-5703.gba6b39e4.el9.x86_6 22/177 2023-08-25T09:48:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:18.0.0-5703.gba6b39e4.el9.x86_64 23/177 2023-08-25T09:48:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 24/177 2023-08-25T09:48:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 25/177 2023-08-25T09:48:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 26/177 2023-08-25T09:48:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 27/177 2023-08-25T09:48:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 28/177 2023-08-25T09:48:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4. 29/177 2023-08-25T09:48:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noa 30/177 2023-08-25T09:48:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.n 31/177 2023-08-25T09:48:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6 32/177 2023-08-25T09:48:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el 33/177 2023-08-25T09:48:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 34/177 2023-08-25T09:48:12.532 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.e 35/177 2023-08-25T09:48:12.532 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 36/177 2023-08-25T09:48:12.532 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 37/177 2023-08-25T09:48:12.532 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-top-2:18.0.0-5703.gba6b39e4.el9.noarch 38/177 2023-08-25T09:48:12.532 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 42/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/177 2023-08-25T09:48:12.533 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.6-15.el9.x86_64 47/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/177 2023-08-25T09:48:12.534 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-devel-2.37.4-14.el9.x86_64 53/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 56/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 59/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/177 2023-08-25T09:48:12.535 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/177 2023-08-25T09:48:12.536 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/177 2023-08-25T09:48:12.536 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/177 2023-08-25T09:48:12.536 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/177 2023-08-25T09:48:12.536 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/177 2023-08-25T09:48:12.536 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/177 2023-08-25T09:48:12.537 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/177 2023-08-25T09:48:12.537 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/177 2023-08-25T09:48:12.537 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/177 2023-08-25T09:48:12.537 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/177 2023-08-25T09:48:12.537 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/177 2023-08-25T09:48:12.537 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/177 2023-08-25T09:48:12.538 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/177 2023-08-25T09:48:12.538 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/177 2023-08-25T09:48:12.538 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/177 2023-08-25T09:48:12.538 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/177 2023-08-25T09:48:12.538 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/177 2023-08-25T09:48:12.538 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/177 2023-08-25T09:48:12.539 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/177 2023-08-25T09:48:12.539 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/177 2023-08-25T09:48:12.539 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/177 2023-08-25T09:48:12.539 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/177 2023-08-25T09:48:12.539 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 85/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 86/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/177 2023-08-25T09:48:12.540 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/177 2023-08-25T09:48:12.541 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.16-1.el9.x86_64 91/177 2023-08-25T09:48:12.541 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : zlib-devel-1.2.11-36.el9.x86_64 92/177 2023-08-25T09:48:12.541 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 93/177 2023-08-25T09:48:12.541 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 94/177 2023-08-25T09:48:12.541 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 95/177 2023-08-25T09:48:12.541 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 96/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 97/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 98/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 99/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 100/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 101/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 102/177 2023-08-25T09:48:12.542 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 103/177 2023-08-25T09:48:12.543 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 104/177 2023-08-25T09:48:12.543 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 105/177 2023-08-25T09:48:12.543 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 106/177 2023-08-25T09:48:12.543 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 107/177 2023-08-25T09:48:12.543 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 108/177 2023-08-25T09:48:12.543 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 109/177 2023-08-25T09:48:12.544 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 110/177 2023-08-25T09:48:12.544 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 111/177 2023-08-25T09:48:12.544 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 112/177 2023-08-25T09:48:12.544 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 113/177 2023-08-25T09:48:12.544 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 114/177 2023-08-25T09:48:12.544 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 115/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 116/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 117/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 118/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 119/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 121/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 122/177 2023-08-25T09:48:12.545 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 123/177 2023-08-25T09:48:12.546 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 124/177 2023-08-25T09:48:12.546 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 125/177 2023-08-25T09:48:12.546 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 126/177 2023-08-25T09:48:12.546 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 127/177 2023-08-25T09:48:12.546 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 128/177 2023-08-25T09:48:12.546 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 129/177 2023-08-25T09:48:12.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-2.el9.x86_64 130/177 2023-08-25T09:48:12.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 131/177 2023-08-25T09:48:12.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 132/177 2023-08-25T09:48:12.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 133/177 2023-08-25T09:48:12.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 134/177 2023-08-25T09:48:12.547 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 135/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 136/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.41.8-1.el9.x86_64 137/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 138/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.41.8-1.el9.x86_64 139/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 140/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.41.8-1.el9.x86_64 141/177 2023-08-25T09:48:12.548 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 142/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.41.8-1.el9.x86_64 143/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 144/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.188-3.el9.x86_64 145/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 146/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.188-3.el9.x86_64 147/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 148/177 2023-08-25T09:48:12.549 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.188-3.el9.x86_64 149/177 2023-08-25T09:48:12.550 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 150/177 2023-08-25T09:48:12.550 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-6.0.0-2.el9.x86_64 151/177 2023-08-25T09:48:12.550 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 152/177 2023-08-25T09:48:12.550 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : iproute-tc-6.0.0-2.el9.x86_64 153/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libblkid-2.37.4-14.el9.x86_64 154/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libblkid-2.37.4-9.el9.x86_64 155/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libfdisk-2.37.4-14.el9.x86_64 156/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libfdisk-2.37.4-9.el9.x86_64 157/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libmount-2.37.4-14.el9.x86_64 158/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libmount-2.37.4-9.el9.x86_64 159/177 2023-08-25T09:48:12.551 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsmartcols-2.37.4-14.el9.x86_64 160/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsmartcols-2.37.4-9.el9.x86_64 161/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-2.37.4-14.el9.x86_64 162/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-2.37.4-9.el9.x86_64 163/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 164/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-2.el9.x86_64 165/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 166/177 2023-08-25T09:48:12.552 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-2.el9.x86_64 167/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-2.37.4-14.el9.x86_64 168/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-2.37.4-9.el9.x86_64 169/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-core-2.37.4-14.el9.x86_64 170/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-core-2.37.4-9.el9.x86_64 171/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-user-2.37.4-14.el9.x86_64 172/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : util-linux-user-2.37.4-9.el9.x86_64 173/177 2023-08-25T09:48:12.553 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 174/177 2023-08-25T09:48:12.554 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsdump-3.1.10-1.el9.x86_64 175/177 2023-08-25T09:48:12.554 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 176/177 2023-08-25T09:48:14.948 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-5.14.2-1.el9.x86_64 177/177 2023-08-25T09:48:14.948 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui-1:1.44.0-2.el9.x86_64 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client-0.189-3.el9.x86_64 2023-08-25T09:48:14.949 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-0.189-3.el9.x86_64 2023-08-25T09:48:14.950 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs-0.189-3.el9.x86_64 2023-08-25T09:48:14.950 INFO:teuthology.orchestra.run.smithi164.stdout: iproute-6.2.0-5.el9.x86_64 2023-08-25T09:48:14.950 INFO:teuthology.orchestra.run.smithi164.stdout: iproute-tc-6.2.0-5.el9.x86_64 2023-08-25T09:48:14.950 INFO:teuthology.orchestra.run.smithi164.stdout: libblkid-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.950 INFO:teuthology.orchestra.run.smithi164.stdout: libfdisk-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.952 INFO:teuthology.orchestra.run.smithi164.stdout: libmount-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.952 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.952 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.952 INFO:teuthology.orchestra.run.smithi164.stdout: libsmartcols-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.952 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-1:3.0.7-24.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs-1:3.0.7-24.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-core-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: util-linux-user-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: xfsdump-3.1.12-3.el9.x86_64 2023-08-25T09:48:14.953 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-5.19.0-4.el9.x86_64 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout: automake-1.16.2-8.el9.noarch 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout: bison-3.7.4-5.el9.x86_64 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.954 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.955 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.956 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top-2:18.0.0-5703.gba6b39e4.el9.noarch 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: flex-2.6.4-9.el9.x86_64 2023-08-25T09:48:14.957 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: fmt-8.1.1-5.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.9-15.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.6-15.el9.x86_64 2023-08-25T09:48:14.958 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-9.el9.x86_64 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-08-25T09:48:14.959 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.960 INFO:teuthology.orchestra.run.smithi164.stdout: libtool-2.4.6-45.el9.x86_64 2023-08-25T09:48:14.961 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2023-08-25T09:48:14.961 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel-2.37.4-14.el9.x86_64 2023-08-25T09:48:14.961 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-08-25T09:48:14.961 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-08-25T09:48:14.961 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-08-25T09:48:14.961 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-13.11-1.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server-13.11-1.el9.x86_64 2023-08-25T09:48:14.962 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-08-25T09:48:14.963 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2023-08-25T09:48:14.963 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-08-25T09:48:14.963 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-08-25T09:48:14.963 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.963 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.963 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.16-1.el9.x86_64 2023-08-25T09:48:14.964 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-08-25T09:48:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-08-25T09:48:14.966 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2023-08-25T09:48:14.967 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.968 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-08-25T09:48:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.970 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:18.0.0-5703.gba6b39e4.el9.x86_64 2023-08-25T09:48:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-2.el9.x86_64 2023-08-25T09:48:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-5.el9.x86_64 2023-08-25T09:48:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2023-08-25T09:48:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: uuid-1.6.2-55.el9.x86_64 2023-08-25T09:48:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-08-25T09:48:14.971 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-08-25T09:48:14.972 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel-1.2.11-36.el9.x86_64 2023-08-25T09:48:14.972 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:14.972 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2023-08-25T09:48:15.522 DEBUG:teuthology.parallel:result is None 2023-08-25T09:48:15.523 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T09:48:15.665 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-25T09:48:15.724 INFO:teuthology.orchestra.run.smithi071.stdout:18.0.0-5703.gba6b39e4.el9 2023-08-25T09:48:15.725 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5703.gba6b39e4.el9 2023-08-25T09:48:15.725 INFO:teuthology.task.install:The correct ceph version 18.0.0-5703.gba6b39e4 is installed. 2023-08-25T09:48:15.726 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba6b39e4f4f77fb95c747099b53db0467d435f56 2023-08-25T09:48:15.942 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-25T09:48:16.000 INFO:teuthology.orchestra.run.smithi164.stdout:18.0.0-5703.gba6b39e4.el9 2023-08-25T09:48:16.000 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5703.gba6b39e4.el9 2023-08-25T09:48:16.001 INFO:teuthology.task.install:The correct ceph version 18.0.0-5703.gba6b39e4 is installed. 2023-08-25T09:48:16.001 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-08-25T09:48:16.002 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:16.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-25T09:48:16.078 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:16.078 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-25T09:48:16.177 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-08-25T09:48:16.178 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:16.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2023-08-25T09:48:16.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-25T09:48:16.366 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:16.367 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2023-08-25T09:48:16.445 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-25T09:48:16.563 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-08-25T09:48:16.564 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:16.564 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-25T09:48:16.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-25T09:48:16.749 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:16.749 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-25T09:48:16.834 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-25T09:48:16.947 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-08-25T09:48:16.948 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:16.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2023-08-25T09:48:17.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-25T09:48:17.165 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:17.165 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2023-08-25T09:48:17.273 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-25T09:48:17.367 INFO:teuthology.run_tasks:Running task ceph... 2023-08-25T09:48:17.447 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-08-25T09:48:17.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2023-08-25T09:48:17.457 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2023-08-25T09:48:17.554 INFO:tasks.ceph:Disabling ceph logrotate... 2023-08-25T09:48:17.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2023-08-25T09:48:17.560 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2023-08-25T09:48:17.644 INFO:tasks.ceph:Creating extra log directories... 2023-08-25T09:48:17.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-08-25T09:48:17.653 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-08-25T09:48:17.762 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-08-25T09:48:17.763 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-08-25T09:48:17.763 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-08-25T09:48:17.763 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:17.763 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-08-25T09:48:17.822 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-08-25T09:48:18.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-08-25T09:48:18.196 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:18.197 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-08-25T09:48:18.242 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-08-25T09:48:18.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-08-25T09:48:18.528 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-08-25T09:48:18.528 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-08-25T09:48:18.529 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327', 'branch': 'wip-rishabh-2023aug3-b5', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7379327', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'a6a69670761025e039dc5ae86619b0cf04c46dc2'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-2023aug3-b5'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'test-nfs-pr-52556', 'sha1': '4b369cf18ed1391a426ab4ae86da834e9c074f81'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'test-nfs-pr-52556', 'suite_path': '/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/rishabh-d-dave/ceph', 'suite_sha1': '4b369cf18ed1391a426ab4ae86da834e9c074f81', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGK5rZRhD9wTEpAxb8Xg5U6DSQJKyvneMkKBiMVKngkXaBc4/SOTGnZ0oZgIeugfCUb2mfTjamoEClErqT06Dgg=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKTJ01r1w1pvwRpAElBNol36OwGpEbAZ+Te5ggmgsQD6Q7jm4EX3vtmJk4+R/SANSZWTU3UzOFyPs4IP3cYEwg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': 'a6a69670761025e039dc5ae86619b0cf04c46dc2'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ba6b39e4f4f77fb95c747099b53db0467d435f56'}}, {'ceph': None}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '449a1bc2027504e7b3c3d7b30fa4178906581da7', 'timestamp': '2023-08-25_06:38:25', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2224'} 2023-08-25T09:48:18.530 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-08-25T09:48:18.540 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-08-25T09:48:18.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2023-08-25T09:48:18.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2023-08-25T09:48:18.697 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:18.698 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2023-08-25T09:48:18.753 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-25T09:48:18.753 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2023-08-25T09:48:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 622 Links: 1 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-08-25 09:48:02.871346386 +0000 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-08-25 09:45:32.569082940 +0000 2023-08-25T09:48:18.807 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-08-25 09:45:32.569082940 +0000 2023-08-25T09:48:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-08-25 09:45:32.569082940 +0000 2023-08-25T09:48:18.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-25T09:48:18.895 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-08-25T09:48:18.895 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-08-25T09:48:18.896 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000365492 s, 1.4 MB/s 2023-08-25T09:48:18.898 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-25T09:48:18.952 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2023-08-25T09:48:19.005 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-25T09:48:19.005 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:19.005 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 643 Links: 1 2023-08-25T09:48:19.005 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:19.005 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:19.006 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-08-25 09:48:02.872346368 +0000 2023-08-25T09:48:19.006 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-08-25 09:45:33.618064746 +0000 2023-08-25T09:48:19.006 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-08-25 09:45:33.618064746 +0000 2023-08-25T09:48:19.006 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-08-25 09:45:33.618064746 +0000 2023-08-25T09:48:19.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-25T09:48:19.096 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-08-25T09:48:19.097 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-08-25T09:48:19.098 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000355373 s, 1.4 MB/s 2023-08-25T09:48:19.100 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-25T09:48:19.156 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2023-08-25T09:48:19.206 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-25T09:48:19.207 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:19.207 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 660 Links: 1 2023-08-25T09:48:19.207 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:19.207 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:19.207 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-08-25 09:48:02.873346351 +0000 2023-08-25T09:48:19.207 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-08-25 09:45:34.621047349 +0000 2023-08-25T09:48:19.208 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-08-25 09:45:34.621047349 +0000 2023-08-25T09:48:19.208 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-08-25 09:45:34.621047349 +0000 2023-08-25T09:48:19.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-25T09:48:19.293 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-08-25T09:48:19.293 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-08-25T09:48:19.294 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00035533 s, 1.4 MB/s 2023-08-25T09:48:19.296 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-25T09:48:19.353 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2023-08-25T09:48:19.404 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-25T09:48:19.404 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:19.404 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 679 Links: 1 2023-08-25T09:48:19.405 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:19.405 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:19.405 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2023-08-25 09:48:02.874346334 +0000 2023-08-25T09:48:19.405 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-08-25 09:45:35.659029346 +0000 2023-08-25T09:48:19.405 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-08-25 09:45:35.659029346 +0000 2023-08-25T09:48:19.405 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2023-08-25 09:45:35.659029346 +0000 2023-08-25T09:48:19.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-25T09:48:19.494 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2023-08-25T09:48:19.495 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2023-08-25T09:48:19.495 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000315818 s, 1.6 MB/s 2023-08-25T09:48:19.498 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-25T09:48:19.557 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:19.558 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:19.559 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2023-08-25T09:48:19.604 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-25T09:48:19.605 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2023-08-25T09:48:19.687 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-25T09:48:19.687 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:19.687 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 622 Links: 1 2023-08-25T09:48:19.687 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:19.688 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:19.688 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-08-25 09:48:08.198219686 +0000 2023-08-25T09:48:19.688 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-08-25 09:45:33.833930598 +0000 2023-08-25T09:48:19.688 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-08-25 09:45:33.833930598 +0000 2023-08-25T09:48:19.688 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2023-08-25 09:45:33.833930598 +0000 2023-08-25T09:48:19.689 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-25T09:48:19.797 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-08-25T09:48:19.797 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-08-25T09:48:19.797 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000359779 s, 1.4 MB/s 2023-08-25T09:48:19.800 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-25T09:48:19.843 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2023-08-25T09:48:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-25T09:48:19.930 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 641 Links: 1 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-08-25 09:48:08.200219653 +0000 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-08-25 09:45:34.872913746 +0000 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-08-25 09:45:34.872913746 +0000 2023-08-25T09:48:19.931 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2023-08-25 09:45:34.872913746 +0000 2023-08-25T09:48:19.932 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-25T09:48:20.045 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-08-25T09:48:20.045 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-08-25T09:48:20.046 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000404775 s, 1.3 MB/s 2023-08-25T09:48:20.049 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-25T09:48:20.092 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2023-08-25T09:48:20.173 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-25T09:48:20.173 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 658 Links: 1 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-08-25 09:48:08.200219653 +0000 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-08-25 09:45:35.919896764 +0000 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-08-25 09:45:35.919896764 +0000 2023-08-25T09:48:20.174 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2023-08-25 09:45:35.919896764 +0000 2023-08-25T09:48:20.175 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-25T09:48:20.284 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-08-25T09:48:20.285 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-08-25T09:48:20.285 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00031712 s, 1.6 MB/s 2023-08-25T09:48:20.288 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-25T09:48:20.330 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2023-08-25T09:48:20.414 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-25T09:48:20.414 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-25T09:48:20.414 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 678 Links: 1 2023-08-25T09:48:20.414 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:48:20.414 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2023-08-25T09:48:20.415 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-08-25 09:48:08.201219637 +0000 2023-08-25T09:48:20.415 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-08-25 09:45:36.945880122 +0000 2023-08-25T09:48:20.415 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-08-25 09:45:36.945880122 +0000 2023-08-25T09:48:20.415 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2023-08-25 09:45:36.945880122 +0000 2023-08-25T09:48:20.415 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-25T09:48:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-08-25T09:48:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-08-25T09:48:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000315414 s, 1.6 MB/s 2023-08-25T09:48:20.525 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-25T09:48:20.566 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:20.567 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-08-25T09:48:20.567 INFO:tasks.ceph:Generating config... 2023-08-25T09:48:20.568 INFO:tasks.ceph:[client] client mount timeout = 600 2023-08-25T09:48:20.569 INFO:tasks.ceph:[client] debug client = 20 2023-08-25T09:48:20.569 INFO:tasks.ceph:[client] debug ms = 1 2023-08-25T09:48:20.569 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-08-25T09:48:20.569 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-08-25T09:48:20.569 INFO:tasks.ceph:[mds] debug mds = 20 2023-08-25T09:48:20.570 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-08-25T09:48:20.570 INFO:tasks.ceph:[mds] debug ms = 1 2023-08-25T09:48:20.570 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-08-25T09:48:20.570 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-08-25T09:48:20.570 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-08-25T09:48:20.570 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-08-25T09:48:20.571 INFO:tasks.ceph:[mds] mds debug frag = True 2023-08-25T09:48:20.571 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-08-25T09:48:20.571 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2023-08-25T09:48:20.571 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-08-25T09:48:20.571 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-08-25T09:48:20.572 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2023-08-25T09:48:20.572 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-08-25T09:48:20.572 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-08-25T09:48:20.572 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-08-25T09:48:20.572 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-08-25T09:48:20.573 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-08-25T09:48:20.573 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-08-25T09:48:20.573 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-08-25T09:48:20.573 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-08-25T09:48:20.573 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-08-25T09:48:20.573 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-08-25T09:48:20.574 INFO:tasks.ceph:[mgr] debug ms = 1 2023-08-25T09:48:20.574 INFO:tasks.ceph:[mon] debug mon = 20 2023-08-25T09:48:20.574 INFO:tasks.ceph:[mon] debug ms = 1 2023-08-25T09:48:20.574 INFO:tasks.ceph:[mon] debug paxos = 20 2023-08-25T09:48:20.574 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-08-25T09:48:20.575 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-08-25T09:48:20.575 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-08-25T09:48:20.575 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-08-25T09:48:20.575 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-08-25T09:48:20.575 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-08-25T09:48:20.575 INFO:tasks.ceph:[osd] debug ms = 1 2023-08-25T09:48:20.576 INFO:tasks.ceph:[osd] debug osd = 20 2023-08-25T09:48:20.576 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-08-25T09:48:20.576 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-08-25T09:48:20.577 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-08-25T09:48:20.577 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-08-25T09:48:20.577 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-08-25T09:48:20.577 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-08-25T09:48:20.577 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-08-25T09:48:20.577 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-08-25T09:48:20.578 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-08-25T09:48:20.578 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-08-25T09:48:20.578 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-08-25T09:48:20.578 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-08-25T09:48:20.578 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-08-25T09:48:20.579 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-08-25T09:48:20.579 INFO:tasks.ceph:Setting up mon.a... 2023-08-25T09:48:20.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-08-25T09:48:20.721 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2023-08-25T09:48:20.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-08-25T09:48:20.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-08-25T09:48:20.945 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.71'), ('mon.b', '172.21.15.164'), ('mon.c', '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]')] 2023-08-25T09:48:20.946 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.71,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-08-25T09:48:20.947 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:20.947 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-08-25T09:48:20.998 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.71 --add b 172.21.15.164 --addv c '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:21.104 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:21.104 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid d3b06f22-7372-48e3-a060-a2382f01af3f 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = pacific 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:fsid d3b06f22-7372-48e3-a060-a2382f01af3f 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2023-08-25T09:48:21.103569+0000 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:created 2023-08-25T09:48:21.103569+0000 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 16 (pacific) 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2023-08-25T09:48:21.105 INFO:teuthology.orchestra.run.smithi071.stdout:0: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.a 2023-08-25T09:48:21.106 INFO:teuthology.orchestra.run.smithi071.stdout:1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.b 2023-08-25T09:48:21.106 INFO:teuthology.orchestra.run.smithi071.stdout:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2023-08-25T09:48:21.106 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-08-25T09:48:21.107 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-08-25T09:48:21.160 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d3b06f22-7372-48e3-a060-a2382f01af3f... 2023-08-25T09:48:21.161 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-08-25T09:48:21.168 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-08-25T09:48:21.326 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2023-08-25T09:48:21.327 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2023-08-25T09:48:21.327 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-25T09:48:21.327 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2023-08-25T09:48:21.327 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.328 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2023-08-25T09:48:21.328 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.328 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2023-08-25T09:48:21.328 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.329 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2023-08-25T09:48:21.329 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2023-08-25T09:48:21.329 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.329 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2023-08-25T09:48:21.329 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2023-08-25T09:48:21.329 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.330 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2023-08-25T09:48:21.331 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.331 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2023-08-25T09:48:21.331 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2023-08-25T09:48:21.331 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2023-08-25T09:48:21.331 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2023-08-25T09:48:21.331 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2023-08-25T09:48:21.332 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2023-08-25T09:48:21.332 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2023-08-25T09:48:21.332 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-25T09:48:21.332 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-25T09:48:21.332 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2023-08-25T09:48:21.332 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.333 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-25T09:48:21.333 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.333 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2023-08-25T09:48:21.333 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.333 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2023-08-25T09:48:21.333 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.71,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = d3b06f22-7372-48e3-a060-a2382f01af3f 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.334 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2023-08-25T09:48:21.335 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2023-08-25T09:48:21.336 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.336 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2023-08-25T09:48:21.336 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2023-08-25T09:48:21.336 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2023-08-25T09:48:21.336 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.336 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2023-08-25T09:48:21.337 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2023-08-25T09:48:21.337 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.337 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2023-08-25T09:48:21.337 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.337 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2023-08-25T09:48:21.338 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression mode = aggressive 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 20 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 20 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2023-08-25T09:48:21.339 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 10 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay max = 1 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay probability = 0.005 2023-08-25T09:48:21.340 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay type = client mds 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 2500 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: osd blocked scrub grace period = 3600 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: osd max markdown count = 1000 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.341 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2023-08-25T09:48:21.342 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2023-08-25T09:48:21.343 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2023-08-25T09:48:21.344 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2023-08-25T09:48:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2023-08-25T09:48:21.347 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.e] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.f] 2023-08-25T09:48:21.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds balancer = 20 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal fragment size max = 10000 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal merge size = 5 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal split bits = 3 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal split size = 100 2023-08-25T09:48:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2023-08-25T09:48:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2023-08-25T09:48:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: mds oft prefetch dirfrags = False 2023-08-25T09:48:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2023-08-25T09:48:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2023-08-25T09:48:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: mds_dir_prefetch = True 2023-08-25T09:48:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: mds_sleep_rank_change = 5000000.0 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay max = 1 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay probability = 0.005 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay type = client mds osd 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 2500 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: rados mon op timeout = 900 2023-08-25T09:48:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: rados osd op timeout = 900 2023-08-25T09:48:21.352 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2023-08-25T09:48:21.352 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2023-08-25T09:48:21.352 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-25T09:48:21.352 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2023-08-25T09:48:21.353 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.354 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2023-08-25T09:48:21.355 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.356 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2023-08-25T09:48:21.357 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.71,172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790]" 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = d3b06f22-7372-48e3-a060-a2382f01af3f 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.358 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2023-08-25T09:48:21.359 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2023-08-25T09:48:21.360 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.361 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-08-25T09:48:21.362 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay max = 1 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay probability = 0.005 2023-08-25T09:48:21.363 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay type = client mds 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 2500 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: osd blocked scrub grace period = 3600 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: osd max markdown count = 1000 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: osd op complaint time = 180 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.364 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2023-08-25T09:48:21.365 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2023-08-25T09:48:21.366 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2023-08-25T09:48:21.367 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2023-08-25T09:48:21.368 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: mon op complaint time = 120 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2023-08-25T09:48:21.369 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout: client mount timeout = 600 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout: debug client = 20 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout: rados mon op timeout = 900 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout: rados osd op timeout = 900 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2023-08-25T09:48:21.370 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.a] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.c] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.e] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.b] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.d] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.f] 2023-08-25T09:48:21.371 INFO:teuthology.orchestra.run.smithi071.stdout:[mds] 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: debug mds = 20 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: debug mds balancer = 20 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: mds bal fragment size max = 10000 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: mds bal merge size = 5 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: mds bal split bits = 3 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: mds bal split size = 100 2023-08-25T09:48:21.372 INFO:teuthology.orchestra.run.smithi071.stdout: mds debug frag = True 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mds debug scatterstat = True 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mds oft prefetch dirfrags = False 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mds op complaint time = 180 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mds verify scatter = True 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mds_dir_prefetch = True 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mds_sleep_rank_change = 5000000.0 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2023-08-25T09:48:21.373 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay max = 1 2023-08-25T09:48:21.374 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay probability = 0.005 2023-08-25T09:48:21.374 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject delay type = client mds osd 2023-08-25T09:48:21.374 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 2500 2023-08-25T09:48:21.374 INFO:teuthology.orchestra.run.smithi071.stdout: osd op complaint time = 180 2023-08-25T09:48:21.374 INFO:teuthology.orchestra.run.smithi071.stdout: rados mon op timeout = 900 2023-08-25T09:48:21.374 INFO:teuthology.orchestra.run.smithi071.stdout: rados osd op timeout = 900 2023-08-25T09:48:21.377 INFO:tasks.ceph:Creating admin key on mon.a... 2023-08-25T09:48:21.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-08-25T09:48:21.519 INFO:tasks.ceph:Copying monmap to all nodes... 2023-08-25T09:48:21.519 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:21.519 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-08-25T09:48:21.574 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:21.574 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-08-25T09:48:21.676 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:48:21.676 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:21.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2023-08-25T09:48:21.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-08-25T09:48:21.821 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:21.821 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:21.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T09:48:21.920 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:21.920 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2023-08-25T09:48:21.921 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-08-25T09:48:22.060 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:48:22.061 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:22.158 INFO:tasks.ceph:Setting up mon nodes... 2023-08-25T09:48:22.159 INFO:tasks.ceph:Setting up mgr nodes... 2023-08-25T09:48:22.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-08-25T09:48:22.353 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-08-25T09:48:22.364 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-08-25T09:48:22.563 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-08-25T09:48:22.574 INFO:tasks.ceph:Setting up mds nodes... 2023-08-25T09:48:22.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-08-25T09:48:22.734 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-08-25T09:48:22.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-08-25T09:48:22.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-08-25T09:48:23.021 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-08-25T09:48:23.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-08-25T09:48:23.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-08-25T09:48:23.305 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-08-25T09:48:23.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-08-25T09:48:23.400 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-08-25T09:48:23.596 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-08-25T09:48:23.605 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-08-25T09:48:23.694 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-08-25T09:48:23.878 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-08-25T09:48:23.887 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-08-25T09:48:23.984 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-08-25T09:48:24.169 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-08-25T09:48:24.179 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-08-25T09:48:24.272 INFO:tasks.ceph_client:Setting up client nodes... 2023-08-25T09:48:24.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-08-25T09:48:24.409 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-08-25T09:48:24.469 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-08-25T09:48:24.469 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-08-25T09:48:24.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-08-25T09:48:24.573 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:24.573 INFO:tasks.ceph:role: osd.0 2023-08-25T09:48:24.573 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:48:24.574 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-08-25T09:48:24.678 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:24.680 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:25.188 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-08-25T09:48:25.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-08-25T09:48:25.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-08-25T09:48:25.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-08-25T09:48:25.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-08-25T09:48:25.768 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:25.769 INFO:tasks.ceph:role: osd.1 2023-08-25T09:48:25.769 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:48:25.769 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-08-25T09:48:25.864 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:25.865 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:25.865 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:25.866 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:25.866 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:25.866 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:25.866 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:26.354 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-08-25T09:48:26.360 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-08-25T09:48:26.361 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-08-25T09:48:26.506 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-08-25T09:48:26.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-08-25T09:48:26.712 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:26.712 INFO:tasks.ceph:role: osd.2 2023-08-25T09:48:26.713 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:48:26.713 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-08-25T09:48:26.818 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:26.819 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:26.819 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:26.819 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:26.819 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:26.819 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:26.820 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:26.820 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:26.820 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:27.338 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-08-25T09:48:27.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-08-25T09:48:27.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-08-25T09:48:27.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-08-25T09:48:27.626 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-08-25T09:48:27.721 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:27.721 INFO:tasks.ceph:role: osd.3 2023-08-25T09:48:27.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:48:27.722 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-08-25T09:48:27.820 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:27.821 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:27.822 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:27.822 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:28.294 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2023-08-25T09:48:28.301 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2023-08-25T09:48:28.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-08-25T09:48:28.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-08-25T09:48:28.858 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:29.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:29.014+0000 7fb2ee015500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-08-25T09:48:29.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:29.015+0000 7fb2ee015500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-08-25T09:48:29.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:29.015+0000 7fb2ee015500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-08-25T09:48:29.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:29.015+0000 7fb2ee015500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-08-25T09:48:29.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:29.016+0000 7fb2ee015500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-08-25T09:48:29.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:29.024+0000 7fb2ee015500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-08-25T09:48:31.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-08-25T09:48:32.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:32.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:32.191+0000 7f9f4d206500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-08-25T09:48:32.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:32.191+0000 7f9f4d206500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-08-25T09:48:32.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:32.191+0000 7f9f4d206500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-08-25T09:48:32.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:32.191+0000 7f9f4d206500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-08-25T09:48:32.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:32.191+0000 7f9f4d206500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-08-25T09:48:32.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:32.194+0000 7f9f4d206500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-08-25T09:48:35.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-08-25T09:48:35.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:35.623+0000 7f2adb01a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-08-25T09:48:35.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:35.624+0000 7f2adb01a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-08-25T09:48:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:35.624+0000 7f2adb01a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-08-25T09:48:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:35.624+0000 7f2adb01a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-08-25T09:48:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:35.624+0000 7f2adb01a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-08-25T09:48:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:35.626+0000 7f2adb01a500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-08-25T09:48:38.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-08-25T09:48:38.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:39.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:39.050+0000 7f1f8f61b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-08-25T09:48:39.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:39.050+0000 7f1f8f61b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-08-25T09:48:39.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:39.050+0000 7f1f8f61b500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-08-25T09:48:39.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:39.050+0000 7f1f8f61b500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-08-25T09:48:39.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:39.050+0000 7f1f8f61b500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-08-25T09:48:39.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:48:39.052+0000 7f1f8f61b500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-08-25T09:48:42.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-08-25T09:48:42.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-08-25T09:48:42.429 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:42.430 INFO:tasks.ceph:role: osd.4 2023-08-25T09:48:42.430 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T09:48:42.430 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-08-25T09:48:42.535 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:42.536 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:42.537 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:42.537 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-08-25T09:48:43.069 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-08-25T09:48:43.070 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-08-25T09:48:43.271 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-08-25T09:48:43.383 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-08-25T09:48:43.483 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:43.484 INFO:tasks.ceph:role: osd.5 2023-08-25T09:48:43.484 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T09:48:43.484 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-08-25T09:48:43.586 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:43.587 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:43.587 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:43.587 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:43.587 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:43.588 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:43.589 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:43.589 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:43.590 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:43.590 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:44.075 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-08-25T09:48:44.084 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-08-25T09:48:44.085 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-08-25T09:48:44.237 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-08-25T09:48:44.332 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-08-25T09:48:44.419 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:44.420 INFO:tasks.ceph:role: osd.6 2023-08-25T09:48:44.420 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T09:48:44.420 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-08-25T09:48:44.517 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:44.518 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:44.519 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:44.519 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:45.025 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-08-25T09:48:45.031 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-08-25T09:48:45.032 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-08-25T09:48:45.223 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-08-25T09:48:45.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-08-25T09:48:45.428 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-08-25T09:48:45.429 INFO:tasks.ceph:role: osd.7 2023-08-25T09:48:45.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T09:48:45.430 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-08-25T09:48:45.526 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-25T09:48:45.527 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-08-25T09:48:45.528 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-25T09:48:45.528 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-25T09:48:45.989 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-08-25T09:48:45.996 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-08-25T09:48:45.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-08-25T09:48:46.154 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-08-25T09:48:46.255 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:46.419 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:46.417+0000 7f61e6521500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-08-25T09:48:46.423 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:46.421+0000 7f61e6521500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-08-25T09:48:46.423 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:46.421+0000 7f61e6521500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-08-25T09:48:46.423 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:46.421+0000 7f61e6521500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-08-25T09:48:46.424 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:46.421+0000 7f61e6521500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-08-25T09:48:46.433 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:46.431+0000 7f61e6521500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-08-25T09:48:48.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:48:48.541 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:48:49.616 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-08-25T09:48:49.717 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:49.860 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:49.857+0000 7f6db328c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-08-25T09:48:49.860 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:49.858+0000 7f6db328c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-08-25T09:48:49.861 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:49.858+0000 7f6db328c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-08-25T09:48:49.861 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:49.858+0000 7f6db328c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-08-25T09:48:49.861 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:49.858+0000 7f6db328c500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-08-25T09:48:49.862 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:49.860+0000 7f6db328c500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-08-25T09:48:52.808 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-08-25T09:48:52.909 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:53.046 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:53.044+0000 7f48c4b9b500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-08-25T09:48:53.047 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:53.045+0000 7f48c4b9b500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-08-25T09:48:53.047 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:53.045+0000 7f48c4b9b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-08-25T09:48:53.047 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:53.045+0000 7f48c4b9b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-08-25T09:48:53.048 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:53.045+0000 7f48c4b9b500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-08-25T09:48:53.049 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:53.047+0000 7f48c4b9b500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-08-25T09:48:56.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-08-25T09:48:56.321 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:48:56.462 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:56.460+0000 7f937a6d2500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-08-25T09:48:56.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:56.460+0000 7f937a6d2500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-08-25T09:48:56.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:56.461+0000 7f937a6d2500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-08-25T09:48:56.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:56.461+0000 7f937a6d2500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-08-25T09:48:56.463 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:56.461+0000 7f937a6d2500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-08-25T09:48:56.465 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:48:56.463+0000 7f937a6d2500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-08-25T09:48:59.635 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-08-25T09:48:59.736 INFO:tasks.ceph:Reading keys from all nodes... 2023-08-25T09:48:59.737 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:59.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-08-25T09:48:59.842 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:59.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-08-25T09:48:59.923 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:48:59.924 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-08-25T09:49:00.059 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:00.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-08-25T09:49:00.163 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:00.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-08-25T09:49:00.255 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:00.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-08-25T09:49:00.351 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:00.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-08-25T09:49:00.448 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:00.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-08-25T09:49:00.538 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:00.539 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-08-25T09:49:00.644 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:00.645 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-08-25T09:49:00.737 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:00.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-08-25T09:49:00.832 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:00.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-08-25T09:49:00.933 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:00.934 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-08-25T09:49:01.032 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:01.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-08-25T09:49:01.127 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:01.127 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-08-25T09:49:01.222 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:01.222 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-08-25T09:49:01.322 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:01.322 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-08-25T09:49:01.381 INFO:tasks.ceph:Adding keys to all mons... 2023-08-25T09:49:01.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2023-08-25T09:49:01.387 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/ceph.keyring 2023-08-25T09:49:01.467 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2023-08-25T09:49:01.467 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDmeOhkoJAEFRAABT5Y31vX0tAG6yQXv35owg== 2023-08-25T09:49:01.467 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.a] 2023-08-25T09:49:01.468 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDmeOhkxEPDKxAAn6sRe8J0cCHouvTCwxwxzA== 2023-08-25T09:49:01.468 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.c] 2023-08-25T09:49:01.468 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDneOhk80A4ARAAzpu4yzBhCEtqszyI3+U9Ng== 2023-08-25T09:49:01.468 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.e] 2023-08-25T09:49:01.468 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDneOhkLdwlEhAAY8Lb5hHtj3ELxfNVpaDyog== 2023-08-25T09:49:01.468 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDteOhk2rj9ABAA/H+sEwmbB2h2k+Vh6mNdcQ== 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDweOhkl06ACxAAxKDCKfVVu6LKp9bPkUy6QA== 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDzeOhkqoFEJRAAAkMWLDJv38MipEvXx0iPDA== 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2023-08-25T09:49:01.469 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQD3eOhk4c4YAxAAMaHSVJdzR/TyQ1mFbwFGoQ== 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDmeOhkD6CHIRAAp1D4m8nYUc0Bzdrb0vjsaw== 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.b] 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDneOhkBbB7IxAAirAqEx16rCwA6WQFi3G+WQ== 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.d] 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDneOhkib1TNBAAqKOrhiWXEJC5ARs07a9AHA== 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout:[mds.f] 2023-08-25T09:49:01.470 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDoeOhkfPgSChAApaU3dmJkMshSDmuNi5rV6Q== 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQD+eOhkyvD5GBAAv5K0VnCRXf3xkWwcan8oWg== 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQABeehkIzg3MxAA9TEZ9ZN8bXTIWLfLjVfSOQ== 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAFeehkqFq/AhAAY2SFrhunfzR7DJS72dkTLQ== 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2023-08-25T09:49:01.471 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAIeehk6eWHGxAApgkkaqo1NYT+uwF3xW7Nfg== 2023-08-25T09:49:01.472 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2023-08-25T09:49:01.472 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDoeOhk7cheGBAAZpeuxp6G9zT65et1J06q0w== 2023-08-25T09:49:01.484 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.y] 2023-08-25T09:49:01.484 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDmeOhkoJAEFRAABT5Y31vX0tAG6yQXv35owg== 2023-08-25T09:49:01.485 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2023-08-25T09:49:01.485 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDmeOhkxEPDKxAAn6sRe8J0cCHouvTCwxwxzA== 2023-08-25T09:49:01.485 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.c] 2023-08-25T09:49:01.485 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDneOhk80A4ARAAzpu4yzBhCEtqszyI3+U9Ng== 2023-08-25T09:49:01.485 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.e] 2023-08-25T09:49:01.485 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDneOhkLdwlEhAAY8Lb5hHtj3ELxfNVpaDyog== 2023-08-25T09:49:01.487 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2023-08-25T09:49:01.488 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDteOhk2rj9ABAA/H+sEwmbB2h2k+Vh6mNdcQ== 2023-08-25T09:49:01.488 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2023-08-25T09:49:01.488 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDweOhkl06ACxAAxKDCKfVVu6LKp9bPkUy6QA== 2023-08-25T09:49:01.488 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2023-08-25T09:49:01.488 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDzeOhkqoFEJRAAAkMWLDJv38MipEvXx0iPDA== 2023-08-25T09:49:01.489 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2023-08-25T09:49:01.489 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQD3eOhk4c4YAxAAMaHSVJdzR/TyQ1mFbwFGoQ== 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDmeOhkD6CHIRAAp1D4m8nYUc0Bzdrb0vjsaw== 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDneOhkBbB7IxAAirAqEx16rCwA6WQFi3G+WQ== 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.d] 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDneOhkib1TNBAAqKOrhiWXEJC5ARs07a9AHA== 2023-08-25T09:49:01.490 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.f] 2023-08-25T09:49:01.491 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDoeOhkfPgSChAApaU3dmJkMshSDmuNi5rV6Q== 2023-08-25T09:49:01.491 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.4] 2023-08-25T09:49:01.491 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQD+eOhkyvD5GBAAv5K0VnCRXf3xkWwcan8oWg== 2023-08-25T09:49:01.491 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.5] 2023-08-25T09:49:01.491 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQABeehkIzg3MxAA9TEZ9ZN8bXTIWLfLjVfSOQ== 2023-08-25T09:49:01.491 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.6] 2023-08-25T09:49:01.492 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAFeehkqFq/AhAAY2SFrhunfzR7DJS72dkTLQ== 2023-08-25T09:49:01.492 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.7] 2023-08-25T09:49:01.492 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAIeehk6eWHGxAApgkkaqo1NYT+uwF3xW7Nfg== 2023-08-25T09:49:01.492 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2023-08-25T09:49:01.492 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDoeOhk7cheGBAAZpeuxp6G9zT65et1J06q0w== 2023-08-25T09:49:01.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-25T09:49:01.522 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-25T09:49:01.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:01.681 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:01.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:01.835 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:01.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:01.992 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:02.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.145 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.288 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.294 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.443 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.603 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:02.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-25T09:49:02.767 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-25T09:49:02.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:02.936 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:03.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:03.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:03.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:03.254 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-08-25T09:49:03.398 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.409 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.558 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.571 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.731 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:03.897 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-25T09:49:04.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-25T09:49:04.055 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-25T09:49:04.209 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-08-25T09:49:04.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-08-25T09:49:04.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-08-25T09:49:04.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-08-25T09:49:04.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-08-25T09:49:04.666 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-08-25T09:49:04.829 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-08-25T09:49:04.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-08-25T09:49:05.006 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-08-25T09:49:05.166 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-08-25T09:49:05.247 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:49:05.256 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-08-25T09:49:05.309 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-08-25T09:49:05.310 INFO:tasks.ceph.mon.a:Restarting daemon 2023-08-25T09:49:05.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-08-25T09:49:05.316 INFO:tasks.ceph.mon.a:Started 2023-08-25T09:49:05.316 INFO:tasks.ceph.mon.b:Restarting daemon 2023-08-25T09:49:05.317 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-08-25T09:49:05.322 INFO:tasks.ceph.mon.b:Started 2023-08-25T09:49:05.322 INFO:tasks.ceph.mon.c:Restarting daemon 2023-08-25T09:49:05.323 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-08-25T09:49:05.328 INFO:tasks.ceph.mon.c:Started 2023-08-25T09:49:05.328 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-08-25T09:49:05.329 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-08-25T09:49:05.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-08-25T09:49:05.336 INFO:tasks.ceph.mgr.y:Started 2023-08-25T09:49:05.337 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-08-25T09:49:05.337 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-08-25T09:49:05.342 INFO:tasks.ceph.mgr.x:Started 2023-08-25T09:49:05.342 INFO:tasks.ceph:Setting crush tunables to default 2023-08-25T09:49:05.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2023-08-25T09:49:05.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.516+0000 7fa9efeb4640 1 Processor -- start 2023-08-25T09:49:05.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.516+0000 7fa9efeb4640 1 -- start start 2023-08-25T09:49:05.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9efeb4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9efeb4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9efeb4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e806e670 con 0x7fa9e814fbd0 2023-08-25T09:49:05.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9efeb4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e81569d0 con 0x7fa9e80c6390 2023-08-25T09:49:05.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9efeb4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e806e950 con 0x7fa9e806fe60 2023-08-25T09:49:05.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9ee451640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9ee451640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45564/0 (socket says 172.21.15.71:45564) 2023-08-25T09:49:05.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.518+0000 7fa9ee451640 1 -- 172.21.15.71:0/610122427 learned_addr learned my addr 172.21.15.71:0/610122427 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:05.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ee451640 1 -- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 msgr2=0x7fa9e80c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-08-25T09:49:05.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ee451640 1 -- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 msgr2=0x7fa9e80c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 msgr2=0x7fa9e80c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 msgr2=0x7fa9e80c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T09:49:05.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.519+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T09:49:05.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.720+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.720+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ee451640 1 -- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 msgr2=0x7fa9e80c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-08-25T09:49:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ee451640 1 -- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 msgr2=0x7fa9e80c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-25T09:49:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T09:49:05.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T09:49:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 msgr2=0x7fa9e80c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-08-25T09:49:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 msgr2=0x7fa9e80c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-25T09:49:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T09:49:05.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.721+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T09:49:05.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.775+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633115590 0 0) 0x7fa9e806e670 con 0x7fa9e814fbd0 2023-08-25T09:49:05.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.775+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d4003520 con 0x7fa9e814fbd0 2023-08-25T09:49:05.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.776+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1471613572 0 0) 0x7fa9d4003520 con 0x7fa9e814fbd0 2023-08-25T09:49:05.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.776+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 msgr2=0x7fa9e80c6a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-25T09:49:05.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.776+0000 7fa9ecc4e640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9e80c6a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.776+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 msgr2=0x7fa9e80c6f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-25T09:49:05.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.776+0000 7fa9ecc4e640 1 --2- 172.21.15.71:0/610122427 >> v2:172.21.15.71:3300/0 conn(0x7fa9e80c6390 0x7fa9e80c6f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.776+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e806ec30 con 0x7fa9e814fbd0 2023-08-25T09:49:05.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fa9e40016e0 con 0x7fa9e814fbd0 2023-08-25T09:49:05.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 >> v1:172.21.15.71:6789/0 conn(0x7fa9e814fbd0 legacy=0x7fa9e806f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:05.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9d4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9d4003fa0 0x7fa9d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4004d00 0x7fa9d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81569d0 con 0x7fa9d4003fa0 2023-08-25T09:49:05.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e806e950 con 0x7fa9d4004d00 2023-08-25T09:49:05.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.777+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d400ff20 con 0x7fa9e806fe60 2023-08-25T09:49:05.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.778+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9d4003fa0 0x7fa9d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.778+0000 7fa9edc50640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9d4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.778+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4004d00 0x7fa9d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.779+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 msgr2=0x7fa9d4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.779+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e806fe60 0x7fa9d4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.779+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4004d00 msgr2=0x7fa9d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.779+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4004d00 0x7fa9d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.779+0000 7fa9ed44f640 1 -- 172.21.15.71:0/610122427 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4003520 con 0x7fa9d4003fa0 2023-08-25T09:49:05.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.779+0000 7fa9ee451640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4004d00 0x7fa9d4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:05.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.781+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9d4003fa0 0x7fa9d40047c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800dae0 tx=0x7fa9d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c079a95a64d757ef server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:05.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.782+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d802c980 con 0x7fa9d4003fa0 2023-08-25T09:49:05.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.782+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d8030070 con 0x7fa9d4003fa0 2023-08-25T09:49:05.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.782+0000 7fa9ecc4e640 1 -- 172.21.15.71:0/610122427 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d802cd60 con 0x7fa9d4003fa0 2023-08-25T09:49:05.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.783+0000 7fa9efeb4640 1 -- 172.21.15.71:0/610122427 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9d4003fa0 msgr2=0x7fa9d40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:05.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.783+0000 7fa9efeb4640 1 --2- 172.21.15.71:0/610122427 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9d4003fa0 0x7fa9d40047c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800dae0 tx=0x7fa9d40057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.784+0000 7fa9ee451640 1 -- 172.21.15.71:0/610122427 reap_dead start 2023-08-25T09:49:05.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.784+0000 7fa9efeb4640 1 -- 172.21.15.71:0/610122427 shutdown_connections 2023-08-25T09:49:05.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.784+0000 7fa9efeb4640 1 -- 172.21.15.71:0/610122427 >> 172.21.15.71:0/610122427 conn(0x7fa9e806bff0 msgr2=0x7fa9e806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:05.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.784+0000 7fa9efeb4640 1 -- 172.21.15.71:0/610122427 shutdown_connections 2023-08-25T09:49:05.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.784+0000 7fa9efeb4640 1 -- 172.21.15.71:0/610122427 wait complete. 2023-08-25T09:49:05.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.789+0000 7fa9efeb4640 1 Processor -- start 2023-08-25T09:49:05.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.789+0000 7fa9efeb4640 1 -- start start 2023-08-25T09:49:05.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9efeb4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4003fa0 0x7fa9e8146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9efeb4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 0x7fa9e81eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9efeb4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e81ea410 0x7fa9e8153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:05.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9efeb4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81471c0 con 0x7fa9e8148980 2023-08-25T09:49:05.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9efeb4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e81474a0 con 0x7fa9d4003fa0 2023-08-25T09:49:05.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9efeb4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e8147bc0 con 0x7fa9e81ea410 2023-08-25T09:49:05.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9ed44f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 0x7fa9e81eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9ed44f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 0x7fa9e81eae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45590/0 (socket says 172.21.15.71:45590) 2023-08-25T09:49:05.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.791+0000 7fa9ed44f640 1 -- 172.21.15.71:0/3410903066 learned_addr learned my addr 172.21.15.71:0/3410903066 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:05.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9edc50640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4003fa0 0x7fa9e8146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9ee451640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e81ea410 0x7fa9e8153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:05.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9ed44f640 1 -- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e81ea410 msgr2=0x7fa9e8153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:05.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e81ea410 0x7fa9e8153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9ed44f640 1 -- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4003fa0 msgr2=0x7fa9e8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:05.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4003fa0 0x7fa9e8146b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.792+0000 7fa9ed44f640 1 -- 172.21.15.71:0/3410903066 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8147ea0 con 0x7fa9e8148980 2023-08-25T09:49:05.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.793+0000 7fa9ed44f640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 0x7fa9e81eae30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8038040 tx=0x7fa9d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:05.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.793+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8045070 con 0x7fa9e8148980 2023-08-25T09:49:05.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.793+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d800c040 con 0x7fa9e8148980 2023-08-25T09:49:05.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.793+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d803f8d0 con 0x7fa9e8148980 2023-08-25T09:49:05.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.793+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e8148180 con 0x7fa9e8148980 2023-08-25T09:49:05.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.794+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e81ea890 con 0x7fa9e8148980 2023-08-25T09:49:05.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.794+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa9d803fd30 con 0x7fa9e8148980 2023-08-25T09:49:05.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.794+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa9d8036d90 con 0x7fa9e8148980 2023-08-25T09:49:05.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.795+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e8069160 con 0x7fa9e8148980 2023-08-25T09:49:05.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.798+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fa9d8032070 con 0x7fa9e8148980 2023-08-25T09:49:05.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.871+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa9e80c13b0 con 0x7fa9e8148980 2023-08-25T09:49:05.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.875+0000 7fa9de7fc640 1 -- 172.21.15.71:0/3410903066 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa9d800d6e0 con 0x7fa9e8148980 2023-08-25T09:49:05.877 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2023-08-25T09:49:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 msgr2=0x7fa9e81eae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 0x7fa9e81eae30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8038040 tx=0x7fa9d80046c0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 shutdown_connections 2023-08-25T09:49:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9e81ea410 0x7fa9e8153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8148980 0x7fa9e81eae30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 --2- 172.21.15.71:0/3410903066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d4003fa0 0x7fa9e8146b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:05.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 >> 172.21.15.71:0/3410903066 conn(0x7fa9e806bff0 msgr2=0x7fa9e814de20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:05.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 shutdown_connections 2023-08-25T09:49:05.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:05.877+0000 7fa9efeb4640 1 -- 172.21.15.71:0/3410903066 wait complete. 2023-08-25T09:49:05.897 INFO:tasks.ceph:check_enable_crimson: False 2023-08-25T09:49:05.897 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-08-25T09:49:05.898 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:05.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-08-25T09:49:05.982 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:05.980+0000 7f91c920e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T09:49:05.987 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:05.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-08-25T09:49:06.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:06.009+0000 7f5a3b042040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-25T09:49:06.123 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:06.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-08-25T09:49:06.209 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:06.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-08-25T09:49:06.346 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:06.346 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-08-25T09:49:06.439 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:06.440 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-08-25T09:49:06.533 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:06.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-08-25T09:49:06.622 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-08-25T09:49:06.623 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-08-25T09:49:06.755 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new ec3e1497-d839-448b-b911-2c867ac9c98e 0 2023-08-25T09:49:06.941 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:06.939+0000 7f91c920e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T09:49:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.951+0000 7f4119f0f640 1 Processor -- start 2023-08-25T09:49:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.951+0000 7f4119f0f640 1 -- start start 2023-08-25T09:49:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f4119f0f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f411415ce50 0x7f41140cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f4119f0f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41141591c0 con 0x7f4114152d80 2023-08-25T09:49:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f4119f0f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41141594a0 con 0x7f41140cacb0 2023-08-25T09:49:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f4119f0f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4114159780 con 0x7f411415ce50 2023-08-25T09:49:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f410bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f41140cacb0 0x7f411415c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55370/0 (socket says 172.21.15.164:55370) 2023-08-25T09:49:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f4113fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f411415ce50 0x7f41140cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.953+0000 7f410bfff640 1 -- 172.21.15.164:0/4030845891 learned_addr learned my addr 172.21.15.164:0/4030845891 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-08-25T09:49:06.963 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.954+0000 7f4112ffd640 1 -- 172.21.15.164:0/4030845891 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008398628 0 0) 0x7f41141594a0 con 0x7f41140cacb0 2023-08-25T09:49:06.963 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.954+0000 7f4112ffd640 1 -- 172.21.15.164:0/4030845891 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4100003520 con 0x7f41140cacb0 2023-08-25T09:49:06.963 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.954+0000 7f4113fff640 1 -- 172.21.15.164:0/4030845891 >> v1:172.21.15.164:6789/0 conn(0x7f41140cacb0 legacy=0x7f411415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:06.963 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.954+0000 7f4113fff640 1 -- 172.21.15.164:0/4030845891 >> v1:172.21.15.71:6789/0 conn(0x7f4114152d80 legacy=0x7f41140c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:06.964 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.954+0000 7f4113fff640 1 -- 172.21.15.164:0/4030845891 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114159a60 con 0x7f411415ce50 2023-08-25T09:49:06.964 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.955+0000 7f4113fff640 1 --2- 172.21.15.164:0/4030845891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f411415ce50 0x7f41140cb140 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f410c009d90 tx=0x7f410c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce32c8179715414a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:06.964 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.955+0000 7f4112ffd640 1 -- 172.21.15.164:0/4030845891 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c02cd00 con 0x7f411415ce50 2023-08-25T09:49:06.965 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.955+0000 7f4112ffd640 1 -- 172.21.15.164:0/4030845891 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c02ce60 con 0x7f411415ce50 2023-08-25T09:49:06.965 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.955+0000 7f4112ffd640 1 -- 172.21.15.164:0/4030845891 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c034640 con 0x7f411415ce50 2023-08-25T09:49:06.965 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 -- 172.21.15.164:0/4030845891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f411415ce50 msgr2=0x7f41140cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:06.965 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 --2- 172.21.15.164:0/4030845891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f411415ce50 0x7f41140cb140 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f410c009d90 tx=0x7f410c02bf50 comp rx=0 tx=0).stop 2023-08-25T09:49:06.966 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 -- 172.21.15.164:0/4030845891 shutdown_connections 2023-08-25T09:49:06.966 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 --2- 172.21.15.164:0/4030845891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f411415ce50 0x7f41140cb140 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:06.966 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 -- 172.21.15.164:0/4030845891 >> 172.21.15.164:0/4030845891 conn(0x7f411406b8f0 msgr2=0x7f41140654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:06.966 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 -- 172.21.15.164:0/4030845891 shutdown_connections 2023-08-25T09:49:06.966 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.956+0000 7f4119f0f640 1 -- 172.21.15.164:0/4030845891 wait complete. 2023-08-25T09:49:06.967 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.958+0000 7f4119f0f640 1 Processor -- start 2023-08-25T09:49:06.967 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.958+0000 7f4119f0f640 1 -- start start 2023-08-25T09:49:06.967 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4119f0f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41140cacb0 0x7f41141f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:06.967 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4119f0f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 0x7f41140b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4119f0f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f411415ce50 0x7f411414db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4119f0f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140ce480 con 0x7f411415ce50 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4119f0f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4114152570 con 0x7f41140cacb0 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4119f0f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4114152290 con 0x7f4114152d80 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 0x7f41140b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f41137fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41140cacb0 0x7f41141f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:06.968 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 0x7f41140b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51250/0 (socket says 172.21.15.164:51250) 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 -- 172.21.15.164:0/1881028693 learned_addr learned my addr 172.21.15.164:0/1881028693 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f4113fff640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f411415ce50 0x7f411414db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 -- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41140cacb0 msgr2=0x7f41141f22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41140cacb0 0x7f41141f22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 -- 172.21.15.164:0/1881028693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f411415ce50 msgr2=0x7f411414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f411415ce50 0x7f411414db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.959+0000 7f410bfff640 1 -- 172.21.15.164:0/1881028693 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c009a70 con 0x7f4114152d80 2023-08-25T09:49:06.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f410bfff640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 0x7f41140b3730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c430 tx=0x7f40fc00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc005ce0 con 0x7f4114152d80 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40fc005e40 con 0x7f4114152d80 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0072f0 con 0x7f4114152d80 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41141f1800 con 0x7f4114152d80 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41140ce140 con 0x7f4114152d80 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f40fc01c440 con 0x7f4114152d80 2023-08-25T09:49:06.970 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.960+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f40fc012810 con 0x7f4114152d80 2023-08-25T09:49:06.971 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.961+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41140b1a60 con 0x7f4114152d80 2023-08-25T09:49:06.971 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:06.966+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f40fc017070 con 0x7f4114152d80 2023-08-25T09:49:07.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:07.028+0000 7f5a3b042040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-25T09:49:07.045 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.044+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ec3e1497-d839-448b-b911-2c867ac9c98e", "id": 0} v 0) v1 -- 0x7f4114069160 con 0x7f4114152d80 2023-08-25T09:49:07.055 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.054+0000 7f4110ff9640 1 -- 172.21.15.164:0/1881028693 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ec3e1497-d839-448b-b911-2c867ac9c98e", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f40fc012af0 con 0x7f4114152d80 2023-08-25T09:49:07.056 INFO:teuthology.orchestra.run.smithi164.stdout:0 2023-08-25T09:49:07.057 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 msgr2=0x7f41140b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.057 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 0x7f41140b3730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00c430 tx=0x7f40fc00c8f0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.057 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 shutdown_connections 2023-08-25T09:49:07.058 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f411415ce50 0x7f411414db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.058 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4114152d80 0x7f41140b3730 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.058 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 --2- 172.21.15.164:0/1881028693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41140cacb0 0x7f41141f22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.059 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 >> 172.21.15.164:0/1881028693 conn(0x7f411406b8f0 msgr2=0x7f41140654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.059 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 shutdown_connections 2023-08-25T09:49:07.059 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.055+0000 7f4119f0f640 1 -- 172.21.15.164:0/1881028693 wait complete. 2023-08-25T09:49:07.073 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new b113539d-3236-4a8e-8000-b76810490755 1 2023-08-25T09:49:07.098 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:07.096+0000 7f91c920e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T09:49:07.154 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:07.153+0000 7f91c920e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T09:49:07.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:07.198+0000 7f5a3b042040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-25T09:49:07.206 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:07.205+0000 7f91c920e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T09:49:07.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:07.257+0000 7f5a3b042040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-25T09:49:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.263+0000 7f4fb759e640 1 Processor -- start 2023-08-25T09:49:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.264+0000 7f4fb759e640 1 -- start start 2023-08-25T09:49:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb759e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4fb80ca010 0x7f4fb80c9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb759e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4fb80c97d0 0x7f4fb8158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb759e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb80ca550 con 0x7f4fb80c8310 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb759e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb815d010 con 0x7f4fb80ca010 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb759e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb815c390 con 0x7f4fb80c97d0 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb6d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4fb80c97d0 0x7f4fb8158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb6d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4fb80c97d0 0x7f4fb8158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45558/0 (socket says 172.21.15.164:45558) 2023-08-25T09:49:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.265+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/4283143946 learned_addr learned my addr 172.21.15.164:0/4283143946 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.266+0000 7f4faffff640 1 --2- 172.21.15.164:0/4283143946 >> v2:172.21.15.71:3300/0 conn(0x7f4fb80ca010 0x7f4fb80c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.266+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/4283143946 >> v2:172.21.15.71:3300/0 conn(0x7f4fb80ca010 msgr2=0x7f4fb80c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.266+0000 7f4fb6d9d640 1 --2- 172.21.15.164:0/4283143946 >> v2:172.21.15.71:3300/0 conn(0x7f4fb80ca010 0x7f4fb80c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.266+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/4283143946 >> v1:172.21.15.71:6789/0 conn(0x7f4fb80c8310 legacy=0x7f4fb80b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.266+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/4283143946 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb815c670 con 0x7f4fb80c97d0 2023-08-25T09:49:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.267+0000 7f4fb6d9d640 1 --2- 172.21.15.164:0/4283143946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8158d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4fb80c8fb0 tx=0x7f4fb002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed647ffb0cd5c2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.267+0000 7f4fb5d9b640 1 -- 172.21.15.164:0/4283143946 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb002cb20 con 0x7f4fb80c97d0 2023-08-25T09:49:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.267+0000 7f4fb5d9b640 1 -- 172.21.15.164:0/4283143946 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb002cc80 con 0x7f4fb80c97d0 2023-08-25T09:49:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.267+0000 7f4fb5d9b640 1 -- 172.21.15.164:0/4283143946 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00343f0 con 0x7f4fb80c97d0 2023-08-25T09:49:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 -- 172.21.15.164:0/4283143946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c97d0 msgr2=0x7f4fb8158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 --2- 172.21.15.164:0/4283143946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8158d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4fb80c8fb0 tx=0x7f4fb002bf80 comp rx=0 tx=0).stop 2023-08-25T09:49:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 -- 172.21.15.164:0/4283143946 shutdown_connections 2023-08-25T09:49:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 --2- 172.21.15.164:0/4283143946 >> v2:172.21.15.71:3300/0 conn(0x7f4fb80ca010 0x7f4fb80c9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 --2- 172.21.15.164:0/4283143946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8158d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.272 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 -- 172.21.15.164:0/4283143946 >> 172.21.15.164:0/4283143946 conn(0x7f4fb806ae20 msgr2=0x7f4fb806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.272 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 -- 172.21.15.164:0/4283143946 shutdown_connections 2023-08-25T09:49:07.272 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.268+0000 7f4fb759e640 1 -- 172.21.15.164:0/4283143946 wait complete. 2023-08-25T09:49:07.275 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.270+0000 7f4fb759e640 1 Processor -- start 2023-08-25T09:49:07.275 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.270+0000 7f4fb759e640 1 -- start start 2023-08-25T09:49:07.276 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c8310 0x7f4fb8158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.276 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb759e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.276 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 0x7f4fb8156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.277 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb759e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb8157a20 con 0x7f4fb80c97d0 2023-08-25T09:49:07.277 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb759e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb81edad0 con 0x7f4fb80c8310 2023-08-25T09:49:07.277 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb8158ed0 con 0x7f4fb80ca010 2023-08-25T09:49:07.277 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb6d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 0x7f4fb8156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.278 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c8310 0x7f4fb8158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.278 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb6d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 0x7f4fb8156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51262/0 (socket says 172.21.15.164:51262) 2023-08-25T09:49:07.278 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.271+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/2224360873 learned_addr learned my addr 172.21.15.164:0/2224360873 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:07.278 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4faffff640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.279 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c8310 msgr2=0x7f4fb8158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.279 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb6d9d640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c8310 0x7f4fb8158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.279 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/2224360873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fb80c97d0 msgr2=0x7f4fb8154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.280 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb6d9d640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.280 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb6d9d640 1 -- 172.21.15.164:0/2224360873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb0009b30 con 0x7f4fb80ca010 2023-08-25T09:49:07.280 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb659c640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c8310 0x7f4fb8158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:49:07.281 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4faffff640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:07.281 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb6d9d640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 0x7f4fb8156030 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0036040 tx=0x7f4fb0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:07.281 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0007e70 con 0x7f4fb80ca010 2023-08-25T09:49:07.282 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb0009510 con 0x7f4fb80ca010 2023-08-25T09:49:07.282 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb003e690 con 0x7f4fb80ca010 2023-08-25T09:49:07.282 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.272+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb81591b0 con 0x7f4fb80ca010 2023-08-25T09:49:07.283 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.273+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb81ee070 con 0x7f4fb80ca010 2023-08-25T09:49:07.283 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.273+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4fb003eaf0 con 0x7f4fb80ca010 2023-08-25T09:49:07.283 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.273+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f4fb0048d50 con 0x7f4fb80ca010 2023-08-25T09:49:07.283 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.274+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb804e420 con 0x7f4fb80ca010 2023-08-25T09:49:07.284 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.278+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f4fb0043020 con 0x7f4fb80ca010 2023-08-25T09:49:07.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:07.314+0000 7f5a3b042040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-25T09:49:07.357 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.356+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b113539d-3236-4a8e-8000-b76810490755", "id": 1} v 0) v1 -- 0x7f4fb80c9df0 con 0x7f4fb80ca010 2023-08-25T09:49:07.367 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.365+0000 7f4faeffd640 1 -- 172.21.15.164:0/2224360873 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b113539d-3236-4a8e-8000-b76810490755", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4fb0048400 con 0x7f4fb80ca010 2023-08-25T09:49:07.368 INFO:teuthology.orchestra.run.smithi164.stdout:1 2023-08-25T09:49:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.367+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 msgr2=0x7f4fb8156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.370 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.367+0000 7f4fb759e640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 0x7f4fb8156030 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0036040 tx=0x7f4fb0004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:07.370 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 shutdown_connections 2023-08-25T09:49:07.370 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fb80ca010 0x7f4fb8156030 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.370 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fb80c97d0 0x7f4fb8154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.371 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 --2- 172.21.15.164:0/2224360873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fb80c8310 0x7f4fb8158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.371 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 >> 172.21.15.164:0/2224360873 conn(0x7f4fb806ae20 msgr2=0x7f4fb80c7140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.371 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 shutdown_connections 2023-08-25T09:49:07.371 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.368+0000 7f4fb759e640 1 -- 172.21.15.164:0/2224360873 wait complete. 2023-08-25T09:49:07.387 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 25701b56-737a-4fe4-9cf8-355b7e2491c1 2 2023-08-25T09:49:07.572 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.571+0000 7fe101b6d640 1 Processor -- start 2023-08-25T09:49:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.571+0000 7fe101b6d640 1 -- start start 2023-08-25T09:49:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.572+0000 7fe101b6d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0fc0cb190 0x7fe0fc0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe101b6d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0fc0b1af0 con 0x7fe0fc15cd00 2023-08-25T09:49:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe101b6d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0fc0b0fb0 con 0x7fe0fc0cb570 2023-08-25T09:49:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe101b6d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0fc0c98f0 con 0x7fe0fc0cb190 2023-08-25T09:49:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe0fbfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe0fc0cb570 0x7fe0fc0c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55380/0 (socket says 172.21.15.164:55380) 2023-08-25T09:49:07.575 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe0fbfff640 1 -- 172.21.15.164:0/3017353219 learned_addr learned my addr 172.21.15.164:0/3017353219 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-08-25T09:49:07.576 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe100b6b640 1 --2- 172.21.15.164:0/3017353219 >> v2:172.21.15.71:3300/0 conn(0x7fe0fc0cb190 0x7fe0fc0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.576 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677998215 0 0) 0x7fe0fc0b0fb0 con 0x7fe0fc0cb570 2023-08-25T09:49:07.576 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.573+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec003520 con 0x7fe0fc0cb570 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1064818531 0 0) 0x7fe0ec003520 con 0x7fe0fc0cb570 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 >> v2:172.21.15.71:3300/0 conn(0x7fe0fc0cb190 msgr2=0x7fe0fc0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 --2- 172.21.15.164:0/3017353219 >> v2:172.21.15.71:3300/0 conn(0x7fe0fc0cb190 0x7fe0fc0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 >> v1:172.21.15.71:6789/0 conn(0x7fe0fc15cd00 legacy=0x7fe0fc158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc15d130 con 0x7fe0fc0cb570 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe0e8001670 con 0x7fe0fc0cb570 2023-08-25T09:49:07.577 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 >> v1:172.21.15.164:6789/0 conn(0x7fe0fc0cb570 legacy=0x7fe0fc0c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0039e0 0x7fe0ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0ec0047c0 0x7fe0ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0ec005180 0x7fe0ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0c98f0 con 0x7fe0ec005180 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec00c090 con 0x7fe0ec0047c0 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec00c250 con 0x7fe0ec0039e0 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fbfff640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0ec0047c0 0x7fe0ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.578 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe10136c640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0ec005180 0x7fe0ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe0fbfff640 1 -- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0039e0 msgr2=0x7fe0ec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.574+0000 7fe100b6b640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0039e0 0x7fe0ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.575+0000 7fe0fbfff640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0039e0 0x7fe0ec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.575+0000 7fe0fbfff640 1 -- 172.21.15.164:0/3017353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0ec005180 msgr2=0x7fe0ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.575+0000 7fe0fbfff640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0ec005180 0x7fe0ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.575+0000 7fe0fbfff640 1 -- 172.21.15.164:0/3017353219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec003520 con 0x7fe0ec0047c0 2023-08-25T09:49:07.579 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.575+0000 7fe10136c640 1 -- 172.21.15.164:0/3017353219 reap_dead start 2023-08-25T09:49:07.580 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.575+0000 7fe100b6b640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0039e0 0x7fe0ec003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:07.582 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.576+0000 7fe0fbfff640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0ec0047c0 0x7fe0ec004c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0cb980 tx=0x7fe0ec0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d69ff3e1e569d8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:07.582 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.576+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8005ad0 con 0x7fe0ec0047c0 2023-08-25T09:49:07.582 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.576+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e8005c50 con 0x7fe0ec0047c0 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.576+0000 7fe0fb7fe640 1 -- 172.21.15.164:0/3017353219 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e802c850 con 0x7fe0ec0047c0 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.576+0000 7fe101b6d640 1 -- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0ec0047c0 msgr2=0x7fe0ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.576+0000 7fe101b6d640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0ec0047c0 0x7fe0ec004c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0cb980 tx=0x7fe0ec0057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.577+0000 7fe101b6d640 1 -- 172.21.15.164:0/3017353219 shutdown_connections 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.577+0000 7fe101b6d640 1 --2- 172.21.15.164:0/3017353219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0ec0047c0 0x7fe0ec004c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.577+0000 7fe101b6d640 1 -- 172.21.15.164:0/3017353219 >> 172.21.15.164:0/3017353219 conn(0x7fe0fc06ae20 msgr2=0x7fe0fc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.577+0000 7fe101b6d640 1 -- 172.21.15.164:0/3017353219 shutdown_connections 2023-08-25T09:49:07.583 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.577+0000 7fe101b6d640 1 -- 172.21.15.164:0/3017353219 wait complete. 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.578+0000 7fe101b6d640 1 Processor -- start 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.578+0000 7fe101b6d640 1 -- start start 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.579+0000 7fe101b6d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 0x7fe0fc0c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.579+0000 7fe101b6d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc0b2c30 0x7fe0fc159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.579+0000 7fe101b6d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0fc1f12c0 0x7fe0fc151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.579+0000 7fe101b6d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc0c3170 con 0x7fe0fc0b2c30 2023-08-25T09:49:07.584 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.579+0000 7fe101b6d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc1eded0 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.579+0000 7fe101b6d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc1ee190 con 0x7fe0ec0047c0 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe100b6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 0x7fe0fc0c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0fc1f12c0 0x7fe0fc151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe100b6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 0x7fe0fc0c3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51286/0 (socket says 172.21.15.164:51286) 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe100b6b640 1 -- 172.21.15.164:0/3618689745 learned_addr learned my addr 172.21.15.164:0/3618689745 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe0fbfff640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc0b2c30 0x7fe0fc159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 -- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 msgr2=0x7fe0fc0c3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.585 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 0x7fe0fc0c3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 -- 172.21.15.164:0/3618689745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc0b2c30 msgr2=0x7fe0fc159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc0b2c30 0x7fe0fc159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 -- 172.21.15.164:0/3618689745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc1ee470 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe0fbfff640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc0b2c30 0x7fe0fc159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe100b6b640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 0x7fe0fc0c3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.580+0000 7fe10136c640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0fc1f12c0 0x7fe0fc151980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc06b9a0 tx=0x7fe0f400e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:07.586 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4006e20 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.587 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f4003970 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.587 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0fc1ee750 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.587 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f4003d20 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.587 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0fc0c40c0 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.587 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe0f4005410 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.587 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.581+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe0f4014da0 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.588 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.582+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc0c1e30 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.588 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.585+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fe0f4019070 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.662+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "25701b56-737a-4fe4-9cf8-355b7e2491c1", "id": 2} v 0) v1 -- 0x7fe0fc04e420 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.667+0000 7fe0f97fa640 1 -- 172.21.15.164:0/3618689745 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "25701b56-737a-4fe4-9cf8-355b7e2491c1", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe0f40056c0 con 0x7fe0fc1f12c0 2023-08-25T09:49:07.669 INFO:teuthology.orchestra.run.smithi164.stdout:2 2023-08-25T09:49:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0fc1f12c0 msgr2=0x7fe0fc151980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0fc1f12c0 0x7fe0fc151980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc06b9a0 tx=0x7fe0f400e9c0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 shutdown_connections 2023-08-25T09:49:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0fc1f12c0 0x7fe0fc151980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0fc0b2c30 0x7fe0fc159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 --2- 172.21.15.164:0/3618689745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0ec0047c0 0x7fe0fc0c3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 >> 172.21.15.164:0/3618689745 conn(0x7fe0fc06ae20 msgr2=0x7fe0fc0cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 shutdown_connections 2023-08-25T09:49:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.669+0000 7fe101b6d640 1 -- 172.21.15.164:0/3618689745 wait complete. 2023-08-25T09:49:07.685 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 0d2b548b-2377-4ab5-b195-e5d93c0893b0 3 2023-08-25T09:49:07.878 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.877+0000 7fb04359e640 1 Processor -- start 2023-08-25T09:49:07.879 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.877+0000 7fb04359e640 1 -- start start 2023-08-25T09:49:07.880 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.878+0000 7fb04359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0440c8310 0x7fb0440b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.880 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb044158d80 0x7fb044159210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.880 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04359e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb044159750 con 0x7fb0440ca010 2023-08-25T09:49:07.880 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04415d010 con 0x7fb044158d80 2023-08-25T09:49:07.880 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440b14f0 con 0x7fb0440c8310 2023-08-25T09:49:07.881 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb041d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb0440ca010 0x7fb0440c9290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55396/0 (socket says 172.21.15.164:55396) 2023-08-25T09:49:07.881 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0440c8310 0x7fb0440b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb041d9b640 1 -- 172.21.15.164:0/2755953763 learned_addr learned my addr 172.21.15.164:0/2755953763 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb042d9d640 1 --2- 172.21.15.164:0/2755953763 >> v2:172.21.15.71:3300/0 conn(0x7fb044158d80 0x7fb044159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04259c640 1 -- 172.21.15.164:0/2755953763 >> v2:172.21.15.71:3300/0 conn(0x7fb044158d80 msgr2=0x7fb044159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04259c640 1 --2- 172.21.15.164:0/2755953763 >> v2:172.21.15.71:3300/0 conn(0x7fb044158d80 0x7fb044159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04259c640 1 -- 172.21.15.164:0/2755953763 >> v1:172.21.15.164:6789/0 conn(0x7fb0440ca010 legacy=0x7fb0440c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04259c640 1 -- 172.21.15.164:0/2755953763 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440ca550 con 0x7fb0440c8310 2023-08-25T09:49:07.882 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.879+0000 7fb04159a640 1 -- 172.21.15.164:0/2755953763 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942656429 0 0) 0x7fb044159750 con 0x7fb0440ca010 2023-08-25T09:49:07.883 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.881+0000 7fb04259c640 1 --2- 172.21.15.164:0/2755953763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0440c8310 0x7fb0440b0fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0340097a0 tx=0x7fb03402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3055bec1d992b8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:07.884 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.881+0000 7fb04159a640 1 -- 172.21.15.164:0/2755953763 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034030800 con 0x7fb0440c8310 2023-08-25T09:49:07.884 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.881+0000 7fb04159a640 1 -- 172.21.15.164:0/2755953763 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034030960 con 0x7fb0440c8310 2023-08-25T09:49:07.884 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.881+0000 7fb04159a640 1 -- 172.21.15.164:0/2755953763 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034030cf0 con 0x7fb0440c8310 2023-08-25T09:49:07.884 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 -- 172.21.15.164:0/2755953763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0440c8310 msgr2=0x7fb0440b0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.884 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 --2- 172.21.15.164:0/2755953763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0440c8310 0x7fb0440b0fb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0340097a0 tx=0x7fb03402b940 comp rx=0 tx=0).stop 2023-08-25T09:49:07.884 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 -- 172.21.15.164:0/2755953763 shutdown_connections 2023-08-25T09:49:07.885 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 --2- 172.21.15.164:0/2755953763 >> v2:172.21.15.71:3300/0 conn(0x7fb044158d80 0x7fb044159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.885 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 --2- 172.21.15.164:0/2755953763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0440c8310 0x7fb0440b0fb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.885 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 -- 172.21.15.164:0/2755953763 >> 172.21.15.164:0/2755953763 conn(0x7fb04406ae20 msgr2=0x7fb04406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.885 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 -- 172.21.15.164:0/2755953763 shutdown_connections 2023-08-25T09:49:07.885 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.882+0000 7fb04359e640 1 -- 172.21.15.164:0/2755953763 wait complete. 2023-08-25T09:49:07.887 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.883+0000 7fb04359e640 1 Processor -- start 2023-08-25T09:49:07.887 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.884+0000 7fb04359e640 1 -- start start 2023-08-25T09:49:07.887 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.884+0000 7fb04359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 0x7fb0441569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0440ca010 0x7fb0440cd800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb044158d80 0x7fb044155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0441575a0 con 0x7fb0440ca010 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04415bcb0 con 0x7fb0440c8310 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0441eebf0 con 0x7fb044158d80 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 0x7fb0441569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.888 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 0x7fb0441569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45600/0 (socket says 172.21.15.164:45600) 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 -- 172.21.15.164:0/1186653769 learned_addr learned my addr 172.21.15.164:0/1186653769 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb042d9d640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb044158d80 0x7fb044155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb041d9b640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0440ca010 0x7fb0440cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 -- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb044158d80 msgr2=0x7fb044155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb044158d80 0x7fb044155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 -- 172.21.15.164:0/1186653769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0440ca010 msgr2=0x7fb0440cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.889 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0440ca010 0x7fb0440cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb04259c640 1 -- 172.21.15.164:0/1186653769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009400 con 0x7fb0440c8310 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.885+0000 7fb041d9b640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0440ca010 0x7fb0440cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb04259c640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 0x7fb0441569e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb03402bf50 tx=0x7fb0340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03403e070 con 0x7fb0440c8310 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034039ad0 con 0x7fb0440c8310 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034039e60 con 0x7fb0440c8310 2023-08-25T09:49:07.890 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04415bf50 con 0x7fb0440c8310 2023-08-25T09:49:07.891 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440cdd40 con 0x7fb0440c8310 2023-08-25T09:49:07.891 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb03404c3f0 con 0x7fb0440c8310 2023-08-25T09:49:07.891 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.886+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fb03404cb30 con 0x7fb0440c8310 2023-08-25T09:49:07.892 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.887+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04415c140 con 0x7fb0440c8310 2023-08-25T09:49:07.892 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.890+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fb034048070 con 0x7fb0440c8310 2023-08-25T09:49:07.969 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.968+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0d2b548b-2377-4ab5-b195-e5d93c0893b0", "id": 3} v 0) v1 -- 0x7fb044157710 con 0x7fb0440c8310 2023-08-25T09:49:07.977 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.976+0000 7fb032ffd640 1 -- 172.21.15.164:0/1186653769 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d2b548b-2377-4ab5-b195-e5d93c0893b0", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb034034070 con 0x7fb0440c8310 2023-08-25T09:49:07.977 INFO:teuthology.orchestra.run.smithi164.stdout:3 2023-08-25T09:49:07.978 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 msgr2=0x7fb0441569e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 0x7fb0441569e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb03402bf50 tx=0x7fb0340046c0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 shutdown_connections 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb044158d80 0x7fb044155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0440ca010 0x7fb0440cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 --2- 172.21.15.164:0/1186653769 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0440c8310 0x7fb0441569e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 >> 172.21.15.164:0/1186653769 conn(0x7fb04406ae20 msgr2=0x7fb04406b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:07.979 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 shutdown_connections 2023-08-25T09:49:07.980 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:07.977+0000 7fb04359e640 1 -- 172.21.15.164:0/1186653769 wait complete. 2023-08-25T09:49:07.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 953c2978-0014-4223-8f3b-20fc431cfabe 4 2023-08-25T09:49:08.189 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.188+0000 7f91c920e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T09:49:08.190 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.188+0000 7f9def147640 1 Processor -- start 2023-08-25T09:49:08.190 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.188+0000 7f9def147640 1 -- start start 2023-08-25T09:49:08.191 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9def147640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9de8152d80 0x7f9de80c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.191 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9def147640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 0x7f9de80c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.191 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9def147640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de81591c0 con 0x7f9de815c740 2023-08-25T09:49:08.192 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9def147640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de81594a0 con 0x7f9de8152d80 2023-08-25T09:49:08.192 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9def147640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8159780 con 0x7f9de80cacb0 2023-08-25T09:49:08.192 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 0x7f9de80c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.193 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 0x7f9de80c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51320/0 (socket says 172.21.15.164:51320) 2023-08-25T09:49:08.193 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9de7fff640 1 -- 172.21.15.164:0/3039942971 learned_addr learned my addr 172.21.15.164:0/3039942971 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:08.193 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.190+0000 7f9decee3640 1 --2- 172.21.15.164:0/3039942971 >> v2:172.21.15.71:3300/0 conn(0x7f9de8152d80 0x7f9de80c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.193 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.191+0000 7f9de7fff640 1 -- 172.21.15.164:0/3039942971 >> v2:172.21.15.71:3300/0 conn(0x7f9de8152d80 msgr2=0x7f9de80c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:08.193 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.191+0000 7f9de7fff640 1 --2- 172.21.15.164:0/3039942971 >> v2:172.21.15.71:3300/0 conn(0x7f9de8152d80 0x7f9de80c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.193 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.191+0000 7f9de7fff640 1 -- 172.21.15.164:0/3039942971 >> v1:172.21.15.71:6789/0 conn(0x7f9de815c740 legacy=0x7f9de815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.194 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.191+0000 7f9de7fff640 1 -- 172.21.15.164:0/3039942971 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8159a60 con 0x7f9de80cacb0 2023-08-25T09:49:08.194 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.192+0000 7f9de7fff640 1 --2- 172.21.15.164:0/3039942971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 0x7f9de80c98a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009870 tx=0x7f9dd402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829c7422d0a98afd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.192+0000 7f9de77fe640 1 -- 172.21.15.164:0/3039942971 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402f840 con 0x7f9de80cacb0 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.192+0000 7f9de77fe640 1 -- 172.21.15.164:0/3039942971 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd402f9a0 con 0x7f9de80cacb0 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.192+0000 7f9de77fe640 1 -- 172.21.15.164:0/3039942971 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402fd30 con 0x7f9de80cacb0 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 -- 172.21.15.164:0/3039942971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 msgr2=0x7f9de80c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 --2- 172.21.15.164:0/3039942971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 0x7f9de80c98a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009870 tx=0x7f9dd402ba20 comp rx=0 tx=0).stop 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 -- 172.21.15.164:0/3039942971 shutdown_connections 2023-08-25T09:49:08.195 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 --2- 172.21.15.164:0/3039942971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de80cacb0 0x7f9de80c98a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.196 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 --2- 172.21.15.164:0/3039942971 >> v2:172.21.15.71:3300/0 conn(0x7f9de8152d80 0x7f9de80c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.196 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 -- 172.21.15.164:0/3039942971 >> 172.21.15.164:0/3039942971 conn(0x7f9de806b8f0 msgr2=0x7f9de80654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:08.196 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 -- 172.21.15.164:0/3039942971 shutdown_connections 2023-08-25T09:49:08.196 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.193+0000 7f9def147640 1 -- 172.21.15.164:0/3039942971 wait complete. 2023-08-25T09:49:08.196 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.195+0000 7f9def147640 1 Processor -- start 2023-08-25T09:49:08.197 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.195+0000 7f9def147640 1 -- start start 2023-08-25T09:49:08.198 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9def147640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de80cacb0 0x7f9de80c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.198 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9def147640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 0x7f9de815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.198 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9def147640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9de815c740 0x7f9de8069850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.198 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9def147640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de81f2a30 con 0x7f9de815c740 2023-08-25T09:49:08.198 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9def147640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de815a030 con 0x7f9de80cacb0 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9def147640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de815a5f0 con 0x7f9de8152d80 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 0x7f9de815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9decee3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de80cacb0 0x7f9de80c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 0x7f9de815aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51330/0 (socket says 172.21.15.164:51330) 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 -- 172.21.15.164:0/74217601 learned_addr learned my addr 172.21.15.164:0/74217601 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9ded6e4640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9de815c740 0x7f9de8069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.199 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 -- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de80cacb0 msgr2=0x7f9de80c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.200 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de80cacb0 0x7f9de80c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.200 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 -- 172.21.15.164:0/74217601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9de815c740 msgr2=0x7f9de8069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.200 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9de815c740 0x7f9de8069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.200 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.196+0000 7f9de7fff640 1 -- 172.21.15.164:0/74217601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40094d0 con 0x7f9de8152d80 2023-08-25T09:49:08.200 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.197+0000 7f9de7fff640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 0x7f9de815aa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9dd402ba60 tx=0x7f9dd4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:08.201 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.197+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4037c40 con 0x7f9de8152d80 2023-08-25T09:49:08.201 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.197+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd4037da0 con 0x7f9de8152d80 2023-08-25T09:49:08.201 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.197+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40426c0 con 0x7f9de8152d80 2023-08-25T09:49:08.201 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.197+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de80b4190 con 0x7f9de8152d80 2023-08-25T09:49:08.201 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.197+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de80b4430 con 0x7f9de8152d80 2023-08-25T09:49:08.202 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.198+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9dd4042b20 con 0x7f9de8152d80 2023-08-25T09:49:08.202 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.198+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f9dd403c070 con 0x7f9de8152d80 2023-08-25T09:49:08.202 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.198+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de80b1a60 con 0x7f9de8152d80 2023-08-25T09:49:08.202 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.201+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f9dd4047070 con 0x7f9de8152d80 2023-08-25T09:49:08.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.249+0000 7f5a3b042040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-25T09:49:08.252 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.251+0000 7f91c920e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T09:49:08.280 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.278+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "953c2978-0014-4223-8f3b-20fc431cfabe", "id": 4} v 0) v1 -- 0x7f9de8069160 con 0x7f9de8152d80 2023-08-25T09:49:08.287 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.285+0000 7f9de57fa640 1 -- 172.21.15.164:0/74217601 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "953c2978-0014-4223-8f3b-20fc431cfabe", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9dd4042dd0 con 0x7f9de8152d80 2023-08-25T09:49:08.287 INFO:teuthology.orchestra.run.smithi164.stdout:4 2023-08-25T09:49:08.288 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 msgr2=0x7f9de815aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.288 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 0x7f9de815aa80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9dd402ba60 tx=0x7f9dd4004730 comp rx=0 tx=0).stop 2023-08-25T09:49:08.288 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 shutdown_connections 2023-08-25T09:49:08.289 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9de815c740 0x7f9de8069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.289 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de8152d80 0x7f9de815aa80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.289 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 --2- 172.21.15.164:0/74217601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de80cacb0 0x7f9de80c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.289 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 >> 172.21.15.164:0/74217601 conn(0x7f9de806b8f0 msgr2=0x7f9de80654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:08.289 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 shutdown_connections 2023-08-25T09:49:08.289 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.287+0000 7f9def147640 1 -- 172.21.15.164:0/74217601 wait complete. 2023-08-25T09:49:08.304 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 6bcee708-4862-41c2-ae1d-c2eccd7d97b0 5 2023-08-25T09:49:08.307 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.305+0000 7f91c920e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T09:49:08.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.338+0000 7f5a3b042040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-25T09:49:08.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.395+0000 7f5a3b042040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-25T09:49:08.433 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.431+0000 7f91c920e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T09:49:08.504 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.502+0000 7fd9d7091640 1 Processor -- start 2023-08-25T09:49:08.504 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.503+0000 7fd9d7091640 1 -- start start 2023-08-25T09:49:08.505 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.504+0000 7fd9d7091640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd9d00ca010 0x7fd9d00c9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.505 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.504+0000 7fd9d7091640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d00ca550 con 0x7fd9d00c97d0 2023-08-25T09:49:08.506 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.504+0000 7fd9d7091640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d015c390 con 0x7fd9d00c8310 2023-08-25T09:49:08.506 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.504+0000 7fd9d7091640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d015d010 con 0x7fd9d00ca010 2023-08-25T09:49:08.506 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.504+0000 7fd9d608f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd9d00c8310 0x7fd9d00b0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55400/0 (socket says 172.21.15.164:55400) 2023-08-25T09:49:08.506 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.504+0000 7fd9d608f640 1 -- 172.21.15.164:0/3679515210 learned_addr learned my addr 172.21.15.164:0/3679515210 (peer_addr_for_me v1:172.21.15.164:0/0) 2023-08-25T09:49:08.508 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d588e640 1 --2- 172.21.15.164:0/3679515210 >> v2:172.21.15.71:3300/0 conn(0x7fd9d00ca010 0x7fd9d00c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.508 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316644659 0 0) 0x7fd9d015c390 con 0x7fd9d00c8310 2023-08-25T09:49:08.508 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9bc003520 con 0x7fd9d00c8310 2023-08-25T09:49:08.509 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1614119358 0 0) 0x7fd9bc003520 con 0x7fd9d00c8310 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 >> v2:172.21.15.71:3300/0 conn(0x7fd9d00ca010 msgr2=0x7fd9d00c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 --2- 172.21.15.164:0/3679515210 >> v2:172.21.15.71:3300/0 conn(0x7fd9d00ca010 0x7fd9d00c9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 >> v1:172.21.15.71:6789/0 conn(0x7fd9d00c97d0 legacy=0x7fd9d0158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d015c670 con 0x7fd9d00c8310 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fd9cc001670 con 0x7fd9d00c8310 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 >> v1:172.21.15.164:6789/0 conn(0x7fd9d00c8310 legacy=0x7fd9d00b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.510 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9bc0039e0 0x7fd9bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.511 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9bc0047c0 0x7fd9bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.511 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc005180 0x7fd9bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.511 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d015d010 con 0x7fd9bc0047c0 2023-08-25T09:49:08.511 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc00c090 con 0x7fd9bc005180 2023-08-25T09:49:08.511 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.505+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc00c250 con 0x7fd9bc0039e0 2023-08-25T09:49:08.511 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d608f640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9bc0039e0 0x7fd9bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d588e640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9bc0047c0 0x7fd9bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d608f640 1 -- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc005180 msgr2=0x7fd9bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d608f640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc005180 0x7fd9bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d608f640 1 -- 172.21.15.164:0/3679515210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9bc0047c0 msgr2=0x7fd9bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d608f640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9bc0047c0 0x7fd9bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d608f640 1 -- 172.21.15.164:0/3679515210 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc003520 con 0x7fd9bc0039e0 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d588e640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9bc0047c0 0x7fd9bc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:08.512 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.506+0000 7fd9d6890640 1 -- 172.21.15.164:0/3679515210 reap_dead start 2023-08-25T09:49:08.515 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.507+0000 7fd9d608f640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9bc0039e0 0x7fd9bc003e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc001050 tx=0x7fd9bc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e96c759f798e602d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:08.515 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.507+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc005ad0 con 0x7fd9bc0039e0 2023-08-25T09:49:08.515 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.507+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc005c30 con 0x7fd9bc0039e0 2023-08-25T09:49:08.516 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.507+0000 7fd9d508d640 1 -- 172.21.15.164:0/3679515210 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc02cac0 con 0x7fd9bc0039e0 2023-08-25T09:49:08.516 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 -- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9bc0039e0 msgr2=0x7fd9bc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.516 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9bc0039e0 0x7fd9bc003e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc001050 tx=0x7fd9bc0057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.516 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 -- 172.21.15.164:0/3679515210 shutdown_connections 2023-08-25T09:49:08.516 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 --2- 172.21.15.164:0/3679515210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9bc0039e0 0x7fd9bc003e00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.516 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 -- 172.21.15.164:0/3679515210 >> 172.21.15.164:0/3679515210 conn(0x7fd9d006ae20 msgr2=0x7fd9d006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 -- 172.21.15.164:0/3679515210 shutdown_connections 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.508+0000 7fd9d7091640 1 -- 172.21.15.164:0/3679515210 wait complete. 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.510+0000 7fd9d7091640 1 Processor -- start 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.510+0000 7fd9d7091640 1 -- start start 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d7091640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 0x7fd9d00c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d7091640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 0x7fd9d00c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d7091640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9d00c2b00 0x7fd9d014e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.517 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d7091640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d0156600 con 0x7fd9d00c2b00 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d7091640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d00c2f80 con 0x7fd9bc0039e0 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d7091640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d00c20e0 con 0x7fd9d00b2a20 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d608f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 0x7fd9d00c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d588e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 0x7fd9d00c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d588e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 0x7fd9d00c25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51358/0 (socket says 172.21.15.164:51358) 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d608f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 0x7fd9d00c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45638/0 (socket says 172.21.15.164:45638) 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d588e640 1 -- 172.21.15.164:0/943809437 learned_addr learned my addr 172.21.15.164:0/943809437 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:08.518 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.511+0000 7fd9d6890640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9d00c2b00 0x7fd9d014e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d608f640 1 -- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 msgr2=0x7fd9d00c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d608f640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 0x7fd9d00c25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d608f640 1 -- 172.21.15.164:0/943809437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9d00c2b00 msgr2=0x7fd9d014e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d608f640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9d00c2b00 0x7fd9d014e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d608f640 1 -- 172.21.15.164:0/943809437 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0156340 con 0x7fd9bc0039e0 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d588e640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 0x7fd9d00c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:49:08.519 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d608f640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 0x7fd9d00c6ad0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc031000 tx=0x7fd9cc000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc03c070 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc02ca90 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d00cda70 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc02ce60 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.512+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d01eea00 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.513+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd9cc032070 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.513+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fd9cc035e00 con 0x7fd9bc0039e0 2023-08-25T09:49:08.520 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.513+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d01ee580 con 0x7fd9bc0039e0 2023-08-25T09:49:08.521 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.517+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fd9d01ee580 con 0x7fd9bc0039e0 2023-08-25T09:49:08.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.530+0000 7f5a3b042040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-25T09:49:08.593 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.591+0000 7f91c920e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T09:49:08.596 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.595+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6bcee708-4862-41c2-ae1d-c2eccd7d97b0", "id": 5} v 0) v1 -- 0x7fd9d00c2250 con 0x7fd9bc0039e0 2023-08-25T09:49:08.604 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.603+0000 7fd9c6ffd640 1 -- 172.21.15.164:0/943809437 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6bcee708-4862-41c2-ae1d-c2eccd7d97b0", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd9d00c2250 con 0x7fd9bc0039e0 2023-08-25T09:49:08.605 INFO:teuthology.orchestra.run.smithi164.stdout:5 2023-08-25T09:49:08.607 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 msgr2=0x7fd9d00c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.607 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 0x7fd9d00c6ad0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc031000 tx=0x7fd9cc000fe0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.607 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 shutdown_connections 2023-08-25T09:49:08.608 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9d00c2b00 0x7fd9d014e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.608 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9d00b2a20 0x7fd9d00c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.608 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 --2- 172.21.15.164:0/943809437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9bc0039e0 0x7fd9d00c6ad0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.609 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 >> 172.21.15.164:0/943809437 conn(0x7fd9d006ae20 msgr2=0x7fd9d00cc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:08.609 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 shutdown_connections 2023-08-25T09:49:08.609 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.604+0000 7fd9d7091640 1 -- 172.21.15.164:0/943809437 wait complete. 2023-08-25T09:49:08.622 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 300928fe-b078-4c53-9e5f-fc450e8773ed 6 2023-08-25T09:49:08.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.698+0000 7f5a3b042040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-25T09:49:08.703 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.701+0000 7f91c920e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T09:49:08.753 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.751+0000 7f91c920e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T09:49:08.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.814+0000 7f5a3b042040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-25T09:49:08.862 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.861+0000 7f573a796640 1 Processor -- start 2023-08-25T09:49:08.862 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.861+0000 7f573a796640 1 -- start start 2023-08-25T09:49:08.864 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.862+0000 7f573a796640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5734152d80 0x7f57340c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.864 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.862+0000 7f573a796640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f57340cacb0 0x7f57340c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.864 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.862+0000 7f573a796640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57341591c0 con 0x7f573415c740 2023-08-25T09:49:08.864 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.862+0000 7f573a796640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57341594a0 con 0x7f5734152d80 2023-08-25T09:49:08.864 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.862+0000 7f573a796640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5734159780 con 0x7f57340cacb0 2023-08-25T09:49:08.865 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f57340cacb0 0x7f57340c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.865 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f57340cacb0 0x7f57340c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45654/0 (socket says 172.21.15.164:45654) 2023-08-25T09:49:08.865 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 -- 172.21.15.164:0/3152189954 learned_addr learned my addr 172.21.15.164:0/3152189954 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:08.865 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f5733fff640 1 --2- 172.21.15.164:0/3152189954 >> v2:172.21.15.71:3300/0 conn(0x7f5734152d80 0x7f57340c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.866 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 -- 172.21.15.164:0/3152189954 >> v2:172.21.15.71:3300/0 conn(0x7f5734152d80 msgr2=0x7f57340c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:08.866 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 --2- 172.21.15.164:0/3152189954 >> v2:172.21.15.71:3300/0 conn(0x7f5734152d80 0x7f57340c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.866 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 -- 172.21.15.164:0/3152189954 >> v1:172.21.15.71:6789/0 conn(0x7f573415c740 legacy=0x7f573415cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:08.866 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.863+0000 7f572b7fe640 1 -- 172.21.15.164:0/3152189954 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734159a60 con 0x7f57340cacb0 2023-08-25T09:49:08.866 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.865+0000 7f572b7fe640 1 --2- 172.21.15.164:0/3152189954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57340cacb0 0x7f57340c98a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57340b2100 tx=0x7f571c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e4e17c318bcfe49 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:08.867 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.865+0000 7f57337fe640 1 -- 172.21.15.164:0/3152189954 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c030860 con 0x7f57340cacb0 2023-08-25T09:49:08.867 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.865+0000 7f57337fe640 1 -- 172.21.15.164:0/3152189954 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c0309c0 con 0x7f57340cacb0 2023-08-25T09:49:08.867 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.865+0000 7f57337fe640 1 -- 172.21.15.164:0/3152189954 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c030d50 con 0x7f57340cacb0 2023-08-25T09:49:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 -- 172.21.15.164:0/3152189954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57340cacb0 msgr2=0x7f57340c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 --2- 172.21.15.164:0/3152189954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57340cacb0 0x7f57340c98a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57340b2100 tx=0x7f571c02b9a0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 -- 172.21.15.164:0/3152189954 shutdown_connections 2023-08-25T09:49:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 --2- 172.21.15.164:0/3152189954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57340cacb0 0x7f57340c98a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 --2- 172.21.15.164:0/3152189954 >> v2:172.21.15.71:3300/0 conn(0x7f5734152d80 0x7f57340c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.868 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 -- 172.21.15.164:0/3152189954 >> 172.21.15.164:0/3152189954 conn(0x7f573406b8f0 msgr2=0x7f57340654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:08.869 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 -- 172.21.15.164:0/3152189954 shutdown_connections 2023-08-25T09:49:08.869 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.866+0000 7f573a796640 1 -- 172.21.15.164:0/3152189954 wait complete. 2023-08-25T09:49:08.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:08.868+0000 7f5a3b042040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-25T09:49:08.870 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.868+0000 7f573a796640 1 Processor -- start 2023-08-25T09:49:08.870 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.869+0000 7f573a796640 1 -- start start 2023-08-25T09:49:08.873 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f573a796640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 0x7f573415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.873 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f573a796640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5734152d80 0x7f573414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f573a796640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573415c740 0x7f573414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f573a796640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573414d150 con 0x7f573415c740 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f573a796640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57340ce210 con 0x7f5734152d80 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f573a796640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57341eef40 con 0x7f57340cacb0 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f5733fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 0x7f573415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f5733fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 0x7f573415a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51362/0 (socket says 172.21.15.164:51362) 2023-08-25T09:49:08.874 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f5733fff640 1 -- 172.21.15.164:0/2745525707 learned_addr learned my addr 172.21.15.164:0/2745525707 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.870+0000 7f572b7fe640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5734152d80 0x7f573414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5738d33640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573415c740 0x7f573414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5733fff640 1 -- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5734152d80 msgr2=0x7f573414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5733fff640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5734152d80 0x7f573414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5733fff640 1 -- 172.21.15.164:0/2745525707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573415c740 msgr2=0x7f573414dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5733fff640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573415c740 0x7f573414dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.875 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5733fff640 1 -- 172.21.15.164:0/2745525707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0094d0 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f5733fff640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 0x7f573415a2a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5724007ef0 tx=0x7f572400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724014620 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.871+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5724014780 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.872+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57341ef220 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.872+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724014b30 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.872+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573415a7e0 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.872+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f572401e460 con 0x7f57340cacb0 2023-08-25T09:49:08.876 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.872+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f572401d9b0 con 0x7f57340cacb0 2023-08-25T09:49:08.877 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.873+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57340b1a60 con 0x7f57340cacb0 2023-08-25T09:49:08.877 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.875+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f572401dc60 con 0x7f57340cacb0 2023-08-25T09:49:08.919 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:08.917+0000 7f91c920e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T09:49:08.954 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.952+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "300928fe-b078-4c53-9e5f-fc450e8773ed", "id": 6} v 0) v1 -- 0x7f5734069160 con 0x7f57340cacb0 2023-08-25T09:49:08.960 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.959+0000 7f57317fa640 1 -- 172.21.15.164:0/2745525707 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "300928fe-b078-4c53-9e5f-fc450e8773ed", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f572401de00 con 0x7f57340cacb0 2023-08-25T09:49:08.961 INFO:teuthology.orchestra.run.smithi164.stdout:6 2023-08-25T09:49:08.961 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 msgr2=0x7f573415a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:08.961 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 0x7f573415a2a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5724007ef0 tx=0x7f572400e4b0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 shutdown_connections 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5734152d80 0x7f573414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573415c740 0x7f573414dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 --2- 172.21.15.164:0/2745525707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57340cacb0 0x7f573415a2a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 >> 172.21.15.164:0/2745525707 conn(0x7f573406b8f0 msgr2=0x7f57340c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 shutdown_connections 2023-08-25T09:49:08.962 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:08.960+0000 7f573a796640 1 -- 172.21.15.164:0/2745525707 wait complete. 2023-08-25T09:49:08.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 558a2369-e26a-44de-b968-f2ce93b5bdc2 7 2023-08-25T09:49:09.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:09.044+0000 7f5a3b042040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-25T09:49:09.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.204+0000 7fba6ad0f640 1 Processor -- start 2023-08-25T09:49:09.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.205+0000 7fba6ad0f640 1 -- start start 2023-08-25T09:49:09.207 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.206+0000 7fba6ad0f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fba64153310 0x7fba6406a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.206+0000 7fba6ad0f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba640cec00 0x7fba640cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.206+0000 7fba6ad0f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba641590c0 con 0x7fba640c3e00 2023-08-25T09:49:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.206+0000 7fba6ad0f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba64159380 con 0x7fba64153310 2023-08-25T09:49:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.206+0000 7fba6ad0f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba64159660 con 0x7fba640cec00 2023-08-25T09:49:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.206+0000 7fba63fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba640cec00 0x7fba640cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba63fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba640cec00 0x7fba640cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45676/0 (socket says 172.21.15.164:45676) 2023-08-25T09:49:09.209 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba63fff640 1 -- 172.21.15.164:0/1558134129 learned_addr learned my addr 172.21.15.164:0/1558134129 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:09.209 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba68aab640 1 --2- 172.21.15.164:0/1558134129 >> v2:172.21.15.71:3300/0 conn(0x7fba64153310 0x7fba6406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.209 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba63fff640 1 -- 172.21.15.164:0/1558134129 >> v2:172.21.15.71:3300/0 conn(0x7fba64153310 msgr2=0x7fba6406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:09.210 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba63fff640 1 --2- 172.21.15.164:0/1558134129 >> v2:172.21.15.71:3300/0 conn(0x7fba64153310 0x7fba6406a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.210 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba63fff640 1 -- 172.21.15.164:0/1558134129 >> v1:172.21.15.71:6789/0 conn(0x7fba640c3e00 legacy=0x7fba640c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:09.210 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.207+0000 7fba63fff640 1 -- 172.21.15.164:0/1558134129 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64159940 con 0x7fba640cec00 2023-08-25T09:49:09.210 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.208+0000 7fba63fff640 1 --2- 172.21.15.164:0/1558134129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba640cf020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba641536f0 tx=0x7fba5802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c5346ce8d7a7ca server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:09.211 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.208+0000 7fba637fe640 1 -- 172.21.15.164:0/1558134129 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58030860 con 0x7fba640cec00 2023-08-25T09:49:09.211 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba637fe640 1 -- 172.21.15.164:0/1558134129 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba580309c0 con 0x7fba640cec00 2023-08-25T09:49:09.211 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba637fe640 1 -- 172.21.15.164:0/1558134129 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58030d50 con 0x7fba640cec00 2023-08-25T09:49:09.211 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1558134129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 msgr2=0x7fba640cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.211 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1558134129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba640cf020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba641536f0 tx=0x7fba5802b9a0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.211 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1558134129 shutdown_connections 2023-08-25T09:49:09.212 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1558134129 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba640cf020 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.213 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1558134129 >> v2:172.21.15.71:3300/0 conn(0x7fba64153310 0x7fba6406a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.213 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1558134129 >> 172.21.15.164:0/1558134129 conn(0x7fba64065910 msgr2=0x7fba64065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:09.213 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1558134129 shutdown_connections 2023-08-25T09:49:09.213 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.209+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1558134129 wait complete. 2023-08-25T09:49:09.215 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.211+0000 7fba6ad0f640 1 Processor -- start 2023-08-25T09:49:09.215 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.211+0000 7fba6ad0f640 1 -- start start 2023-08-25T09:49:09.216 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba6ad0f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba640c3e00 0x7fba64149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.216 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba6ad0f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba641f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.216 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba6ad0f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba64153310 0x7fba640b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.216 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba6ad0f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba641ef810 con 0x7fba64153310 2023-08-25T09:49:09.216 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba6ad0f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6415c250 con 0x7fba640cec00 2023-08-25T09:49:09.216 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba6ad0f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640b1550 con 0x7fba640c3e00 2023-08-25T09:49:09.217 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba63fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba641f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.217 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba68aab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba640c3e00 0x7fba64149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.217 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba63fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba641f1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45682/0 (socket says 172.21.15.164:45682) 2023-08-25T09:49:09.217 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.212+0000 7fba63fff640 1 -- 172.21.15.164:0/1740572322 learned_addr learned my addr 172.21.15.164:0/1740572322 (peer_addr_for_me v2:172.21.15.164:0/0) 2023-08-25T09:49:09.217 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba692ac640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba64153310 0x7fba640b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.217 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba63fff640 1 -- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba640c3e00 msgr2=0x7fba64149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba63fff640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba640c3e00 0x7fba64149330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba63fff640 1 -- 172.21.15.164:0/1740572322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba64153310 msgr2=0x7fba640b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba63fff640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba64153310 0x7fba640b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba63fff640 1 -- 172.21.15.164:0/1740572322 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba580094d0 con 0x7fba640cec00 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba692ac640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba64153310 0x7fba640b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba63fff640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba641f1fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba58000c00 tx=0x7fba58004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:09.218 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58039c90 con 0x7fba640cec00 2023-08-25T09:49:09.219 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58039df0 con 0x7fba640cec00 2023-08-25T09:49:09.219 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58043620 con 0x7fba640cec00 2023-08-25T09:49:09.219 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.213+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba640c5d30 con 0x7fba640cec00 2023-08-25T09:49:09.219 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.214+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6415c4d0 con 0x7fba640cec00 2023-08-25T09:49:09.219 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.214+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fba58043a80 con 0x7fba640cec00 2023-08-25T09:49:09.219 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.214+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fba5803e070 con 0x7fba640cec00 2023-08-25T09:49:09.220 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.215+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64159e60 con 0x7fba640cec00 2023-08-25T09:49:09.220 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.217+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fba58034070 con 0x7fba640cec00 2023-08-25T09:49:09.296 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.294+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "558a2369-e26a-44de-b968-f2ce93b5bdc2", "id": 7} v 0) v1 -- 0x7fba64069160 con 0x7fba640cec00 2023-08-25T09:49:09.302 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.301+0000 7fba617fa640 1 -- 172.21.15.164:0/1740572322 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "558a2369-e26a-44de-b968-f2ce93b5bdc2", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fba5804d530 con 0x7fba640cec00 2023-08-25T09:49:09.302 INFO:teuthology.orchestra.run.smithi164.stdout:7 2023-08-25T09:49:09.303 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 msgr2=0x7fba641f1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.303 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba641f1fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba58000c00 tx=0x7fba58004730 comp rx=0 tx=0).stop 2023-08-25T09:49:09.304 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 shutdown_connections 2023-08-25T09:49:09.304 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba640cec00 0x7fba641f1fd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.305 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fba64153310 0x7fba640b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.305 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 --2- 172.21.15.164:0/1740572322 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba640c3e00 0x7fba64149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.305 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.302+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 >> 172.21.15.164:0/1740572322 conn(0x7fba64065910 msgr2=0x7fba64150af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:09.305 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.303+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 shutdown_connections 2023-08-25T09:49:09.306 INFO:teuthology.orchestra.run.smithi164.stderr:2023-08-25T09:49:09.303+0000 7fba6ad0f640 1 -- 172.21.15.164:0/1740572322 wait complete. 2023-08-25T09:49:09.319 INFO:tasks.ceph.osd.0:Restarting daemon 2023-08-25T09:49:09.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-08-25T09:49:09.329 INFO:tasks.ceph.osd.0:Started 2023-08-25T09:49:09.330 INFO:tasks.ceph.osd.1:Restarting daemon 2023-08-25T09:49:09.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-08-25T09:49:09.338 INFO:tasks.ceph.osd.1:Started 2023-08-25T09:49:09.339 INFO:tasks.ceph.osd.2:Restarting daemon 2023-08-25T09:49:09.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-08-25T09:49:09.344 INFO:tasks.ceph.osd.2:Started 2023-08-25T09:49:09.345 INFO:tasks.ceph.osd.3:Restarting daemon 2023-08-25T09:49:09.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-08-25T09:49:09.353 INFO:tasks.ceph.osd.3:Started 2023-08-25T09:49:09.353 INFO:tasks.ceph.osd.4:Restarting daemon 2023-08-25T09:49:09.353 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-08-25T09:49:09.369 INFO:tasks.ceph.osd.4:Started 2023-08-25T09:49:09.370 INFO:tasks.ceph.osd.5:Restarting daemon 2023-08-25T09:49:09.371 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-08-25T09:49:09.379 INFO:tasks.ceph.osd.5:Started 2023-08-25T09:49:09.379 INFO:tasks.ceph.osd.6:Restarting daemon 2023-08-25T09:49:09.380 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-08-25T09:49:09.385 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:09.384+0000 7f91c920e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T09:49:09.388 INFO:tasks.ceph.osd.6:Started 2023-08-25T09:49:09.388 INFO:tasks.ceph.osd.7:Restarting daemon 2023-08-25T09:49:09.389 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-08-25T09:49:09.394 INFO:tasks.ceph.osd.7:Started 2023-08-25T09:49:09.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:09.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:09.571+0000 7f5a3b042040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-25T09:49:09.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.603+0000 7f7daa586640 1 Processor -- start 2023-08-25T09:49:09.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.604+0000 7f7daa586640 1 -- start start 2023-08-25T09:49:09.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7daa586640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da40c4510 0x7f7da40cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7daa586640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da41590c0 con 0x7f7da4153310 2023-08-25T09:49:09.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7daa586640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da4159380 con 0x7f7da40cec00 2023-08-25T09:49:09.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7daa586640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4159660 con 0x7f7da40c4510 2023-08-25T09:49:09.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7da3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7da4153310 0x7f7da406a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51156/0 (socket says 172.21.15.71:51156) 2023-08-25T09:49:09.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7da3fff640 1 -- 172.21.15.71:0/1272863053 learned_addr learned my addr 172.21.15.71:0/1272863053 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:09.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.605+0000 7f7da8b23640 1 --2- 172.21.15.71:0/1272863053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da40c4510 0x7f7da40cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022254668 0 0) 0x7f7da41590c0 con 0x7f7da4153310 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8c003520 con 0x7f7da4153310 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500323154 0 0) 0x7f7da4159380 con 0x7f7da40cec00 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da41590c0 con 0x7f7da40cec00 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da8b23640 1 -- 172.21.15.71:0/1272863053 >> v1:172.21.15.164:6789/0 conn(0x7f7da40cec00 legacy=0x7f7da40c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da8b23640 1 -- 172.21.15.71:0/1272863053 >> v1:172.21.15.71:6789/0 conn(0x7f7da4153310 legacy=0x7f7da406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.606+0000 7f7da8b23640 1 -- 172.21.15.71:0/1272863053 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4159940 con 0x7f7da40c4510 2023-08-25T09:49:09.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.607+0000 7f7da8b23640 1 --2- 172.21.15.71:0/1272863053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da40c4510 0x7f7da40cf020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009dc0 tx=0x7f7d9c02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f660d7d2c68352c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:09.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.607+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c02cd50 con 0x7f7da40c4510 2023-08-25T09:49:09.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.607+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c034440 con 0x7f7da40c4510 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.607+0000 7f7da2ffd640 1 -- 172.21.15.71:0/1272863053 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c0347d0 con 0x7f7da40c4510 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 -- 172.21.15.71:0/1272863053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da40c4510 msgr2=0x7f7da40cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 --2- 172.21.15.71:0/1272863053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da40c4510 0x7f7da40cf020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009dc0 tx=0x7f7d9c02bfa0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 -- 172.21.15.71:0/1272863053 shutdown_connections 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 --2- 172.21.15.71:0/1272863053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da40c4510 0x7f7da40cf020 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 -- 172.21.15.71:0/1272863053 >> 172.21.15.71:0/1272863053 conn(0x7f7da4065910 msgr2=0x7f7da4065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:09.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 -- 172.21.15.71:0/1272863053 shutdown_connections 2023-08-25T09:49:09.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.608+0000 7f7daa586640 1 -- 172.21.15.71:0/1272863053 wait complete. 2023-08-25T09:49:09.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.610+0000 7f7daa586640 1 Processor -- start 2023-08-25T09:49:09.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.610+0000 7f7daa586640 1 -- start start 2023-08-25T09:49:09.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7daa586640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 0x7f7da414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7daa586640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7da40cec00 0x7f7da41f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7daa586640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da4153310 0x7f7da40c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:09.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7daa586640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da414ac40 con 0x7f7da40c4510 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7daa586640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da41f2260 con 0x7f7da40cec00 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7daa586640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da40c5e30 con 0x7f7da4153310 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 0x7f7da414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 0x7f7da414bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45618/0 (socket says 172.21.15.71:45618) 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 -- 172.21.15.71:0/1489804211 learned_addr learned my addr 172.21.15.71:0/1489804211 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da8b23640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da4153310 0x7f7da40c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da37fe640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7da40cec00 0x7f7da41f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:09.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 -- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da4153310 msgr2=0x7f7da40c58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da4153310 0x7f7da40c58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 -- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7da40cec00 msgr2=0x7f7da41f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7da40cec00 0x7f7da41f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da3fff640 1 -- 172.21.15.71:0/1489804211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c009a70 con 0x7f7da40c4510 2023-08-25T09:49:09.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.611+0000 7f7da8b23640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da4153310 0x7f7da40c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:09.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7da3fff640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 0x7f7da414bb00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c410 tx=0x7f7d9400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:09.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94005ce0 con 0x7f7da40c4510 2023-08-25T09:49:09.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d94005e40 con 0x7f7da40c4510 2023-08-25T09:49:09.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d940072f0 con 0x7f7da40c4510 2023-08-25T09:49:09.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da414c8d0 con 0x7f7da40c4510 2023-08-25T09:49:09.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da414a3a0 con 0x7f7da40c4510 2023-08-25T09:49:09.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.612+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7d94012400 con 0x7f7da40c4510 2023-08-25T09:49:09.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.613+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f7d94012d80 con 0x7f7da40c4510 2023-08-25T09:49:09.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.613+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da406bf70 con 0x7f7da40c4510 2023-08-25T09:49:09.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.617+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7f7d94021060 con 0x7f7da40c4510 2023-08-25T09:49:09.666 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:09.664+0000 7f91c920e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T09:49:09.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.681+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7da40c4b00 con 0x7f7da40c4510 2023-08-25T09:49:09.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.682+0000 7f7da0ff9640 1 -- 172.21.15.71:0/1489804211 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f7d94017070 con 0x7f7da40c4510 2023-08-25T09:49:09.683 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:09.684 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:09.296445+0000","last_up_change":"0.000000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:09.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 msgr2=0x7f7da414bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 0x7f7da414bb00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c410 tx=0x7f7d9400c8d0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 shutdown_connections 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7da40cec00 0x7f7da41f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7da4153310 0x7f7da40c58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 --2- 172.21.15.71:0/1489804211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7da40c4510 0x7f7da414bb00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 >> 172.21.15.71:0/1489804211 conn(0x7f7da4065910 msgr2=0x7f7da406b860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 shutdown_connections 2023-08-25T09:49:09.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:09.683+0000 7f7daa586640 1 -- 172.21.15.71:0/1489804211 wait complete. 2023-08-25T09:49:09.701 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-08-25T09:49:09.701 INFO:tasks.ceph:Waiting for OSDs to come up 2023-08-25T09:49:09.732 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:09.730+0000 7f91c920e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T09:49:09.791 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:09.790+0000 7f91c920e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T09:49:09.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:09.795+0000 7f5a3b042040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-25T09:49:09.854 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:09.852+0000 7f91c920e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T09:49:09.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:09.858+0000 7f5a3b042040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-25T09:49:09.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:09.911+0000 7f5a3b042040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-25T09:49:09.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:09.971+0000 7f5a3b042040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-25T09:49:10.061 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T09:49:10.059+0000 7f9f09815500 -1 Falling back to public interface 2023-08-25T09:49:10.062 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T09:49:10.060+0000 7f62f4821500 -1 Falling back to public interface 2023-08-25T09:49:10.063 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T09:49:10.062+0000 7f69f9f6b500 -1 Falling back to public interface 2023-08-25T09:49:10.070 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T09:49:10.069+0000 7f69f8f12640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6819/0: (98) Address already in use 2023-08-25T09:49:10.073 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T09:49:10.072+0000 7f2760374500 -1 Falling back to public interface 2023-08-25T09:49:10.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T09:49:10.083+0000 7fb292941500 -1 Falling back to public interface 2023-08-25T09:49:10.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T09:49:10.098+0000 7f89b20f4500 -1 Falling back to public interface 2023-08-25T09:49:10.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T09:49:10.105+0000 7f27f4a19500 -1 Falling back to public interface 2023-08-25T09:49:10.114 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T09:49:10.113+0000 7fbb78213500 -1 Falling back to public interface 2023-08-25T09:49:10.161 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.159+0000 7f91c920e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T09:49:10.220 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.218+0000 7f91c920e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T09:49:10.274 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.272+0000 7f91c920e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T09:49:10.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.281+0000 7f5a3b042040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-25T09:49:10.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.340+0000 7f5a3b042040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-25T09:49:10.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.401+0000 7f5a3b042040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-25T09:49:10.439 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.437+0000 7f91c920e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T09:49:10.490 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.488+0000 7f91c920e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T09:49:10.506 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:10.553 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.551+0000 7f91c920e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T09:49:10.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.566+0000 7f5a3b042040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-25T09:49:10.608 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:10.607+0000 7f91c920e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T09:49:10.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.616+0000 7f5a3b042040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-25T09:49:10.678 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.677+0000 7fe7f5778640 1 Processor -- start 2023-08-25T09:49:10.679 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.677+0000 7fe7f5778640 1 -- start start 2023-08-25T09:49:10.680 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.678+0000 7fe7f5778640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe7f0150f70 0x7fe7f00cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:10.681 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7f5778640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f0157b40 con 0x7fe7f00c8820 2023-08-25T09:49:10.681 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7f5778640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f0151350 con 0x7fe7f00c40f0 2023-08-25T09:49:10.681 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7e7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe7f00c8820 0x7fe7f00c39e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44280/0 (socket says 172.21.15.71:44280) 2023-08-25T09:49:10.681 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7e7fff640 1 -- 172.21.15.71:0/4080165306 learned_addr learned my addr 172.21.15.71:0/4080165306 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:10.681 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7f5778640 1 -- 172.21.15.71:0/4080165306 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f0156f30 con 0x7fe7f0150f70 2023-08-25T09:49:10.681 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7eeffd640 1 --2- 172.21.15.71:0/4080165306 >> v2:172.21.15.164:3300/0 conn(0x7fe7f0150f70 0x7fe7f00cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:10.682 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7ee7fc640 1 -- 172.21.15.71:0/4080165306 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503293318 0 0) 0x7fe7f0157b40 con 0x7fe7f00c8820 2023-08-25T09:49:10.682 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7ee7fc640 1 -- 172.21.15.71:0/4080165306 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc003520 con 0x7fe7f00c8820 2023-08-25T09:49:10.682 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7eeffd640 1 -- 172.21.15.71:0/4080165306 >> v1:172.21.15.164:6789/0 conn(0x7fe7f00c40f0 legacy=0x7fe7f015a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:10.682 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7eeffd640 1 -- 172.21.15.71:0/4080165306 >> v1:172.21.15.71:6789/0 conn(0x7fe7f00c8820 legacy=0x7fe7f00c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:10.682 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.679+0000 7fe7eeffd640 1 -- 172.21.15.71:0/4080165306 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f0157210 con 0x7fe7f0150f70 2023-08-25T09:49:10.683 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.680+0000 7fe7eeffd640 1 --2- 172.21.15.71:0/4080165306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f0150f70 0x7fe7f00cc490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00097a0 tx=0x7fe7e002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc741c1fe3f85657 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:10.683 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7ee7fc640 1 -- 172.21.15.71:0/4080165306 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0030610 con 0x7fe7f0150f70 2023-08-25T09:49:10.683 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7ee7fc640 1 -- 172.21.15.71:0/4080165306 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e0030770 con 0x7fe7f0150f70 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7ee7fc640 1 -- 172.21.15.71:0/4080165306 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0030610 con 0x7fe7f0150f70 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7f5778640 1 -- 172.21.15.71:0/4080165306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f0150f70 msgr2=0x7fe7f00cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7f5778640 1 --2- 172.21.15.71:0/4080165306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f0150f70 0x7fe7f00cc490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00097a0 tx=0x7fe7e002b960 comp rx=0 tx=0).stop 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7f5778640 1 -- 172.21.15.71:0/4080165306 shutdown_connections 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.681+0000 7fe7f5778640 1 --2- 172.21.15.71:0/4080165306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f0150f70 0x7fe7f00cc490 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.682+0000 7fe7f5778640 1 -- 172.21.15.71:0/4080165306 >> 172.21.15.71:0/4080165306 conn(0x7fe7f0065580 msgr2=0x7fe7f0065960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:10.684 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.682+0000 7fe7f5778640 1 -- 172.21.15.71:0/4080165306 shutdown_connections 2023-08-25T09:49:10.685 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.682+0000 7fe7f5778640 1 -- 172.21.15.71:0/4080165306 wait complete. 2023-08-25T09:49:10.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.683+0000 7f5a3b042040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-25T09:49:10.686 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.683+0000 7fe7f5778640 1 Processor -- start 2023-08-25T09:49:10.686 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.683+0000 7fe7f5778640 1 -- start start 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7f5778640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f00c40f0 0x7fe7f0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7f5778640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f00c8820 0x7fe7f01525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7f5778640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 0x7fe7f01ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7f5778640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f01537d0 con 0x7fe7f0150f70 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7f5778640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f0151c00 con 0x7fe7f00c8820 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7f5778640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f01560f0 con 0x7fe7f00c40f0 2023-08-25T09:49:10.687 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.684+0000 7fe7ef7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 0x7fe7f01ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 0x7fe7f01ef480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53650/0 (socket says 172.21.15.71:53650) 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 -- 172.21.15.71:0/1201390801 learned_addr learned my addr 172.21.15.71:0/1201390801 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7e7fff640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f00c8820 0x7fe7f01525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7eeffd640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f00c40f0 0x7fe7f0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 -- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f00c40f0 msgr2=0x7fe7f0152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f00c40f0 0x7fe7f0152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:10.688 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 -- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f00c8820 msgr2=0x7fe7f01525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:10.689 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f00c8820 0x7fe7f01525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:10.689 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 -- 172.21.15.71:0/1201390801 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0009400 con 0x7fe7f0150f70 2023-08-25T09:49:10.689 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7e7fff640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f00c8820 0x7fe7f01525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:10.689 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7ef7fe640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 0x7fe7f01ef480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c760 tx=0x7fe7e800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:10.690 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8006d50 con 0x7fe7f0150f70 2023-08-25T09:49:10.690 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800ce10 con 0x7fe7f0150f70 2023-08-25T09:49:10.690 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f015a010 con 0x7fe7f0150f70 2023-08-25T09:49:10.690 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.685+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8003bb0 con 0x7fe7f0150f70 2023-08-25T09:49:10.690 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.686+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f0159d30 con 0x7fe7f0150f70 2023-08-25T09:49:10.690 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.687+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 2) v1 ==== 44122+0+0 (secure 0 0 0) 0x7fe7e8003d10 con 0x7fe7f0150f70 2023-08-25T09:49:10.691 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.687+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f0069160 con 0x7fe7f0150f70 2023-08-25T09:49:10.691 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.687+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe7e8048920 con 0x7fe7f0150f70 2023-08-25T09:49:10.691 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.689+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72429 (secure 0 0 0) 0x7fe7e8019310 con 0x7fe7f0150f70 2023-08-25T09:49:10.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2023-08-25T09:49:10.738+0000 7f5a3b042040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-25T09:49:10.756 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.754+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7f01577a0 con 0x7fe7f0150f70 2023-08-25T09:49:10.757 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.756+0000 7fe7e77fe640 1 -- 172.21.15.71:0/1201390801 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe7e8011b70 con 0x7fe7f0150f70 2023-08-25T09:49:10.757 INFO:teuthology.misc.health.smithi071.stdout: 2023-08-25T09:49:10.758 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:09.296445+0000","last_up_change":"0.000000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 msgr2=0x7fe7f01ef480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 0x7fe7f01ef480 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c760 tx=0x7fe7e800cc20 comp rx=0 tx=0).stop 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 shutdown_connections 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7f00c8820 0x7fe7f01525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7f0150f70 0x7fe7f01ef480 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 --2- 172.21.15.71:0/1201390801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7f00c40f0 0x7fe7f0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:10.759 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 >> 172.21.15.71:0/1201390801 conn(0x7fe7f0065580 msgr2=0x7fe7f00c59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:10.760 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 shutdown_connections 2023-08-25T09:49:10.760 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:10.757+0000 7fe7f5778640 1 -- 172.21.15.71:0/1201390801 wait complete. 2023-08-25T09:49:10.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-08-25T09:49:11.651 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:11.650+0000 7f916a32a640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-08-25T09:49:11.654 INFO:tasks.ceph.mgr.y.smithi071.stderr:2023-08-25T09:49:11.653+0000 7f916a32a640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-08-25T09:49:13.399 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T09:49:13.397+0000 7f9f09815500 -1 osd.2 0 log_to_monitors true 2023-08-25T09:49:13.430 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T09:49:13.428+0000 7f69f9f6b500 -1 osd.0 0 log_to_monitors true 2023-08-25T09:49:13.616 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T09:49:13.615+0000 7f62f4821500 -1 osd.1 0 log_to_monitors true 2023-08-25T09:49:13.678 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T09:49:13.676+0000 7f2760374500 -1 osd.3 0 log_to_monitors true 2023-08-25T09:49:13.695 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T09:49:13.693+0000 7fb292941500 -1 osd.4 0 log_to_monitors true 2023-08-25T09:49:13.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T09:49:13.716+0000 7f27f4a19500 -1 osd.7 0 log_to_monitors true 2023-08-25T09:49:13.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T09:49:13.727+0000 7fbb78213500 -1 osd.6 0 log_to_monitors true 2023-08-25T09:49:13.892 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T09:49:13.890+0000 7f89b20f4500 -1 osd.5 0 log_to_monitors true 2023-08-25T09:49:16.066 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T09:49:16.064+0000 7f69f5ef7640 -1 osd.0 0 waiting for initial osdmap 2023-08-25T09:49:16.086 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T09:49:16.084+0000 7f69edd17640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:16.106 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T09:49:16.104+0000 7f62f0163640 -1 osd.1 0 waiting for initial osdmap 2023-08-25T09:49:16.122 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T09:49:16.121+0000 7f62e7f83640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:16.139 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T09:49:16.138+0000 7f9f05250640 -1 osd.2 0 waiting for initial osdmap 2023-08-25T09:49:16.156 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T09:49:16.154+0000 7f9efd070640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:16.744 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T09:49:16.743+0000 7f275c302640 -1 osd.3 0 waiting for initial osdmap 2023-08-25T09:49:16.762 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T09:49:16.760+0000 7f2754122640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:17.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T09:49:17.260+0000 7f89ae080640 -1 osd.5 0 waiting for initial osdmap 2023-08-25T09:49:17.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T09:49:17.264+0000 7fbb743e8640 -1 osd.6 0 waiting for initial osdmap 2023-08-25T09:49:17.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T09:49:17.277+0000 7f89a5ea0640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:17.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T09:49:17.282+0000 7fbb6b9f2640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:17.317 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T09:49:17.313+0000 7f27f03e3640 -1 osd.7 0 waiting for initial osdmap 2023-08-25T09:49:17.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T09:49:17.325+0000 7f27e8203640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:17.327 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T09:49:17.326+0000 7fb28f0e3640 -1 osd.4 0 waiting for initial osdmap 2023-08-25T09:49:17.338 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T09:49:17.337+0000 7fb2866ed640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-25T09:49:17.577 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:17.768 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.766+0000 7fcaa8d22640 1 Processor -- start 2023-08-25T09:49:17.768 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.766+0000 7fcaa8d22640 1 -- start start 2023-08-25T09:49:17.769 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.768+0000 7fcaa8d22640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcaa4153160 0x7fcaa40c9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.770 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.768+0000 7fcaa8d22640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40cb460 con 0x7fcaa4069850 2023-08-25T09:49:17.770 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.768+0000 7fcaa8d22640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40c3bf0 con 0x7fcaa415cc10 2023-08-25T09:49:17.770 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.768+0000 7fcaa8d22640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa40b1dd0 con 0x7fcaa4153160 2023-08-25T09:49:17.772 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.768+0000 7fcaa3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcaa4069850 0x7fcaa4069ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44296/0 (socket says 172.21.15.71:44296) 2023-08-25T09:49:17.772 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.768+0000 7fcaa3fff640 1 -- 172.21.15.71:0/936906 learned_addr learned my addr 172.21.15.71:0/936906 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:17.772 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa37fe640 1 --2- 172.21.15.71:0/936906 >> v2:172.21.15.164:3300/0 conn(0x7fcaa4153160 0x7fcaa40c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.772 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836489171 0 0) 0x7fcaa40cb460 con 0x7fcaa4069850 2023-08-25T09:49:17.772 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca90003520 con 0x7fcaa4069850 2023-08-25T09:49:17.772 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2800105682 0 0) 0x7fca90003520 con 0x7fcaa4069850 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 >> v2:172.21.15.164:3300/0 conn(0x7fcaa4153160 msgr2=0x7fcaa40c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 --2- 172.21.15.71:0/936906 >> v2:172.21.15.164:3300/0 conn(0x7fcaa4153160 0x7fcaa40c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 >> v1:172.21.15.164:6789/0 conn(0x7fcaa415cc10 legacy=0x7fcaa40c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa415d100 con 0x7fcaa4069850 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fca9c001670 con 0x7fcaa4069850 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 >> v1:172.21.15.71:6789/0 conn(0x7fcaa4069850 legacy=0x7fcaa4069ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca900039e0 0x7fca90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.773 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fca90004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca90005180 0x7fca9000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9000c060 con 0x7fca900047c0 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9000c1d0 con 0x7fca900039e0 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.769+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9000c390 con 0x7fca90005180 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fca90004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa37fe640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca900039e0 0x7fca90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.774 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa3fff640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca90005180 0x7fca9000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.775 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa2ffd640 1 -- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca90005180 msgr2=0x7fca9000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:17.775 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca90005180 0x7fca9000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.775 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa2ffd640 1 -- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca900039e0 msgr2=0x7fca90003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:17.775 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca900039e0 0x7fca90003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.775 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa2ffd640 1 -- 172.21.15.71:0/936906 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca90003520 con 0x7fca900047c0 2023-08-25T09:49:17.775 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.770+0000 7fcaa3fff640 1 -- 172.21.15.71:0/936906 reap_dead start 2023-08-25T09:49:17.777 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.771+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fca90004c40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fca8c001720 tx=0x7fca900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16206db0fd81a083 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:17.777 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0052d0 con 0x7fca900047c0 2023-08-25T09:49:17.777 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c02e050 con 0x7fca900047c0 2023-08-25T09:49:17.777 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa27fc640 1 -- 172.21.15.71:0/936906 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c003830 con 0x7fca900047c0 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 -- 172.21.15.71:0/936906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 msgr2=0x7fca90004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fca90004c40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fca8c001720 tx=0x7fca900057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 -- 172.21.15.71:0/936906 shutdown_connections 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/936906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fca90004c40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 -- 172.21.15.71:0/936906 >> 172.21.15.71:0/936906 conn(0x7fcaa4065580 msgr2=0x7fcaa4065960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 -- 172.21.15.71:0/936906 shutdown_connections 2023-08-25T09:49:17.778 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.772+0000 7fcaa8d22640 1 -- 172.21.15.71:0/936906 wait complete. 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.774+0000 7fcaa8d22640 1 Processor -- start 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.774+0000 7fcaa8d22640 1 -- start start 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa8d22640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fcaa4157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa8d22640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcaa41501c0 0x7fcaa40cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa8d22640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcaa4158540 0x7fcaa41589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa8d22640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa41580b0 con 0x7fca900047c0 2023-08-25T09:49:17.779 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa8d22640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa414fd90 con 0x7fcaa41501c0 2023-08-25T09:49:17.780 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa8d22640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa4157ad0 con 0x7fcaa4158540 2023-08-25T09:49:17.780 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.775+0000 7fcaa37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fcaa4157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.780 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fcaa4157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53706/0 (socket says 172.21.15.71:53706) 2023-08-25T09:49:17.780 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 -- 172.21.15.71:0/3396292527 learned_addr learned my addr 172.21.15.71:0/3396292527 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:17.781 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcaa41501c0 0x7fcaa40cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.781 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa3fff640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcaa4158540 0x7fcaa41589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 -- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcaa4158540 msgr2=0x7fcaa41589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcaa4158540 0x7fcaa41589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 -- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcaa41501c0 msgr2=0x7fcaa40cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcaa41501c0 0x7fcaa40cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 -- 172.21.15.71:0/3396292527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4158f00 con 0x7fca900047c0 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcaa41501c0 0x7fcaa40cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:17.782 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.776+0000 7fcaa37fe640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fcaa4157080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca94007f90 tx=0x7fca94011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.777+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9400e030 con 0x7fca900047c0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.777+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca94011e90 con 0x7fca900047c0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.777+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa41f2280 con 0x7fca900047c0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.777+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca940168c0 con 0x7fca900047c0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.777+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa4157d40 con 0x7fca900047c0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.778+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa404e420 con 0x7fca900047c0 2023-08-25T09:49:17.783 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.779+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 4) v1 ==== 88285+0+0 (secure 0 0 0) 0x7fca94016a20 con 0x7fca900047c0 2023-08-25T09:49:17.784 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.780+0000 7fca9bfff640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fca84069fc0 0x7fca8406c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:17.784 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.780+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3351+0+0 (secure 0 0 0) 0x7fca9400f070 con 0x7fca900047c0 2023-08-25T09:49:17.785 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.781+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fca84069fc0 0x7fca8406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:17.785 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.781+0000 7fcaa2ffd640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fca84069fc0 0x7fca8406c480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fca8c033000 tx=0x7fca8c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:17.787 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.785+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fca94054c90 con 0x7fca900047c0 2023-08-25T09:49:17.982 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.980+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaa40c9440 con 0x7fca900047c0 2023-08-25T09:49:17.983 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.982+0000 7fca9bfff640 1 -- 172.21.15.71:0/3396292527 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8964 (secure 0 0 0) 0x7fca940582b0 con 0x7fca900047c0 2023-08-25T09:49:17.984 INFO:teuthology.misc.health.smithi071.stdout: 2023-08-25T09:49:17.984 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":14,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:17.067099+0000","last_up_change":"2023-08-25T09:49:17.067099+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:17.986 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.984+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fca84069fc0 msgr2=0x7fca8406c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.987 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.984+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fca84069fc0 0x7fca8406c480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fca8c033000 tx=0x7fca8c009030 comp rx=0 tx=0).stop 2023-08-25T09:49:17.987 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 msgr2=0x7fcaa4157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:17.987 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fcaa4157080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fca94007f90 tx=0x7fca94011660 comp rx=0 tx=0).stop 2023-08-25T09:49:17.987 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 shutdown_connections 2023-08-25T09:49:17.987 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcaa41501c0 0x7fcaa40cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.987 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fca84069fc0 0x7fca8406c480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.988 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcaa4158540 0x7fcaa41589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.988 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 --2- 172.21.15.71:0/3396292527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca900047c0 0x7fcaa4157080 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:17.988 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 >> 172.21.15.71:0/3396292527 conn(0x7fcaa4065580 msgr2=0x7fcaa40b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:17.988 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 shutdown_connections 2023-08-25T09:49:17.988 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:17.985+0000 7fcaa8d22640 1 -- 172.21.15.71:0/3396292527 wait complete. 2023-08-25T09:49:18.000 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-08-25T09:49:18.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:49:18.717 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:49:24.807 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:24.993 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.991+0000 7f9d0a184640 1 Processor -- start 2023-08-25T09:49:24.994 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.991+0000 7f9d0a184640 1 -- start start 2023-08-25T09:49:24.995 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.992+0000 7f9d0a184640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9d041510b0 0x7f9d041577b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:24.995 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.992+0000 7f9d0a184640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9d040c57e0 0x7f9d040c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:24.995 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.992+0000 7f9d0a184640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d04157320 con 0x7f9d040c38b0 2023-08-25T09:49:24.995 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.992+0000 7f9d0a184640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d04070fc0 con 0x7f9d041510b0 2023-08-25T09:49:24.996 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.992+0000 7f9d0a184640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d040c3cf0 con 0x7f9d040c57e0 2023-08-25T09:49:24.996 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9d041510b0 0x7f9d041577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:24.997 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9d041510b0 0x7f9d041577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47134/0 (socket says 172.21.15.71:47134) 2023-08-25T09:49:24.997 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 -- 172.21.15.71:0/3282004552 learned_addr learned my addr 172.21.15.71:0/3282004552 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:24.997 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 -- 172.21.15.71:0/3282004552 >> v2:172.21.15.164:3300/0 conn(0x7f9d040c57e0 msgr2=0x7f9d040c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:24.997 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 --2- 172.21.15.71:0/3282004552 >> v2:172.21.15.164:3300/0 conn(0x7f9d040c57e0 0x7f9d040c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:24.997 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 -- 172.21.15.71:0/3282004552 >> v1:172.21.15.71:6789/0 conn(0x7f9d040c38b0 legacy=0x7f9d040c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:24.997 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.993+0000 7f9d037fe640 1 -- 172.21.15.71:0/3282004552 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d040c6150 con 0x7f9d041510b0 2023-08-25T09:49:24.998 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.994+0000 7f9d037fe640 1 --2- 172.21.15.71:0/3282004552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d041510b0 0x7f9d041577b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d040667b0 tx=0x7f9cf402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989e216308cdb7b5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:24.999 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d027fc640 1 -- 172.21.15.71:0/3282004552 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4030860 con 0x7f9d041510b0 2023-08-25T09:49:24.999 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d027fc640 1 -- 172.21.15.71:0/3282004552 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf40309c0 con 0x7f9d041510b0 2023-08-25T09:49:24.999 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d027fc640 1 -- 172.21.15.71:0/3282004552 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4030d50 con 0x7f9d041510b0 2023-08-25T09:49:24.999 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 -- 172.21.15.71:0/3282004552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d041510b0 msgr2=0x7f9d041577b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:24.999 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 --2- 172.21.15.71:0/3282004552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d041510b0 0x7f9d041577b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9d040667b0 tx=0x7f9cf402b9a0 comp rx=0 tx=0).stop 2023-08-25T09:49:24.999 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 -- 172.21.15.71:0/3282004552 shutdown_connections 2023-08-25T09:49:25.000 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 --2- 172.21.15.71:0/3282004552 >> v2:172.21.15.164:3300/0 conn(0x7f9d040c57e0 0x7f9d040c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.000 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 --2- 172.21.15.71:0/3282004552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d041510b0 0x7f9d041577b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.000 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 -- 172.21.15.71:0/3282004552 >> 172.21.15.71:0/3282004552 conn(0x7f9d040ca3e0 msgr2=0x7f9d040ca7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:25.000 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 -- 172.21.15.71:0/3282004552 shutdown_connections 2023-08-25T09:49:25.000 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.995+0000 7f9d0a184640 1 -- 172.21.15.71:0/3282004552 wait complete. 2023-08-25T09:49:25.002 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.997+0000 7f9d0a184640 1 Processor -- start 2023-08-25T09:49:25.002 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.997+0000 7f9d0a184640 1 -- start start 2023-08-25T09:49:25.002 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d0a184640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d040c38b0 0x7f9d041489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d0a184640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 0x7f9d040c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d0a184640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d041510b0 0x7f9d04142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d0a184640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04152120 con 0x7f9d040c57e0 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d0a184640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04151530 con 0x7f9d041510b0 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d0a184640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d041541e0 con 0x7f9d040c38b0 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d02ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 0x7f9d040c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.003 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d02ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 0x7f9d040c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47142/0 (socket says 172.21.15.71:47142) 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.998+0000 7f9d02ffd640 1 -- 172.21.15.71:0/1318197242 learned_addr learned my addr 172.21.15.71:0/1318197242 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d03fff640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d041510b0 0x7f9d04142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d02ffd640 1 -- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d040c38b0 msgr2=0x7f9d041489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d037fe640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d040c38b0 0x7f9d041489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d02ffd640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d040c38b0 0x7f9d041489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d02ffd640 1 -- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d041510b0 msgr2=0x7f9d04142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.004 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d02ffd640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d041510b0 0x7f9d04142980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d02ffd640 1 -- 172.21.15.71:0/1318197242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf40094d0 con 0x7f9d040c57e0 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d03fff640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d041510b0 0x7f9d04142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d02ffd640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 0x7f9d040c9930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c3c0 tx=0x7f9cf800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80052d0 con 0x7f9d040c57e0 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf8005450 con 0x7f9d040c57e0 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8007260 con 0x7f9d040c57e0 2023-08-25T09:49:25.005 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d04149060 con 0x7f9d040c57e0 2023-08-25T09:49:25.006 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:24.999+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d04151a60 con 0x7f9d040c57e0 2023-08-25T09:49:25.006 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.001+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d04069160 con 0x7f9d040c57e0 2023-08-25T09:49:25.007 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.002+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9cf80073c0 con 0x7f9d040c57e0 2023-08-25T09:49:25.007 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.003+0000 7f9cebfff640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9cf006a050 0x7f9cf006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.007 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.003+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9cf80875b0 con 0x7f9d040c57e0 2023-08-25T09:49:25.007 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.003+0000 7f9d037fe640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9cf006a050 0x7f9cf006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.008 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.004+0000 7f9d037fe640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9cf006a050 0x7f9cf006c510 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4036000 tx=0x7f9cf403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:25.010 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.008+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9cf8053020 con 0x7f9d040c57e0 2023-08-25T09:49:25.200 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.198+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d040c5280 con 0x7f9d040c57e0 2023-08-25T09:49:25.202 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.200+0000 7f9cebfff640 1 -- 172.21.15.71:0/1318197242 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f9cf8056640 con 0x7f9d040c57e0 2023-08-25T09:49:25.203 INFO:teuthology.misc.health.smithi071.stdout: 2023-08-25T09:49:25.203 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":18,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:21.290299+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:25.205 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.202+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9cf006a050 msgr2=0x7f9cf006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.205 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.202+0000 7f9d0a184640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9cf006a050 0x7f9cf006c510 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4036000 tx=0x7f9cf403b040 comp rx=0 tx=0).stop 2023-08-25T09:49:25.205 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.202+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 msgr2=0x7f9d040c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.205 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.202+0000 7f9d0a184640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 0x7f9d040c9930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800c3c0 tx=0x7f9cf800c880 comp rx=0 tx=0).stop 2023-08-25T09:49:25.206 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 shutdown_connections 2023-08-25T09:49:25.206 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d041510b0 0x7f9d04142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.206 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9cf006a050 0x7f9cf006c510 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.206 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d040c57e0 0x7f9d040c9930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.207 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 --2- 172.21.15.71:0/1318197242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d040c38b0 0x7f9d041489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.207 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 >> 172.21.15.71:0/1318197242 conn(0x7f9d040ca3e0 msgr2=0x7f9d04069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:25.207 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 shutdown_connections 2023-08-25T09:49:25.207 INFO:teuthology.misc.health.smithi071.stderr:2023-08-25T09:49:25.203+0000 7f9d0a184640 1 -- 172.21.15.71:0/1318197242 wait complete. 2023-08-25T09:49:25.220 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-08-25T09:49:25.220 INFO:tasks.ceph:Creating RBD pool 2023-08-25T09:49:25.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd pool create rbd 8 2023-08-25T09:49:25.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.427+0000 7f8cac4fa640 1 Processor -- start 2023-08-25T09:49:25.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.427+0000 7f8cac4fa640 1 -- start start 2023-08-25T09:49:25.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.428+0000 7f8cac4fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca40c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.428+0000 7f8cac4fa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8ca40c3a40 0x7f8ca406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.428+0000 7f8cac4fa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40c1dc0 con 0x7f8ca41509d0 2023-08-25T09:49:25.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.428+0000 7f8cac4fa640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca4157a80 con 0x7f8ca40c3a40 2023-08-25T09:49:25.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.428+0000 7f8cac4fa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca406cf70 con 0x7f8ca40c3660 2023-08-25T09:49:25.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8ca9a95640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8ca40c3a40 0x7f8ca406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8caaa97640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8ca41509d0 0x7f8ca4150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51204/0 (socket says 172.21.15.71:51204) 2023-08-25T09:49:25.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8caaa97640 1 -- 172.21.15.71:0/2959773242 learned_addr learned my addr 172.21.15.71:0/2959773242 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:25.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8caa296640 1 --2- 172.21.15.71:0/2959773242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca40c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8ca9a95640 1 -- 172.21.15.71:0/2959773242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 msgr2=0x7f8ca40c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:25.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8ca9a95640 1 --2- 172.21.15.71:0/2959773242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca40c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8ca9a95640 1 -- 172.21.15.71:0/2959773242 >> v1:172.21.15.71:6789/0 conn(0x7f8ca41509d0 legacy=0x7f8ca4150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8ca9a95640 1 -- 172.21.15.71:0/2959773242 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca4156e00 con 0x7f8ca40c3a40 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.429+0000 7f8ca9294640 1 -- 172.21.15.71:0/2959773242 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602600560 0 0) 0x7f8ca40c1dc0 con 0x7f8ca41509d0 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.430+0000 7f8ca9a95640 1 --2- 172.21.15.71:0/2959773242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca406ca30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c940097a0 tx=0x7f8c9402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3799e5c1d339fe1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8ca9294640 1 -- 172.21.15.71:0/2959773242 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94030800 con 0x7f8ca40c3a40 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8ca9294640 1 -- 172.21.15.71:0/2959773242 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c94030960 con 0x7f8ca40c3a40 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8ca9294640 1 -- 172.21.15.71:0/2959773242 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94030cf0 con 0x7f8ca40c3a40 2023-08-25T09:49:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 -- 172.21.15.71:0/2959773242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 msgr2=0x7f8ca406ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/2959773242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca406ca30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c940097a0 tx=0x7f8c9402b940 comp rx=0 tx=0).stop 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 -- 172.21.15.71:0/2959773242 shutdown_connections 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/2959773242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca406ca30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/2959773242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca40c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 -- 172.21.15.71:0/2959773242 >> 172.21.15.71:0/2959773242 conn(0x7f8ca40654f0 msgr2=0x7f8ca4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.431+0000 7f8cac4fa640 1 -- 172.21.15.71:0/2959773242 shutdown_connections 2023-08-25T09:49:25.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.432+0000 7f8cac4fa640 1 -- 172.21.15.71:0/2959773242 wait complete. 2023-08-25T09:49:25.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.433+0000 7f8cac4fa640 1 Processor -- start 2023-08-25T09:49:25.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.433+0000 7f8cac4fa640 1 -- start start 2023-08-25T09:49:25.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8cac4fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca4070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8cac4fa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca41594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8cac4fa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ca41509d0 0x7f8ca41f1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8cac4fa640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca4065ec0 con 0x7f8ca40c3a40 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8cac4fa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca41585d0 con 0x7f8ca41509d0 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8cac4fa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca41f1e40 con 0x7f8ca40c3660 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8ca9a95640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca41594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8ca9a95640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca41594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47156/0 (socket says 172.21.15.71:47156) 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.434+0000 7f8ca9a95640 1 -- 172.21.15.71:0/3461778255 learned_addr learned my addr 172.21.15.71:0/3461778255 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8caa296640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca4070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8caaa97640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ca41509d0 0x7f8ca41f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8ca9a95640 1 -- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 msgr2=0x7f8ca4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8ca9a95640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca4070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8ca9a95640 1 -- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ca41509d0 msgr2=0x7f8ca41f1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8ca9a95640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ca41509d0 0x7f8ca41f1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:25.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8ca9a95640 1 -- 172.21.15.71:0/3461778255 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94009400 con 0x7f8ca40c3a40 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8ca9a95640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca41594a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8c9402bf50 tx=0x7f8c940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9403e070 con 0x7f8ca40c3a40 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c94039ad0 con 0x7f8ca40c3a40 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c94039e60 con 0x7f8ca40c3a40 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.435+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca406f770 con 0x7f8ca40c3a40 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.436+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca406fd30 con 0x7f8ca40c3a40 2023-08-25T09:49:25.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.437+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca40c29c0 con 0x7f8ca40c3a40 2023-08-25T09:49:25.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.438+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8c9404b400 con 0x7f8ca40c3a40 2023-08-25T09:49:25.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.438+0000 7f8c9affd640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8c8806a050 0x7f8c8806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:25.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.439+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8c940aa690 con 0x7f8ca40c3a40 2023-08-25T09:49:25.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.439+0000 7f8caa296640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8c8806a050 0x7f8c8806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:25.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.439+0000 7f8caa296640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8c8806a050 0x7f8c8806c510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0006fd0 tx=0x7f8ca0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:25.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.443+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8c94076220 con 0x7f8ca40c3a40 2023-08-25T09:49:25.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:25.632+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f8ca4069160 con 0x7f8ca40c3a40 2023-08-25T09:49:26.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.505+0000 7f8c9affd640 1 -- 172.21.15.71:0/3461778255 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f8c94079840 con 0x7f8ca40c3a40 2023-08-25T09:49:26.508 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'rbd' created 2023-08-25T09:49:26.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.508+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8c8806a050 msgr2=0x7f8c8806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:26.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.508+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8c8806a050 0x7f8c8806c510 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0006fd0 tx=0x7f8ca0008040 comp rx=0 tx=0).stop 2023-08-25T09:49:26.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 msgr2=0x7f8ca41594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:26.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca41594a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8c9402bf50 tx=0x7f8c940046c0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 shutdown_connections 2023-08-25T09:49:26.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ca41509d0 0x7f8ca41f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8c8806a050 0x7f8c8806c510 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca40c3a40 0x7f8ca41594a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 --2- 172.21.15.71:0/3461778255 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ca40c3660 0x7f8ca4070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.509+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 >> 172.21.15.71:0/3461778255 conn(0x7f8ca40654f0 msgr2=0x7f8ca41565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:26.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.510+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 shutdown_connections 2023-08-25T09:49:26.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.510+0000 7f8cac4fa640 1 -- 172.21.15.71:0/3461778255 wait complete. 2023-08-25T09:49:26.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-08-25T09:49:26.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.737+0000 7f0ba1b39640 1 Processor -- start 2023-08-25T09:49:26.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.738+0000 7f0ba1b39640 1 -- start start 2023-08-25T09:49:26.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.739+0000 7f0ba1b39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 0x7f0b9c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.739+0000 7f0ba1b39640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0b9c0c4560 0x7f0b9c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.739+0000 7f0ba1b39640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9c06eb60 con 0x7f0b9c070cf0 2023-08-25T09:49:26.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.739+0000 7f0ba1b39640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9c06ca30 con 0x7f0b9c0c4560 2023-08-25T09:49:26.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.739+0000 7f0ba1b39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06cf90 con 0x7f0b9c152d80 2023-08-25T09:49:26.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0b9c070cf0 0x7f0b9c15c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51206/0 (socket says 172.21.15.71:51206) 2023-08-25T09:49:26.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9bfff640 1 -- 172.21.15.71:0/1331128923 learned_addr learned my addr 172.21.15.71:0/1331128923 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:26.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9b7fe640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 0x7f0b9c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b93fff640 1 --2- 172.21.15.71:0/1331128923 >> v2:172.21.15.164:3300/0 conn(0x7f0b9c0c4560 0x7f0b9c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110200418 0 0) 0x7f0b9c06eb60 con 0x7f0b9c070cf0 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b88003520 con 0x7f0b9c070cf0 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2530856752 0 0) 0x7f0b88003520 con 0x7f0b9c070cf0 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 msgr2=0x7f0b9c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 0x7f0b9c06e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 >> v2:172.21.15.164:3300/0 conn(0x7f0b9c0c4560 msgr2=0x7f0b9c0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:26.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 --2- 172.21.15.71:0/1331128923 >> v2:172.21.15.164:3300/0 conn(0x7f0b9c0c4560 0x7f0b9c0c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c158fd0 con 0x7f0b9c070cf0 2023-08-25T09:49:26.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0b940019f0 con 0x7f0b9c070cf0 2023-08-25T09:49:26.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 >> v1:172.21.15.71:6789/0 conn(0x7f0b9c070cf0 legacy=0x7f0b9c15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:26.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.740+0000 7f0b9affd640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880039e0 0x7f0b88003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9affd640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 0x7f0b880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9affd640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b88004d00 0x7f0b88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06ca30 con 0x7f0b88004d00 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c06cf90 con 0x7f0b880039e0 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8800b9d0 con 0x7f0b9c152d80 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b88004d00 0x7f0b88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b93fff640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 0x7f0b880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9b7fe640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880039e0 0x7f0b88003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9bfff640 1 -- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 msgr2=0x7f0b880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:26.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c152d80 0x7f0b880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9bfff640 1 -- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880039e0 msgr2=0x7f0b88003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:26.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b880039e0 0x7f0b88003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.741+0000 7f0b9bfff640 1 -- 172.21.15.71:0/1331128923 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88003520 con 0x7f0b88004d00 2023-08-25T09:49:26.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.742+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b88004d00 0x7f0b88005180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400a860 tx=0x7f0b880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7eee181a8762390 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:26.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94034630 con 0x7f0b88004d00 2023-08-25T09:49:26.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9402f070 con 0x7f0b88004d00 2023-08-25T09:49:26.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0b9affd640 1 -- 172.21.15.71:0/1331128923 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b940349f0 con 0x7f0b88004d00 2023-08-25T09:49:26.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0ba1b39640 1 -- 172.21.15.71:0/1331128923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b88004d00 msgr2=0x7f0b88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:26.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/1331128923 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b88004d00 0x7f0b88005180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400a860 tx=0x7f0b880057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0b9bfff640 1 -- 172.21.15.71:0/1331128923 reap_dead start 2023-08-25T09:49:26.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0ba1b39640 1 -- 172.21.15.71:0/1331128923 shutdown_connections 2023-08-25T09:49:26.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0ba1b39640 1 -- 172.21.15.71:0/1331128923 >> 172.21.15.71:0/1331128923 conn(0x7f0b9c0654f0 msgr2=0x7f0b9c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:26.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0ba1b39640 1 -- 172.21.15.71:0/1331128923 shutdown_connections 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.743+0000 7f0ba1b39640 1 -- 172.21.15.71:0/1331128923 wait complete. 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.745+0000 7f0ba1b39640 1 Processor -- start 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.745+0000 7f0ba1b39640 1 -- start start 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0ba1b39640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b88004d00 0x7f0b9c1ee810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0ba1b39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0c1880 0x7f0b9c15bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0ba1b39640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 0x7f0b9c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0ba1b39640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c1569e0 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0ba1b39640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c1eed50 con 0x7f0b88004d00 2023-08-25T09:49:26.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0ba1b39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9c15a3c0 con 0x7f0b9c0c1880 2023-08-25T09:49:26.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0b9bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 0x7f0b9c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0b9bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 0x7f0b9c15c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47174/0 (socket says 172.21.15.71:47174) 2023-08-25T09:49:26.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.746+0000 7f0b9bfff640 1 -- 172.21.15.71:0/3177688710 learned_addr learned my addr 172.21.15.71:0/3177688710 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:26.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b93fff640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0c1880 0x7f0b9c15bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9b7fe640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b88004d00 0x7f0b9c1ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9bfff640 1 -- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0c1880 msgr2=0x7f0b9c15bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0c1880 0x7f0b9c15bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9bfff640 1 -- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b88004d00 msgr2=0x7f0b9c1ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b88004d00 0x7f0b9c1ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9bfff640 1 -- 172.21.15.71:0/3177688710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c15afa0 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b9bfff640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 0x7f0b9c15c2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b9402cba0 tx=0x7f0b94001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b94001fd0 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b94034640 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b940349b0 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.747+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9c15b280 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.748+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c15b840 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.748+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b9c15ba30 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.750+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0b94031030 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.751+0000 7f0b98ff9640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0b7c06a050 0x7f0b7c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.751+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f0b940aafb0 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.751+0000 7f0b9b7fe640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0b7c06a050 0x7f0b7c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.752+0000 7f0b9b7fe640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0b7c06a050 0x7f0b7c06c510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009990 tx=0x7f0b8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.755+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0b9407a3e0 con 0x7f0b9c0c1cc0 2023-08-25T09:49:26.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:26.943+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0b9c15ba30 con 0x7f0b9c0c1cc0 2023-08-25T09:49:27.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.516+0000 7f0b98ff9640 1 -- 172.21.15.71:0/3177688710 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0b94079f70 con 0x7f0b9c0c1cc0 2023-08-25T09:49:27.521 INFO:teuthology.orchestra.run.smithi071.stderr:enabled application 'rbd' on pool 'rbd' 2023-08-25T09:49:27.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.523+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0b7c06a050 msgr2=0x7f0b7c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:27.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.523+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0b7c06a050 0x7f0b7c06c510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009990 tx=0x7f0b8c008040 comp rx=0 tx=0).stop 2023-08-25T09:49:27.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.523+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 msgr2=0x7f0b9c15c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:27.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 0x7f0b9c15c2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0b9402cba0 tx=0x7f0b94001d80 comp rx=0 tx=0).stop 2023-08-25T09:49:27.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 shutdown_connections 2023-08-25T09:49:27.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0b7c06a050 0x7f0b7c06c510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b9c0c1cc0 0x7f0b9c15c2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b9c0c1880 0x7f0b9c15bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 --2- 172.21.15.71:0/3177688710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b88004d00 0x7f0b9c1ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 >> 172.21.15.71:0/3177688710 conn(0x7f0b9c0654f0 msgr2=0x7f0b9c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:27.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 shutdown_connections 2023-08-25T09:49:27.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.524+0000 7f0ba1b39640 1 -- 172.21.15.71:0/3177688710 wait complete. 2023-08-25T09:49:27.553 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-08-25T09:49:27.553 INFO:tasks.ceph.mds.a:Restarting daemon 2023-08-25T09:49:27.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-08-25T09:49:27.560 INFO:tasks.ceph.mds.a:Started 2023-08-25T09:49:27.560 INFO:tasks.ceph.mds.c:Restarting daemon 2023-08-25T09:49:27.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-08-25T09:49:27.566 INFO:tasks.ceph.mds.c:Started 2023-08-25T09:49:27.567 INFO:tasks.ceph.mds.e:Restarting daemon 2023-08-25T09:49:27.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-08-25T09:49:27.573 INFO:tasks.ceph.mds.e:Started 2023-08-25T09:49:27.573 INFO:tasks.ceph.mds.b:Restarting daemon 2023-08-25T09:49:27.573 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-08-25T09:49:27.582 INFO:tasks.ceph.mds.b:Started 2023-08-25T09:49:27.583 INFO:tasks.ceph.mds.d:Restarting daemon 2023-08-25T09:49:27.583 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-08-25T09:49:27.591 INFO:tasks.ceph.mds.d:Started 2023-08-25T09:49:27.592 INFO:tasks.ceph.mds.f:Restarting daemon 2023-08-25T09:49:27.592 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-08-25T09:49:27.599 INFO:tasks.ceph.mds.f:Started 2023-08-25T09:49:27.600 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-08-25T09:49:27.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:27.723 INFO:tasks.ceph.mds.a.smithi071.stdout:starting mds.a at 2023-08-25T09:49:27.724 INFO:tasks.ceph.mds.c.smithi071.stdout:starting mds.c at 2023-08-25T09:49:27.735 INFO:tasks.ceph.mds.e.smithi071.stdout:starting mds.e at 2023-08-25T09:49:27.751 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2023-08-25T09:49:27.755 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2023-08-25T09:49:27.763 INFO:tasks.ceph.mds.f.smithi164.stdout:starting mds.f at 2023-08-25T09:49:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.794+0000 7f01785d6640 1 Processor -- start 2023-08-25T09:49:27.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.794+0000 7f01785d6640 1 -- start start 2023-08-25T09:49:27.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f01785d6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0170152d80 0x7f017006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:27.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f01785d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01700c4560 0x7f01700c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:27.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f01785d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017006eb60 con 0x7f0170070cf0 2023-08-25T09:49:27.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f01785d6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017006ca30 con 0x7f0170152d80 2023-08-25T09:49:27.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f01785d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017006cf90 con 0x7f01700c4560 2023-08-25T09:49:27.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0170152d80 0x7f017006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:27.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0170152d80 0x7f017006e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47222/0 (socket says 172.21.15.71:47222) 2023-08-25T09:49:27.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 -- 172.21.15.71:0/504442435 learned_addr learned my addr 172.21.15.71:0/504442435 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:27.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 -- 172.21.15.71:0/504442435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01700c4560 msgr2=0x7f01700c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:27.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0175b71640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01700c4560 0x7f01700c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:27.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01700c4560 0x7f01700c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 -- 172.21.15.71:0/504442435 >> v1:172.21.15.164:6789/0 conn(0x7f0170070cf0 legacy=0x7f017015c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:27.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0176372640 1 -- 172.21.15.71:0/504442435 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0170158fd0 con 0x7f0170152d80 2023-08-25T09:49:27.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.796+0000 7f0175b71640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01700c4560 0x7f01700c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:27.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f0176372640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f017006e620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f016c009870 tx=0x7f016c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49aab2daa4ded4d2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:27.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f0175370640 1 -- 172.21.15.71:0/504442435 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0317c0 con 0x7f0170152d80 2023-08-25T09:49:27.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f0175370640 1 -- 172.21.15.71:0/504442435 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c031920 con 0x7f0170152d80 2023-08-25T09:49:27.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f0175370640 1 -- 172.21.15.71:0/504442435 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c031cb0 con 0x7f0170152d80 2023-08-25T09:49:27.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f01785d6640 1 -- 172.21.15.71:0/504442435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 msgr2=0x7f017006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:27.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f01785d6640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f017006e620 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f016c009870 tx=0x7f016c02b860 comp rx=0 tx=0).stop 2023-08-25T09:49:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f01785d6640 1 -- 172.21.15.71:0/504442435 shutdown_connections 2023-08-25T09:49:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f01785d6640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01700c4560 0x7f01700c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.798+0000 7f01785d6640 1 --2- 172.21.15.71:0/504442435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f017006e620 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.799+0000 7f01785d6640 1 -- 172.21.15.71:0/504442435 >> 172.21.15.71:0/504442435 conn(0x7f01700654f0 msgr2=0x7f0170065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:27.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.799+0000 7f01785d6640 1 -- 172.21.15.71:0/504442435 shutdown_connections 2023-08-25T09:49:27.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.799+0000 7f01785d6640 1 -- 172.21.15.71:0/504442435 wait complete. 2023-08-25T09:49:27.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.800+0000 7f01785d6640 1 Processor -- start 2023-08-25T09:49:27.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.800+0000 7f01785d6640 1 -- start start 2023-08-25T09:49:27.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.801+0000 7f01785d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0170070cf0 0x7f017014b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:27.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.801+0000 7f01785d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01700c4560 0x7f017014ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:27.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.801+0000 7f01785d6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f0170155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:27.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.801+0000 7f01785d6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017014b870 con 0x7f0170152d80 2023-08-25T09:49:27.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.801+0000 7f01785d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01700c34b0 con 0x7f01700c4560 2023-08-25T09:49:27.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.801+0000 7f01785d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017014c580 con 0x7f0170070cf0 2023-08-25T09:49:27.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f0170155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:27.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f0170155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47236/0 (socket says 172.21.15.71:47236) 2023-08-25T09:49:27.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 -- 172.21.15.71:0/994034258 learned_addr learned my addr 172.21.15.71:0/994034258 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:27.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0175b71640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01700c4560 0x7f017014ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:27.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176372640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0170070cf0 0x7f017014b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:27.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 -- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0170070cf0 msgr2=0x7f017014b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:27.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0170070cf0 0x7f017014b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 -- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01700c4560 msgr2=0x7f017014ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:27.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01700c4560 0x7f017014ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:27.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 -- 172.21.15.71:0/994034258 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0094d0 con 0x7f0170152d80 2023-08-25T09:49:27.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176372640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0170070cf0 0x7f017014b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:27.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.802+0000 7f0176b73640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f0170155dd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f016800b970 tx=0x7f016800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:27.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.803+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800cb40 con 0x7f0170152d80 2023-08-25T09:49:27.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.803+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016800cca0 con 0x7f0170152d80 2023-08-25T09:49:27.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.803+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0168014680 con 0x7f0170152d80 2023-08-25T09:49:27.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.803+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01700c1610 con 0x7f0170152d80 2023-08-25T09:49:27.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.803+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01700c3700 con 0x7f0170152d80 2023-08-25T09:49:27.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.804+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017014c6f0 con 0x7f0170152d80 2023-08-25T09:49:27.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.805+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f016801d3b0 con 0x7f0170152d80 2023-08-25T09:49:27.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.806+0000 7f0166ffd640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f015406a050 0x7f015406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:27.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.806+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0168087f90 con 0x7f0170152d80 2023-08-25T09:49:27.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.806+0000 7f0176372640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f015406a050 0x7f015406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:27.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.807+0000 7f0176372640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f015406a050 0x7f015406c510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f016c036000 tx=0x7f016c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:27.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.811+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f01680ba990 con 0x7f0170152d80 2023-08-25T09:49:27.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.995+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0170069160 con 0x7f0170152d80 2023-08-25T09:49:28.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:27.998+0000 7f0166ffd640 1 -- 172.21.15.71:0/994034258 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14649 (secure 0 0 0) 0x7f0168053a80 con 0x7f0170152d80 2023-08-25T09:49:28.002 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:28.002 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":20,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:27.502262+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:28.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f015406a050 msgr2=0x7f015406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f015406a050 0x7f015406c510 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f016c036000 tx=0x7f016c03d040 comp rx=0 tx=0).stop 2023-08-25T09:49:28.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 msgr2=0x7f0170155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f0170155dd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f016800b970 tx=0x7f016800be30 comp rx=0 tx=0).stop 2023-08-25T09:49:28.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 shutdown_connections 2023-08-25T09:49:28.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01700c4560 0x7f017014ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f015406a050 0x7f015406c510 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0170152d80 0x7f0170155dd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 --2- 172.21.15.71:0/994034258 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0170070cf0 0x7f017014b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.000+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 >> 172.21.15.71:0/994034258 conn(0x7f01700654f0 msgr2=0x7f01700658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:28.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.001+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 shutdown_connections 2023-08-25T09:49:28.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.001+0000 7f01785d6640 1 -- 172.21.15.71:0/994034258 wait complete. 2023-08-25T09:49:28.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T09:49:28.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:49:28.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.232+0000 7fa3d8e57640 1 Processor -- start 2023-08-25T09:49:28.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.232+0000 7fa3d8e57640 1 -- start start 2023-08-25T09:49:28.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d8e57640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa3d4153310 0x7fa3d406a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d8e57640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d40cec00 0x7fa3d40cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d8e57640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa3d40c3e00 0x7fa3d40c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d8e57640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d40c47b0 con 0x7fa3d4153310 2023-08-25T09:49:28.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d8e57640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d41590c0 con 0x7fa3d40c3e00 2023-08-25T09:49:28.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d8e57640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4159380 con 0x7fa3d40cec00 2023-08-25T09:49:28.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa3d4153310 0x7fa3d406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa3d4153310 0x7fa3d406a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47248/0 (socket says 172.21.15.71:47248) 2023-08-25T09:49:28.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.234+0000 7fa3d259c640 1 -- 172.21.15.71:0/3753950741 learned_addr learned my addr 172.21.15.71:0/3753950741 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d2d9d640 1 --2- 172.21.15.71:0/3753950741 >> v2:172.21.15.164:3300/0 conn(0x7fa3d40c3e00 0x7fa3d40c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d259c640 1 -- 172.21.15.71:0/3753950741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d40cec00 msgr2=0x7fa3d40cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d1d9b640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d40cec00 0x7fa3d40cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d259c640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d40cec00 0x7fa3d40cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d259c640 1 -- 172.21.15.71:0/3753950741 >> v2:172.21.15.164:3300/0 conn(0x7fa3d40c3e00 msgr2=0x7fa3d40c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d259c640 1 --2- 172.21.15.71:0/3753950741 >> v2:172.21.15.164:3300/0 conn(0x7fa3d40c3e00 0x7fa3d40c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d259c640 1 -- 172.21.15.71:0/3753950741 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4159660 con 0x7fa3d4153310 2023-08-25T09:49:28.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d2d9d640 1 --2- 172.21.15.71:0/3753950741 >> v2:172.21.15.164:3300/0 conn(0x7fa3d40c3e00 0x7fa3d40c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:28.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.235+0000 7fa3d1d9b640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d40cec00 0x7fa3d40cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:28.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.236+0000 7fa3d259c640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4153310 0x7fa3d406a620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40cc590 tx=0x7fa3c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de2aa8c60cbf4e4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:28.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.236+0000 7fa3d159a640 1 -- 172.21.15.71:0/3753950741 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c402cb00 con 0x7fa3d4153310 2023-08-25T09:49:28.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.236+0000 7fa3d159a640 1 -- 172.21.15.71:0/3753950741 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c402cc60 con 0x7fa3d4153310 2023-08-25T09:49:28.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.236+0000 7fa3d159a640 1 -- 172.21.15.71:0/3753950741 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40343f0 con 0x7fa3d4153310 2023-08-25T09:49:28.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 -- 172.21.15.71:0/3753950741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4153310 msgr2=0x7fa3d406a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4153310 0x7fa3d406a620 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40cc590 tx=0x7fa3c402bf60 comp rx=0 tx=0).stop 2023-08-25T09:49:28.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 -- 172.21.15.71:0/3753950741 shutdown_connections 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/3753950741 >> v2:172.21.15.164:3300/0 conn(0x7fa3d40c3e00 0x7fa3d40c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d40cec00 0x7fa3d40cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/3753950741 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4153310 0x7fa3d406a620 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 -- 172.21.15.71:0/3753950741 >> 172.21.15.71:0/3753950741 conn(0x7fa3d4065910 msgr2=0x7fa3d4065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 -- 172.21.15.71:0/3753950741 shutdown_connections 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.237+0000 7fa3d8e57640 1 -- 172.21.15.71:0/3753950741 wait complete. 2023-08-25T09:49:28.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.239+0000 7fa3d8e57640 1 Processor -- start 2023-08-25T09:49:28.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.239+0000 7fa3d8e57640 1 -- start start 2023-08-25T09:49:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d8e57640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 0x7fa3d41516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d8e57640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3d40cec00 0x7fa3d41f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d8e57640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d4153310 0x7fa3d41f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d8e57640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40cdbd0 con 0x7fa3d40c3e00 2023-08-25T09:49:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d8e57640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d415b290 con 0x7fa3d40cec00 2023-08-25T09:49:28.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d8e57640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40cd550 con 0x7fa3d4153310 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 0x7fa3d41516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 0x7fa3d41516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47250/0 (socket says 172.21.15.71:47250) 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.240+0000 7fa3d259c640 1 -- 172.21.15.71:0/290269875 learned_addr learned my addr 172.21.15.71:0/290269875 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d259c640 1 -- 172.21.15.71:0/290269875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d4153310 msgr2=0x7fa3d41f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d259c640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d4153310 0x7fa3d41f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d259c640 1 -- 172.21.15.71:0/290269875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3d40cec00 msgr2=0x7fa3d41f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d259c640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3d40cec00 0x7fa3d41f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d259c640 1 -- 172.21.15.71:0/290269875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4009b30 con 0x7fa3d40c3e00 2023-08-25T09:49:28.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d259c640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 0x7fa3d41516e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4000f80 tx=0x7fa3c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:28.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c403e450 con 0x7fa3d40c3e00 2023-08-25T09:49:28.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c403e5f0 con 0x7fa3d40c3e00 2023-08-25T09:49:28.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c403e9a0 con 0x7fa3d40c3e00 2023-08-25T09:49:28.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d415c280 con 0x7fa3d40c3e00 2023-08-25T09:49:28.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.241+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d415c560 con 0x7fa3d40c3e00 2023-08-25T09:49:28.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.242+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d4072790 con 0x7fa3d40c3e00 2023-08-25T09:49:28.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.243+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa3c403eb00 con 0x7fa3d40c3e00 2023-08-25T09:49:28.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.244+0000 7fa3caffd640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3b406a080 0x7fa3b406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.244+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa3c40ab190 con 0x7fa3d40c3e00 2023-08-25T09:49:28.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.244+0000 7fa3d1d9b640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3b406a080 0x7fa3b406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.245+0000 7fa3d1d9b640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3b406a080 0x7fa3b406c540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc006fd0 tx=0x7fa3bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:28.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.248+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa3c4076d00 con 0x7fa3d40c3e00 2023-08-25T09:49:28.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.440+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa3d404dbd0 con 0x7fa3d40c3e00 2023-08-25T09:49:28.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.441+0000 7fa3caffd640 1 -- 172.21.15.71:0/290269875 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3c403edb0 con 0x7fa3d40c3e00 2023-08-25T09:49:28.444 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:49:28.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3b406a080 msgr2=0x7fa3b406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3b406a080 0x7fa3b406c540 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc006fd0 tx=0x7fa3bc008040 comp rx=0 tx=0).stop 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 msgr2=0x7fa3d41516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 0x7fa3d41516e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4000f80 tx=0x7fa3c4004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 shutdown_connections 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3d40cec00 0x7fa3d41f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3b406a080 0x7fa3b406c540 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3d4153310 0x7fa3d41f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 --2- 172.21.15.71:0/290269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d40c3e00 0x7fa3d41516e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 >> 172.21.15.71:0/290269875 conn(0x7fa3d4065910 msgr2=0x7fa3d41525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:28.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 shutdown_connections 2023-08-25T09:49:28.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.444+0000 7fa3d8e57640 1 -- 172.21.15.71:0/290269875 wait complete. 2023-08-25T09:49:28.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:49:28.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.682+0000 7f7c92fd2640 1 Processor -- start 2023-08-25T09:49:28.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.682+0000 7f7c92fd2640 1 -- start start 2023-08-25T09:49:28.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.683+0000 7f7c92fd2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.683+0000 7f7c92fd2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c8c0c4fb0 0x7f7c8c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.683+0000 7f7c92fd2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8c157bf0 con 0x7f7c8c1542d0 2023-08-25T09:49:28.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.683+0000 7f7c92fd2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8c0c4b60 con 0x7f7c8c0c4fb0 2023-08-25T09:49:28.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.683+0000 7f7c92fd2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c06a130 con 0x7f7c8c069850 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c8c0c4fb0 0x7f7c8c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c8c0c4fb0 0x7f7c8c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47258/0 (socket says 172.21.15.71:47258) 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 -- 172.21.15.71:0/1497320912 learned_addr learned my addr 172.21.15.71:0/1497320912 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c917cf640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 -- 172.21.15.71:0/1497320912 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 msgr2=0x7f7c8c1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 -- 172.21.15.71:0/1497320912 >> v1:172.21.15.164:6789/0 conn(0x7f7c8c1542d0 legacy=0x7f7c8c0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:28.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c927d1640 1 -- 172.21.15.71:0/1497320912 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c1546b0 con 0x7f7c8c0c4fb0 2023-08-25T09:49:28.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.684+0000 7f7c917cf640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:28.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.685+0000 7f7c927d1640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c0c5410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9377b8b6e45ea98 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:28.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c90fce640 1 -- 172.21.15.71:0/1497320912 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8402ca70 con 0x7f7c8c0c4fb0 2023-08-25T09:49:28.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c90fce640 1 -- 172.21.15.71:0/1497320912 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8402cbd0 con 0x7f7c8c0c4fb0 2023-08-25T09:49:28.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c90fce640 1 -- 172.21.15.71:0/1497320912 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c840343f0 con 0x7f7c8c0c4fb0 2023-08-25T09:49:28.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1497320912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c0c4fb0 msgr2=0x7f7c8c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c0c5410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c8402bed0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1497320912 shutdown_connections 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c0c5410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1497320912 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.686+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1497320912 >> 172.21.15.71:0/1497320912 conn(0x7f7c8c06c280 msgr2=0x7f7c8c06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.687+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1497320912 shutdown_connections 2023-08-25T09:49:28.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.687+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1497320912 wait complete. 2023-08-25T09:49:28.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.688+0000 7f7c92fd2640 1 Processor -- start 2023-08-25T09:49:28.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.688+0000 7f7c92fd2640 1 -- start start 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.689+0000 7f7c92fd2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c0c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.689+0000 7f7c92fd2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.689+0000 7f7c92fd2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 0x7f7c8c156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.689+0000 7f7c92fd2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0c0740 con 0x7f7c8c1542d0 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.689+0000 7f7c92fd2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c14f3a0 con 0x7f7c8c0c4fb0 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.689+0000 7f7c92fd2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c1559f0 con 0x7f7c8c069850 2023-08-25T09:49:28.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 0x7f7c8c156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 0x7f7c8c156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47270/0 (socket says 172.21.15.71:47270) 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 -- 172.21.15.71:0/1208274135 learned_addr learned my addr 172.21.15.71:0/1208274135 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c917cf640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c91fd0640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c0c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 -- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 msgr2=0x7f7c8c0c2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c0c2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 -- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c8c0c4fb0 msgr2=0x7f7c8c156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 -- 172.21.15.71:0/1208274135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c84009b30 con 0x7f7c8c1542d0 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c917cf640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c91fd0640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c0c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.690+0000 7f7c927d1640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 0x7f7c8c156e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.691+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c84009510 con 0x7f7c8c1542d0 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.691+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8403e450 con 0x7f7c8c1542d0 2023-08-25T09:49:28.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.691+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8c1564c0 con 0x7f7c8c1542d0 2023-08-25T09:49:28.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.691+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8403e820 con 0x7f7c8c1542d0 2023-08-25T09:49:28.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.691+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c0c0480 con 0x7f7c8c1542d0 2023-08-25T09:49:28.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.692+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c156650 con 0x7f7c8c1542d0 2023-08-25T09:49:28.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.693+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7c8403e980 con 0x7f7c8c1542d0 2023-08-25T09:49:28.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.694+0000 7f7c827fc640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c7006a080 0x7f7c7006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:28.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.694+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7c840aab70 con 0x7f7c8c1542d0 2023-08-25T09:49:28.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.694+0000 7f7c91fd0640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c7006a080 0x7f7c7006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:28.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.694+0000 7f7c91fd0640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c7006a080 0x7f7c7006c540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800a830 tx=0x7f7c88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:28.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.698+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7c840765a0 con 0x7f7c8c1542d0 2023-08-25T09:49:28.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.893+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7c8c157bf0 con 0x7f7c8c1542d0 2023-08-25T09:49:28.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.894+0000 7f7c827fc640 1 -- 172.21.15.71:0/1208274135 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7c8c157bf0 con 0x7f7c8c1542d0 2023-08-25T09:49:28.897 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:49:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c7006a080 msgr2=0x7f7c7006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c7006a080 0x7f7c7006c540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800a830 tx=0x7f7c88008040 comp rx=0 tx=0).stop 2023-08-25T09:49:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 msgr2=0x7f7c8c156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 0x7f7c8c156e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7c84009e50 tx=0x7f7c84004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 shutdown_connections 2023-08-25T09:49:28.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c8c0c4fb0 0x7f7c8c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c7006a080 0x7f7c7006c540 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c8c1542d0 0x7f7c8c156e90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 --2- 172.21.15.71:0/1208274135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c8c069850 0x7f7c8c0c2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:28.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.896+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 >> 172.21.15.71:0/1208274135 conn(0x7f7c8c06c280 msgr2=0x7f7c8c0af180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:28.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.897+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 shutdown_connections 2023-08-25T09:49:28.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:28.897+0000 7f7c92fd2640 1 -- 172.21.15.71:0/1208274135 wait complete. 2023-08-25T09:49:28.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:49:29.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.128+0000 7fadde2bf640 1 Processor -- start 2023-08-25T09:49:29.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.128+0000 7fadde2bf640 1 -- start start 2023-08-25T09:49:29.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadde2bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fadd80c3660 0x7fadd80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadde2bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 0x7fadd8150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadde2bf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd80c1dc0 con 0x7fadd80c3a40 2023-08-25T09:49:29.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadde2bf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd8157a80 con 0x7fadd80c3660 2023-08-25T09:49:29.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadde2bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8151360 con 0x7fadd81509d0 2023-08-25T09:49:29.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadcf7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fadd80c3a40 0x7fadd806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51212/0 (socket says 172.21.15.71:51212) 2023-08-25T09:49:29.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.130+0000 7fadcf7fe640 1 -- 172.21.15.71:0/868233278 learned_addr learned my addr 172.21.15.71:0/868233278 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:29.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd7fff640 1 --2- 172.21.15.71:0/868233278 >> v2:172.21.15.164:3300/0 conn(0x7fadd80c3660 0x7fadd80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7faddc85c640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 0x7fadd8150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899815320 0 0) 0x7fadd80c1dc0 con 0x7fadd80c3a40 2023-08-25T09:49:29.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4003520 con 0x7fadd80c3a40 2023-08-25T09:49:29.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389128682 0 0) 0x7fadc4003520 con 0x7fadd80c3a40 2023-08-25T09:49:29.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 msgr2=0x7fadd8150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:29.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 0x7fadd8150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 >> v2:172.21.15.164:3300/0 conn(0x7fadd80c3660 msgr2=0x7fadd80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:29.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 --2- 172.21.15.71:0/868233278 >> v2:172.21.15.164:3300/0 conn(0x7fadd80c3660 0x7fadd80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd8156e00 con 0x7fadd80c3a40 2023-08-25T09:49:29.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fadc0001670 con 0x7fadd80c3a40 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 >> v1:172.21.15.71:6789/0 conn(0x7fadd80c3a40 legacy=0x7fadd806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 0x7fadc40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8157a80 con 0x7fadc4003f60 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8151360 con 0x7fadc4004d00 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.131+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc400ba30 con 0x7fadd81509d0 2023-08-25T09:49:29.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadd7fff640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 0x7fadc40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7faddc85c640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadcf7fe640 1 -- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 msgr2=0x7fadc40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd81509d0 0x7fadc40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadcf7fe640 1 -- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc4004d00 msgr2=0x7fadc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.132+0000 7fadcf7fe640 1 -- 172.21.15.71:0/868233278 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4003520 con 0x7fadc4003f60 2023-08-25T09:49:29.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.133+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadc40047c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fadc000c210 tx=0x7fadc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e009421b7f84a4be server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:29.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.133+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc0005ad0 con 0x7fadc4003f60 2023-08-25T09:49:29.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.133+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc0005c30 con 0x7fadc4003f60 2023-08-25T09:49:29.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.133+0000 7fadd77fe640 1 -- 172.21.15.71:0/868233278 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc002caf0 con 0x7fadc4003f60 2023-08-25T09:49:29.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7fadde2bf640 1 -- 172.21.15.71:0/868233278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 msgr2=0x7fadc40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7fadde2bf640 1 --2- 172.21.15.71:0/868233278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadc40047c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fadc000c210 tx=0x7fadc40057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7faddc85c640 1 -- 172.21.15.71:0/868233278 reap_dead start 2023-08-25T09:49:29.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7fadde2bf640 1 -- 172.21.15.71:0/868233278 shutdown_connections 2023-08-25T09:49:29.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7fadde2bf640 1 -- 172.21.15.71:0/868233278 >> 172.21.15.71:0/868233278 conn(0x7fadd80654f0 msgr2=0x7fadd8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:29.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7fadde2bf640 1 -- 172.21.15.71:0/868233278 shutdown_connections 2023-08-25T09:49:29.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.134+0000 7fadde2bf640 1 -- 172.21.15.71:0/868233278 wait complete. 2023-08-25T09:49:29.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.136+0000 7fadde2bf640 1 Processor -- start 2023-08-25T09:49:29.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.136+0000 7fadde2bf640 1 -- start start 2023-08-25T09:49:29.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadde2bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadd81f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadde2bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81ef4a0 0x7fadd8145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadde2bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd806abc0 0x7fadd814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadde2bf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd81ee7b0 con 0x7fadc4003f60 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadde2bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd81ef010 con 0x7fadd81ef4a0 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadde2bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8149ca0 con 0x7fadd806abc0 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadd7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadd81f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadd7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadd81f1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47284/0 (socket says 172.21.15.71:47284) 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.137+0000 7fadd7fff640 1 -- 172.21.15.71:0/3214404827 learned_addr learned my addr 172.21.15.71:0/3214404827 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:29.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81ef4a0 0x7fadd8145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd7fff640 1 -- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd806abc0 msgr2=0x7fadd814b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd7fff640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd806abc0 0x7fadd814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd7fff640 1 -- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81ef4a0 msgr2=0x7fadd8145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd7fff640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81ef4a0 0x7fadd8145eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd7fff640 1 -- 172.21.15.71:0/3214404827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd8154440 con 0x7fadc4003f60 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd7fff640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadd81f1d40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadc80119d0 tx=0x7fadc8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc800fa70 con 0x7fadc4003f60 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc800fbd0 con 0x7fadc4003f60 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8014440 con 0x7fadc4003f60 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd8149f40 con 0x7fadc4003f60 2023-08-25T09:49:29.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.138+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd81ee9c0 con 0x7fadc4003f60 2023-08-25T09:49:29.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.139+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd8069160 con 0x7fadc4003f60 2023-08-25T09:49:29.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.141+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fadc80145a0 con 0x7fadc4003f60 2023-08-25T09:49:29.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.141+0000 7fadd57fa640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fadb806a080 0x7fadb806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.141+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fadc8088a70 con 0x7fadc4003f60 2023-08-25T09:49:29.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.142+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fadb806a080 0x7fadb806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.142+0000 7fadcf7fe640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fadb806a080 0x7fadb806c540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fadc0034000 tx=0x7fadc0007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:29.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.146+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fadc8054420 con 0x7fadc4003f60 2023-08-25T09:49:29.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.372+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadd814bca0 con 0x7fadc4003f60 2023-08-25T09:49:29.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.374+0000 7fadd57fa640 1 -- 172.21.15.71:0/3214404827 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5280 (secure 0 0 0) 0x7fadc8022070 con 0x7fadc4003f60 2023-08-25T09:49:29.377 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:29.378 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4306,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-08-25T09:49:29.378 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 3 2023-08-25T09:49:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.376+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fadb806a080 msgr2=0x7fadb806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.376+0000 7fadde2bf640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fadb806a080 0x7fadb806c540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fadc0034000 tx=0x7fadc0007000 comp rx=0 tx=0).stop 2023-08-25T09:49:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 msgr2=0x7fadd81f1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadd81f1d40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadc80119d0 tx=0x7fadc8011e90 comp rx=0 tx=0).stop 2023-08-25T09:49:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 shutdown_connections 2023-08-25T09:49:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fadd81ef4a0 0x7fadd8145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fadb806a080 0x7fadb806c540 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fadd806abc0 0x7fadd814b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 --2- 172.21.15.71:0/3214404827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fadc4003f60 0x7fadd81f1d40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 >> 172.21.15.71:0/3214404827 conn(0x7fadd80654f0 msgr2=0x7fadd80c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:29.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 shutdown_connections 2023-08-25T09:49:29.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.377+0000 7fadde2bf640 1 -- 172.21.15.71:0/3214404827 wait complete. 2023-08-25T09:49:29.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:29.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.616+0000 7f544b6cc640 1 Processor -- start 2023-08-25T09:49:29.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.616+0000 7f544b6cc640 1 -- start start 2023-08-25T09:49:29.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f544b6cc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5444159fc0 0x7f54440c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f544b6cc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544406a960 con 0x7f544406cc70 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f544b6cc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544406ac20 con 0x7f54441594d0 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f544b6cc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544406af00 con 0x7f5444159fc0 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f5448c67640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f544406cc70 0x7f54441598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51220/0 (socket says 172.21.15.71:51220) 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f5448c67640 1 -- 172.21.15.71:0/3617360863 learned_addr learned my addr 172.21.15.71:0/3617360863 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.618+0000 7f5449c69640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5444159fc0 0x7f54440c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.619+0000 7f5449c69640 1 -- 172.21.15.71:0/3617360863 >> v1:172.21.15.164:6789/0 conn(0x7f54441594d0 legacy=0x7f544414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.619+0000 7f5449c69640 1 -- 172.21.15.71:0/3617360863 >> v1:172.21.15.71:6789/0 conn(0x7f544406cc70 legacy=0x7f54441598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.619+0000 7f5449c69640 1 -- 172.21.15.71:0/3617360863 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440c2e60 con 0x7f5444159fc0 2023-08-25T09:49:29.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.619+0000 7f543bfff640 1 -- 172.21.15.71:0/3617360863 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081582535 0 0) 0x7f544406a960 con 0x7f544406cc70 2023-08-25T09:49:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.620+0000 7f5449c69640 1 --2- 172.21.15.71:0/3617360863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5444159fc0 0x7f54440c4c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f543c009d90 tx=0x7f543c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd4fff18a8bf51a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.620+0000 7f543bfff640 1 -- 172.21.15.71:0/3617360863 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c02cd80 con 0x7f5444159fc0 2023-08-25T09:49:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.620+0000 7f543bfff640 1 -- 172.21.15.71:0/3617360863 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c034440 con 0x7f5444159fc0 2023-08-25T09:49:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.620+0000 7f543bfff640 1 -- 172.21.15.71:0/3617360863 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0347d0 con 0x7f5444159fc0 2023-08-25T09:49:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 -- 172.21.15.71:0/3617360863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5444159fc0 msgr2=0x7f54440c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 --2- 172.21.15.71:0/3617360863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5444159fc0 0x7f54440c4c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f543c009d90 tx=0x7f543c02bfd0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 -- 172.21.15.71:0/3617360863 shutdown_connections 2023-08-25T09:49:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 --2- 172.21.15.71:0/3617360863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5444159fc0 0x7f54440c4c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 -- 172.21.15.71:0/3617360863 >> 172.21.15.71:0/3617360863 conn(0x7f54440654f0 msgr2=0x7f5444065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 -- 172.21.15.71:0/3617360863 shutdown_connections 2023-08-25T09:49:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.621+0000 7f544b6cc640 1 -- 172.21.15.71:0/3617360863 wait complete. 2023-08-25T09:49:29.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.623+0000 7f544b6cc640 1 Processor -- start 2023-08-25T09:49:29.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.623+0000 7f544b6cc640 1 -- start start 2023-08-25T09:49:29.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f544b6cc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f544406cc70 0x7f54441e1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f544b6cc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54441594d0 0x7f54440c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f544b6cc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 0x7f544413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f544b6cc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444156e30 con 0x7f54440c8da0 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f544b6cc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54440c80f0 con 0x7f54441594d0 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f544b6cc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5444158ce0 con 0x7f544406cc70 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 0x7f544413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 0x7f544413cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47306/0 (socket says 172.21.15.71:47306) 2023-08-25T09:49:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 -- 172.21.15.71:0/2373202709 learned_addr learned my addr 172.21.15.71:0/2373202709 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449468640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f544406cc70 0x7f54441e1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 -- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f544406cc70 msgr2=0x7f54441e1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f544406cc70 0x7f54441e1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 -- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54441594d0 msgr2=0x7f54440c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54441594d0 0x7f54440c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449c69640 1 -- 172.21.15.71:0/2373202709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c009a70 con 0x7f54440c8da0 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.624+0000 7f5449468640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f544406cc70 0x7f54441e1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.625+0000 7f5449c69640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 0x7f544413cb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f543c037040 tx=0x7f543c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.625+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c040650 con 0x7f54440c8da0 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.625+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c0407b0 con 0x7f54440c8da0 2023-08-25T09:49:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.625+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c040b60 con 0x7f54440c8da0 2023-08-25T09:49:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.625+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5444158fc0 con 0x7f54440c8da0 2023-08-25T09:49:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.625+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54440c9af0 con 0x7f54440c8da0 2023-08-25T09:49:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.626+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54440c9cb0 con 0x7f54440c8da0 2023-08-25T09:49:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.627+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f543c040cc0 con 0x7f54440c8da0 2023-08-25T09:49:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.628+0000 7f5439ffb640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f542806a080 0x7f542806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:29.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.628+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f543c0ab2d0 con 0x7f54440c8da0 2023-08-25T09:49:29.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.628+0000 7f5449468640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f542806a080 0x7f542806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:29.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.629+0000 7f5449468640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f542806a080 0x7f542806c540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5440000f60 tx=0x7f5440009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:29.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.632+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f543c076dc0 con 0x7f54440c8da0 2023-08-25T09:49:29.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.822+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5444069160 con 0x7f54440c8da0 2023-08-25T09:49:29.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.823+0000 7f5439ffb640 1 -- 172.21.15.71:0/2373202709 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14636 (secure 0 0 0) 0x7f543c07a3e0 con 0x7f54440c8da0 2023-08-25T09:49:29.826 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:29.826 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:28.512304+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:29.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.825+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f542806a080 msgr2=0x7f542806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.825+0000 7f544b6cc640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f542806a080 0x7f542806c540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5440000f60 tx=0x7f5440009040 comp rx=0 tx=0).stop 2023-08-25T09:49:29.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.825+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 msgr2=0x7f544413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:29.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.825+0000 7f544b6cc640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 0x7f544413cb60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f543c037040 tx=0x7f543c004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:29.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 shutdown_connections 2023-08-25T09:49:29.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54441594d0 0x7f54440c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f542806a080 0x7f542806c540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54440c8da0 0x7f544413cb60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 --2- 172.21.15.71:0/2373202709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f544406cc70 0x7f54441e1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:29.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 >> 172.21.15.71:0/2373202709 conn(0x7f54440654f0 msgr2=0x7f544406c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:29.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 shutdown_connections 2023-08-25T09:49:29.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:29.826+0000 7f544b6cc640 1 -- 172.21.15.71:0/2373202709 wait complete. 2023-08-25T09:49:29.849 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-08-25T09:49:29.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:49:30.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.065+0000 7f3799f49640 1 Processor -- start 2023-08-25T09:49:30.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.066+0000 7f3799f49640 1 -- start start 2023-08-25T09:49:30.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f3799f49640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3794071760 0x7f379406a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f3799f49640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37940c0480 con 0x7f37940af290 2023-08-25T09:49:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f3799f49640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3794069850 con 0x7f379406ad90 2023-08-25T09:49:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f3799f49640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379406b1a0 con 0x7f3794071760 2023-08-25T09:49:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f3798f47640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f37940af290 0x7f379406b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46968/0 (socket says 172.21.15.71:46968) 2023-08-25T09:49:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f3798f47640 1 -- 172.21.15.71:0/1488541567 learned_addr learned my addr 172.21.15.71:0/1488541567 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715922145 0 0) 0x7f37940c0480 con 0x7f37940af290 2023-08-25T09:49:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.068+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3784003520 con 0x7f37940af290 2023-08-25T09:49:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f3793fff640 1 --2- 172.21.15.71:0/1488541567 >> v2:172.21.15.164:3300/0 conn(0x7f3794071760 0x7f379406a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397598110 0 0) 0x7f3784003520 con 0x7f37940af290 2023-08-25T09:49:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 >> v2:172.21.15.164:3300/0 conn(0x7f3794071760 msgr2=0x7f379406a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 --2- 172.21.15.71:0/1488541567 >> v2:172.21.15.164:3300/0 conn(0x7f3794071760 0x7f379406a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 >> v1:172.21.15.164:6789/0 conn(0x7f379406ad90 legacy=0x7f379415c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794069ad0 con 0x7f37940af290 2023-08-25T09:49:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f3788001670 con 0x7f37940af290 2023-08-25T09:49:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 >> v1:172.21.15.71:6789/0 conn(0x7f37940af290 legacy=0x7f379406b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37840039e0 0x7f3784003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840047c0 0x7f3784004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37840051f0 0x7f378400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379406b1a0 con 0x7f37840039e0 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378400c150 con 0x7f37840051f0 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378400c310 con 0x7f37840047c0 2023-08-25T09:49:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.069+0000 7f3798f47640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37840039e0 0x7f3784003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3798f47640 1 -- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840047c0 msgr2=0x7f3784004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3799748640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37840051f0 0x7f378400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3793fff640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840047c0 0x7f3784004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3798f47640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840047c0 0x7f3784004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3798f47640 1 -- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37840051f0 msgr2=0x7f378400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3798f47640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37840051f0 0x7f378400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3798f47640 1 -- 172.21.15.71:0/1488541567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784003520 con 0x7f37840039e0 2023-08-25T09:49:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3799748640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37840051f0 0x7f378400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.070+0000 7f3799748640 1 -- 172.21.15.71:0/1488541567 reap_dead start 2023-08-25T09:49:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.071+0000 7f3798f47640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37840039e0 0x7f3784003e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f378800c190 tx=0x7f37840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e002e4d13bf0cf54 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.071+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3788005ad0 con 0x7f37840039e0 2023-08-25T09:49:30.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.071+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3788005c30 con 0x7f37840039e0 2023-08-25T09:49:30.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.071+0000 7f37937fe640 1 -- 172.21.15.71:0/1488541567 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378802caf0 con 0x7f37840039e0 2023-08-25T09:49:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.071+0000 7f3799f49640 1 -- 172.21.15.71:0/1488541567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37840039e0 msgr2=0x7f3784003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.071+0000 7f3799f49640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37840039e0 0x7f3784003e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f378800c190 tx=0x7f37840057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.072+0000 7f3799f49640 1 -- 172.21.15.71:0/1488541567 shutdown_connections 2023-08-25T09:49:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.072+0000 7f3799f49640 1 --2- 172.21.15.71:0/1488541567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37840039e0 0x7f3784003e00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.072+0000 7f3799f49640 1 -- 172.21.15.71:0/1488541567 >> 172.21.15.71:0/1488541567 conn(0x7f379406c280 msgr2=0x7f379406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.072+0000 7f3799f49640 1 -- 172.21.15.71:0/1488541567 shutdown_connections 2023-08-25T09:49:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.072+0000 7f3799f49640 1 -- 172.21.15.71:0/1488541567 wait complete. 2023-08-25T09:49:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.073+0000 7f3799f49640 1 Processor -- start 2023-08-25T09:49:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.074+0000 7f3799f49640 1 -- start start 2023-08-25T09:49:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799f49640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840039e0 0x7f3794151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799f49640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 0x7f379415b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799f49640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37941f19b0 0x7f37941f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799f49640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940c3740 con 0x7f37940c3b60 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799f49640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940c1520 con 0x7f37941f19b0 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799f49640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37940c1780 con 0x7f37840039e0 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 0x7f379415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 0x7f379415b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54502/0 (socket says 172.21.15.71:54502) 2023-08-25T09:49:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 -- 172.21.15.71:0/3817541777 learned_addr learned my addr 172.21.15.71:0/3817541777 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3799748640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37941f19b0 0x7f37941f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 -- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840039e0 msgr2=0x7f3794151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840039e0 0x7f3794151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 -- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37941f19b0 msgr2=0x7f37941f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37941f19b0 0x7f37941f1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.075+0000 7f3793fff640 1 -- 172.21.15.71:0/3817541777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37940c27e0 con 0x7f37940c3b60 2023-08-25T09:49:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.076+0000 7f3793fff640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 0x7f379415b550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f378000e540 tx=0x7f378000ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.076+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780013610 con 0x7f37940c3b60 2023-08-25T09:49:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.076+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780013770 con 0x7f37940c3b60 2023-08-25T09:49:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.076+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780013ae0 con 0x7f37940c3b60 2023-08-25T09:49:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.076+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37940c2ac0 con 0x7f37940c3b60 2023-08-25T09:49:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.076+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37940c3080 con 0x7f37940c3b60 2023-08-25T09:49:30.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.077+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379404e420 con 0x7f37940c3b60 2023-08-25T09:49:30.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.078+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3780013c40 con 0x7f37940c3b60 2023-08-25T09:49:30.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.079+0000 7f37917fa640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f377806a050 0x7f377806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.079+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3780087e10 con 0x7f37940c3b60 2023-08-25T09:49:30.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.079+0000 7f3798f47640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f377806a050 0x7f377806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.080+0000 7f3798f47640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f377806a050 0x7f377806c510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3788034000 tx=0x7f3788039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.083+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3780053850 con 0x7f37940c3b60 2023-08-25T09:49:30.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.269+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37940bfb20 con 0x7f37940c3b60 2023-08-25T09:49:30.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.270+0000 7f37917fa640 1 -- 172.21.15.71:0/3817541777 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3780056e70 con 0x7f37940c3b60 2023-08-25T09:49:30.273 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:49:30.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f377806a050 msgr2=0x7f377806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f377806a050 0x7f377806c510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3788034000 tx=0x7f3788039040 comp rx=0 tx=0).stop 2023-08-25T09:49:30.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 msgr2=0x7f379415b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 0x7f379415b550 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f378000e540 tx=0x7f378000ea00 comp rx=0 tx=0).stop 2023-08-25T09:49:30.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 shutdown_connections 2023-08-25T09:49:30.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37941f19b0 0x7f37941f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f377806a050 0x7f377806c510 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f37940c3b60 0x7f379415b550 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 --2- 172.21.15.71:0/3817541777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37840039e0 0x7f3794151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 >> 172.21.15.71:0/3817541777 conn(0x7f379406c280 msgr2=0x7f379406c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:30.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 shutdown_connections 2023-08-25T09:49:30.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.273+0000 7f3799f49640 1 -- 172.21.15.71:0/3817541777 wait complete. 2023-08-25T09:49:30.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:49:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.508+0000 7f4e6db42640 1 Processor -- start 2023-08-25T09:49:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.508+0000 7f4e6db42640 1 -- start start 2023-08-25T09:49:30.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e6db42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 0x7f4e680cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e6db42640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e680c3e00 0x7f4e680c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e6db42640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e680c47b0 con 0x7f4e68153310 2023-08-25T09:49:30.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e6db42640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e681590c0 con 0x7f4e680c3e00 2023-08-25T09:49:30.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e6db42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e68159380 con 0x7f4e680cec00 2023-08-25T09:49:30.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e677fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4e68153310 0x7f4e6806a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46982/0 (socket says 172.21.15.71:46982) 2023-08-25T09:49:30.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.510+0000 7f4e677fe640 1 -- 172.21.15.71:0/693698763 learned_addr learned my addr 172.21.15.71:0/693698763 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:30.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e5ffff640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 0x7f4e680cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e67fff640 1 --2- 172.21.15.71:0/693698763 >> v2:172.21.15.164:3300/0 conn(0x7f4e680c3e00 0x7f4e680c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957087804 0 0) 0x7f4e680c47b0 con 0x7f4e68153310 2023-08-25T09:49:30.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e54003520 con 0x7f4e68153310 2023-08-25T09:49:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2620861922 0 0) 0x7f4e54003520 con 0x7f4e68153310 2023-08-25T09:49:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 msgr2=0x7f4e680cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 0x7f4e680cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 >> v2:172.21.15.164:3300/0 conn(0x7f4e680c3e00 msgr2=0x7f4e680c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 --2- 172.21.15.71:0/693698763 >> v2:172.21.15.164:3300/0 conn(0x7f4e680c3e00 0x7f4e680c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68159660 con 0x7f4e68153310 2023-08-25T09:49:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4e58001670 con 0x7f4e68153310 2023-08-25T09:49:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 >> v1:172.21.15.71:6789/0 conn(0x7f4e68153310 legacy=0x7f4e6806a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e540039e0 0x7f4e54003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 0x7f4e540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e54004d00 0x7f4e54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e681590c0 con 0x7f4e540039e0 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e68159380 con 0x7f4e54004d00 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.511+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5400ba00 con 0x7f4e680cec00 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e677fe640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e540039e0 0x7f4e54003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e5ffff640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 0x7f4e540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e677fe640 1 -- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 msgr2=0x7f4e540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e677fe640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e680cec00 0x7f4e540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e677fe640 1 -- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e54004d00 msgr2=0x7f4e54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e677fe640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e54004d00 0x7f4e54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.512+0000 7f4e677fe640 1 -- 172.21.15.71:0/693698763 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e54003520 con 0x7f4e540039e0 2023-08-25T09:49:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e677fe640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e540039e0 0x7f4e54003e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800c240 tx=0x7f4e540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d645a056913008c8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5802cb00 con 0x7f4e540039e0 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5802cc60 con 0x7f4e540039e0 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e66ffd640 1 -- 172.21.15.71:0/693698763 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e58005af0 con 0x7f4e540039e0 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e6db42640 1 -- 172.21.15.71:0/693698763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e540039e0 msgr2=0x7f4e54003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e6db42640 1 --2- 172.21.15.71:0/693698763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e540039e0 0x7f4e54003e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4e5800c240 tx=0x7f4e540057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.514+0000 7f4e67fff640 1 -- 172.21.15.71:0/693698763 reap_dead start 2023-08-25T09:49:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.515+0000 7f4e6db42640 1 -- 172.21.15.71:0/693698763 shutdown_connections 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.515+0000 7f4e6db42640 1 -- 172.21.15.71:0/693698763 >> 172.21.15.71:0/693698763 conn(0x7f4e68065910 msgr2=0x7f4e68065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.515+0000 7f4e6db42640 1 -- 172.21.15.71:0/693698763 shutdown_connections 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.515+0000 7f4e6db42640 1 -- 172.21.15.71:0/693698763 wait complete. 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.516+0000 7f4e6db42640 1 Processor -- start 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.517+0000 7f4e6db42640 1 -- start start 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e6db42640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e540039e0 0x7f4e681f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e6db42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 0x7f4e68153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e6db42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e681481b0 0x7f4e68148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e6db42640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e681558f0 con 0x7f4e680cde50 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e6db42640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e681f2740 con 0x7f4e540039e0 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e6db42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e68157860 con 0x7f4e681481b0 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 0x7f4e68153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 0x7f4e68153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54512/0 (socket says 172.21.15.71:54512) 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 -- 172.21.15.71:0/1923958357 learned_addr learned my addr 172.21.15.71:0/1923958357 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 -- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e681481b0 msgr2=0x7f4e68148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e67fff640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e681481b0 0x7f4e68148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e681481b0 0x7f4e68148630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e677fe640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e540039e0 0x7f4e681f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 -- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e540039e0 msgr2=0x7f4e681f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e540039e0 0x7f4e681f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.518+0000 7f4e5ffff640 1 -- 172.21.15.71:0/1923958357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68157e70 con 0x7f4e680cde50 2023-08-25T09:49:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.519+0000 7f4e5ffff640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 0x7f4e68153720 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e50011540 tx=0x7f4e50011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.519+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50015930 con 0x7f4e680cde50 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.519+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e50015a90 con 0x7f4e680cde50 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.519+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e50015e60 con 0x7f4e680cde50 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.519+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e68158150 con 0x7f4e680cde50 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.519+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e68158710 con 0x7f4e680cde50 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.520+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e681588d0 con 0x7f4e680cde50 2023-08-25T09:49:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.521+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4e5000f460 con 0x7f4e680cde50 2023-08-25T09:49:30.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.522+0000 7f4e64ff9640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e4806a080 0x7f4e4806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.522+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4e50088b30 con 0x7f4e680cde50 2023-08-25T09:49:30.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.522+0000 7f4e677fe640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e4806a080 0x7f4e4806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.523+0000 7f4e677fe640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e4806a080 0x7f4e4806c540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e58034000 tx=0x7f4e58007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.527+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4e50057ed0 con 0x7f4e680cde50 2023-08-25T09:49:30.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.718+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4e68069160 con 0x7f4e680cde50 2023-08-25T09:49:30.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.719+0000 7f4e64ff9640 1 -- 172.21.15.71:0/1923958357 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e500220a0 con 0x7f4e680cde50 2023-08-25T09:49:30.722 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:49:30.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e4806a080 msgr2=0x7f4e4806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e4806a080 0x7f4e4806c540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e58034000 tx=0x7f4e58007000 comp rx=0 tx=0).stop 2023-08-25T09:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 msgr2=0x7f4e68153720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 0x7f4e68153720 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e50011540 tx=0x7f4e50011a00 comp rx=0 tx=0).stop 2023-08-25T09:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 shutdown_connections 2023-08-25T09:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e4806a080 0x7f4e4806c540 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e681481b0 0x7f4e68148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e680cde50 0x7f4e68153720 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 --2- 172.21.15.71:0/1923958357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e540039e0 0x7f4e681f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 >> 172.21.15.71:0/1923958357 conn(0x7f4e68065910 msgr2=0x7f4e68150ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 shutdown_connections 2023-08-25T09:49:30.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.722+0000 7f4e6db42640 1 -- 172.21.15.71:0/1923958357 wait complete. 2023-08-25T09:49:30.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-08-25T09:49:30.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.955+0000 7f3be85b5640 1 Processor -- start 2023-08-25T09:49:30.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.955+0000 7f3be85b5640 1 -- start start 2023-08-25T09:49:30.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.957+0000 7f3be85b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be00c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.957+0000 7f3be85b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3be01592d0 0x7f3be0159730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.957+0000 7f3be85b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be0158e40 con 0x7f3be00c7740 2023-08-25T09:49:30.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.957+0000 7f3be85b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be006bc80 con 0x7f3be01592d0 2023-08-25T09:49:30.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.957+0000 7f3be85b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be01534a0 con 0x7f3be01530c0 2023-08-25T09:49:30.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be00c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be00c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:39794/0 (socket says 172.21.15.71:39794) 2023-08-25T09:49:30.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 -- 172.21.15.71:0/4013003591 learned_addr learned my addr 172.21.15.71:0/4013003591 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:30.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6b52640 1 --2- 172.21.15.71:0/4013003591 >> v2:172.21.15.164:3300/0 conn(0x7f3be01592d0 0x7f3be0159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 -- 172.21.15.71:0/4013003591 >> v2:172.21.15.164:3300/0 conn(0x7f3be01592d0 msgr2=0x7f3be0159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:30.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 --2- 172.21.15.71:0/4013003591 >> v2:172.21.15.164:3300/0 conn(0x7f3be01592d0 0x7f3be0159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 -- 172.21.15.71:0/4013003591 >> v1:172.21.15.164:6789/0 conn(0x7f3be00c7740 legacy=0x7f3be006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.958+0000 7f3be6351640 1 -- 172.21.15.71:0/4013003591 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be00c7380 con 0x7f3be01530c0 2023-08-25T09:49:30.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.960+0000 7f3be6351640 1 --2- 172.21.15.71:0/4013003591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be00c6e40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009870 tx=0x7f3bdc02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f16d24fe237e7533 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.960+0000 7f3be534f640 1 -- 172.21.15.71:0/4013003591 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc02f840 con 0x7f3be01530c0 2023-08-25T09:49:30.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.960+0000 7f3be534f640 1 -- 172.21.15.71:0/4013003591 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc02f9a0 con 0x7f3be01530c0 2023-08-25T09:49:30.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.960+0000 7f3be534f640 1 -- 172.21.15.71:0/4013003591 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc02fd30 con 0x7f3be01530c0 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 -- 172.21.15.71:0/4013003591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 msgr2=0x7f3be00c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 --2- 172.21.15.71:0/4013003591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be00c6e40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009870 tx=0x7f3bdc02ba20 comp rx=0 tx=0).stop 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 -- 172.21.15.71:0/4013003591 shutdown_connections 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 --2- 172.21.15.71:0/4013003591 >> v2:172.21.15.164:3300/0 conn(0x7f3be01592d0 0x7f3be0159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 --2- 172.21.15.71:0/4013003591 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be00c6e40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 -- 172.21.15.71:0/4013003591 >> 172.21.15.71:0/4013003591 conn(0x7f3be00654f0 msgr2=0x7f3be0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 -- 172.21.15.71:0/4013003591 shutdown_connections 2023-08-25T09:49:30.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.961+0000 7f3be85b5640 1 -- 172.21.15.71:0/4013003591 wait complete. 2023-08-25T09:49:30.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.963+0000 7f3be85b5640 1 Processor -- start 2023-08-25T09:49:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.963+0000 7f3be85b5640 1 -- start start 2023-08-25T09:49:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.964+0000 7f3be85b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be00c7740 0x7f3be0155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.964+0000 7f3be85b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be01eeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.964+0000 7f3be85b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 0x7f3be01eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.964+0000 7f3be85b5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be01547a0 con 0x7f3be01592d0 2023-08-25T09:49:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.964+0000 7f3be85b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0152400 con 0x7f3be00c7740 2023-08-25T09:49:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.964+0000 7f3be85b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be014f570 con 0x7f3be01530c0 2023-08-25T09:49:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 0x7f3be01eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 0x7f3be01eefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54516/0 (socket says 172.21.15.71:54516) 2023-08-25T09:49:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 -- 172.21.15.71:0/553396216 learned_addr learned my addr 172.21.15.71:0/553396216 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6351640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be00c7740 0x7f3be0155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be5b50640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be01eeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 -- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 msgr2=0x7f3be01eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be01eeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 -- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be00c7740 msgr2=0x7f3be0155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:30.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be00c7740 0x7f3be0155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:30.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.965+0000 7f3be6b52640 1 -- 172.21.15.71:0/553396216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc0094d0 con 0x7f3be01592d0 2023-08-25T09:49:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.966+0000 7f3be6b52640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 0x7f3be01eefe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800efc0 tx=0x7f3bd800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.966+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8014480 con 0x7f3be01592d0 2023-08-25T09:49:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.966+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd800f040 con 0x7f3be01592d0 2023-08-25T09:49:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.966+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8014820 con 0x7f3be01592d0 2023-08-25T09:49:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.966+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be0152140 con 0x7f3be01592d0 2023-08-25T09:49:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.966+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be0154200 con 0x7f3be01592d0 2023-08-25T09:49:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.967+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be01ee780 con 0x7f3be01592d0 2023-08-25T09:49:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.968+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3bd8014980 con 0x7f3be01592d0 2023-08-25T09:49:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.969+0000 7f3bd6ffd640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bc406a050 0x7f3bc406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.969+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3bd8088e10 con 0x7f3be01592d0 2023-08-25T09:49:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.969+0000 7f3be6351640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bc406a050 0x7f3bc406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.970+0000 7f3be6351640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bc406a050 0x7f3bc406c510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:30.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:30.973+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3bd80548e0 con 0x7f3be01592d0 2023-08-25T09:49:31.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.164+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3be0069160 con 0x7f3be01592d0 2023-08-25T09:49:31.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.165+0000 7f3bd6ffd640 1 -- 172.21.15.71:0/553396216 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v21) v1 ==== 84+0+126 (secure 0 0 0) 0x7f3bd8018090 con 0x7f3be01592d0 2023-08-25T09:49:31.168 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:31.168 INFO:teuthology.orchestra.run.smithi071.stdout:[ 2023-08-25T09:49:31.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-08-25T09:49:31.168 INFO:teuthology.orchestra.run.smithi071.stdout: "poolnum": 1, 2023-08-25T09:49:31.168 INFO:teuthology.orchestra.run.smithi071.stdout: "poolname": ".mgr" 2023-08-25T09:49:31.169 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:49:31.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2023-08-25T09:49:31.169 INFO:teuthology.orchestra.run.smithi071.stdout: "poolnum": 2, 2023-08-25T09:49:31.169 INFO:teuthology.orchestra.run.smithi071.stdout: "poolname": "rbd" 2023-08-25T09:49:31.169 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-08-25T09:49:31.169 INFO:teuthology.orchestra.run.smithi071.stdout:] 2023-08-25T09:49:31.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bc406a050 msgr2=0x7f3bc406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bc406a050 0x7f3bc406c510 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc036000 tx=0x7f3bdc008030 comp rx=0 tx=0).stop 2023-08-25T09:49:31.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 msgr2=0x7f3be01eefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 0x7f3be01eefe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800efc0 tx=0x7f3bd800c490 comp rx=0 tx=0).stop 2023-08-25T09:49:31.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 shutdown_connections 2023-08-25T09:49:31.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bc406a050 0x7f3bc406c510 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be01592d0 0x7f3be01eefe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3be01530c0 0x7f3be01eeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 --2- 172.21.15.71:0/553396216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3be00c7740 0x7f3be0155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.167+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 >> 172.21.15.71:0/553396216 conn(0x7f3be00654f0 msgr2=0x7f3be00c7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:31.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.168+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 shutdown_connections 2023-08-25T09:49:31.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.168+0000 7f3be85b5640 1 -- 172.21.15.71:0/553396216 wait complete. 2023-08-25T09:49:31.189 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-08-25T09:49:31.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-08-25T09:49:31.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.401+0000 7f3d59a9f640 1 Processor -- start 2023-08-25T09:49:31.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.402+0000 7f3d59a9f640 1 -- start start 2023-08-25T09:49:31.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d59a9f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3d540cb570 0x7f3d540ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d59a9f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d540b1af0 con 0x7f3d540cb190 2023-08-25T09:49:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d59a9f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d540b0fb0 con 0x7f3d5415cd00 2023-08-25T09:49:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d59a9f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d540c98f0 con 0x7f3d540cb570 2023-08-25T09:49:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d58a9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3d540cb190 0x7f3d540c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46990/0 (socket says 172.21.15.71:46990) 2023-08-25T09:49:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d58a9d640 1 -- 172.21.15.71:0/695130487 learned_addr learned my addr 172.21.15.71:0/695130487 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:31.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.403+0000 7f3d53fff640 1 --2- 172.21.15.71:0/695130487 >> v2:172.21.15.71:3300/0 conn(0x7f3d540cb570 0x7f3d540ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:31.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.404+0000 7f3d537fe640 1 -- 172.21.15.71:0/695130487 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136366128 0 0) 0x7f3d540b1af0 con 0x7f3d540cb190 2023-08-25T09:49:31.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.404+0000 7f3d537fe640 1 -- 172.21.15.71:0/695130487 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c003520 con 0x7f3d540cb190 2023-08-25T09:49:31.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.404+0000 7f3d53fff640 1 -- 172.21.15.71:0/695130487 >> v1:172.21.15.164:6789/0 conn(0x7f3d5415cd00 legacy=0x7f3d54158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.404+0000 7f3d53fff640 1 -- 172.21.15.71:0/695130487 >> v1:172.21.15.71:6789/0 conn(0x7f3d540cb190 legacy=0x7f3d540c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.404+0000 7f3d53fff640 1 -- 172.21.15.71:0/695130487 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5415d130 con 0x7f3d540cb570 2023-08-25T09:49:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.405+0000 7f3d53fff640 1 --2- 172.21.15.71:0/695130487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb570 0x7f3d540ca910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886362fc4863e849 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d537fe640 1 -- 172.21.15.71:0/695130487 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4802f810 con 0x7f3d540cb570 2023-08-25T09:49:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d537fe640 1 -- 172.21.15.71:0/695130487 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4802f970 con 0x7f3d540cb570 2023-08-25T09:49:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d537fe640 1 -- 172.21.15.71:0/695130487 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4802fd00 con 0x7f3d540cb570 2023-08-25T09:49:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 -- 172.21.15.71:0/695130487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb570 msgr2=0x7f3d540ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/695130487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb570 0x7f3d540ca910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802b9f0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 -- 172.21.15.71:0/695130487 shutdown_connections 2023-08-25T09:49:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/695130487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb570 0x7f3d540ca910 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 -- 172.21.15.71:0/695130487 >> 172.21.15.71:0/695130487 conn(0x7f3d5406ae20 msgr2=0x7f3d5406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 -- 172.21.15.71:0/695130487 shutdown_connections 2023-08-25T09:49:31.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.406+0000 7f3d59a9f640 1 -- 172.21.15.71:0/695130487 wait complete. 2023-08-25T09:49:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.408+0000 7f3d59a9f640 1 Processor -- start 2023-08-25T09:49:31.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.408+0000 7f3d59a9f640 1 -- start start 2023-08-25T09:49:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.409+0000 7f3d59a9f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 0x7f3d540c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.409+0000 7f3d59a9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d540cb570 0x7f3d540c2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.409+0000 7f3d59a9f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5415cd00 0x7f3d54155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:31.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.409+0000 7f3d59a9f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540ca3b0 con 0x7f3d540cb190 2023-08-25T09:49:31.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.409+0000 7f3d59a9f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d54154af0 con 0x7f3d5415cd00 2023-08-25T09:49:31.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.409+0000 7f3d59a9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540cde70 con 0x7f3d540cb570 2023-08-25T09:49:31.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 0x7f3d540c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:31.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 0x7f3d540c1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54536/0 (socket says 172.21.15.71:54536) 2023-08-25T09:49:31.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 -- 172.21.15.71:0/2332198841 learned_addr learned my addr 172.21.15.71:0/2332198841 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 -- 172.21.15.71:0/2332198841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d540cb570 msgr2=0x7f3d540c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d540cb570 0x7f3d540c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 -- 172.21.15.71:0/2332198841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5415cd00 msgr2=0x7f3d54155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5415cd00 0x7f3d54155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:31.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 -- 172.21.15.71:0/2332198841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48009400 con 0x7f3d540cb190 2023-08-25T09:49:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d58a9d640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 0x7f3d540c1de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400cbc0 tx=0x7f3d4400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4400ad40 con 0x7f3d540cb190 2023-08-25T09:49:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d44012400 con 0x7f3d540cb190 2023-08-25T09:49:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.410+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4400aea0 con 0x7f3d540cb190 2023-08-25T09:49:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.411+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d540cecd0 con 0x7f3d540cb190 2023-08-25T09:49:31.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.411+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d540ce3a0 con 0x7f3d540cb190 2023-08-25T09:49:31.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.412+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5404e420 con 0x7f3d540cb190 2023-08-25T09:49:31.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.413+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3d44007070 con 0x7f3d540cb190 2023-08-25T09:49:31.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.414+0000 7f3d517fa640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d4006a080 0x7f3d4006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:31.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.414+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3d440882b0 con 0x7f3d540cb190 2023-08-25T09:49:31.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.414+0000 7f3d53fff640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d4006a080 0x7f3d4006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:31.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.415+0000 7f3d53fff640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d4006a080 0x7f3d4006c540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d48036000 tx=0x7f3d48039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:31.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.418+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3d44053c50 con 0x7f3d540cb190 2023-08-25T09:49:31.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:31.628+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3d540c9df0 con 0x7f3d540cb190 2023-08-25T09:49:32.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.562+0000 7f3d517fa640 1 -- 172.21.15.71:0/2332198841 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v22) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3d44057270 con 0x7f3d540cb190 2023-08-25T09:49:32.566 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'cephfs_metadata' created 2023-08-25T09:49:32.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.569+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d4006a080 msgr2=0x7f3d4006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.569+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d4006a080 0x7f3d4006c540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d48036000 tx=0x7f3d48039040 comp rx=0 tx=0).stop 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.569+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 msgr2=0x7f3d540c1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.569+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 0x7f3d540c1de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400cbc0 tx=0x7f3d4400a430 comp rx=0 tx=0).stop 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 shutdown_connections 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5415cd00 0x7f3d54155040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d4006a080 0x7f3d4006c540 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d540cb570 0x7f3d540c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 --2- 172.21.15.71:0/2332198841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d540cb190 0x7f3d540c1de0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 >> 172.21.15.71:0/2332198841 conn(0x7f3d5406ae20 msgr2=0x7f3d54072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 shutdown_connections 2023-08-25T09:49:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.570+0000 7f3d59a9f640 1 -- 172.21.15.71:0/2332198841 wait complete. 2023-08-25T09:49:32.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-08-25T09:49:32.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.808+0000 7f489061f640 1 Processor -- start 2023-08-25T09:49:32.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.808+0000 7f489061f640 1 -- start start 2023-08-25T09:49:32.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.809+0000 7f489061f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f48880c3a40 0x7f48880c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f489061f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48880c1dc0 con 0x7f48881509d0 2023-08-25T09:49:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f489061f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4888156e00 con 0x7f48880c3660 2023-08-25T09:49:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f489061f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4888157a80 con 0x7f48880c3a40 2023-08-25T09:49:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f488dbba640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f48880c3a40 0x7f48880c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f488dbba640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f48880c3a40 0x7f48880c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54542/0 (socket says 172.21.15.71:54542) 2023-08-25T09:49:32.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f488dbba640 1 -- 172.21.15.71:0/3337662734 learned_addr learned my addr 172.21.15.71:0/3337662734 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:32.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f488dbba640 1 -- 172.21.15.71:0/3337662734 >> v1:172.21.15.164:6789/0 conn(0x7f48880c3660 legacy=0x7f488806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:32.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f488dbba640 1 -- 172.21.15.71:0/3337662734 >> v1:172.21.15.71:6789/0 conn(0x7f48881509d0 legacy=0x7f4888150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:32.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.810+0000 7f488dbba640 1 -- 172.21.15.71:0/3337662734 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48881570e0 con 0x7f48880c3a40 2023-08-25T09:49:32.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.811+0000 7f488dbba640 1 --2- 172.21.15.71:0/3337662734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3a40 0x7f48880c1710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4878009870 tx=0x7f487802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb599919a141aef2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f488d3b9640 1 -- 172.21.15.71:0/3337662734 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487802f850 con 0x7f48880c3a40 2023-08-25T09:49:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f488d3b9640 1 -- 172.21.15.71:0/3337662734 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487802f9b0 con 0x7f48880c3a40 2023-08-25T09:49:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f488d3b9640 1 -- 172.21.15.71:0/3337662734 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487802fd40 con 0x7f48880c3a40 2023-08-25T09:49:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 -- 172.21.15.71:0/3337662734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3a40 msgr2=0x7f48880c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 --2- 172.21.15.71:0/3337662734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3a40 0x7f48880c1710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4878009870 tx=0x7f487802ba30 comp rx=0 tx=0).stop 2023-08-25T09:49:32.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 -- 172.21.15.71:0/3337662734 shutdown_connections 2023-08-25T09:49:32.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 --2- 172.21.15.71:0/3337662734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3a40 0x7f48880c1710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 -- 172.21.15.71:0/3337662734 >> 172.21.15.71:0/3337662734 conn(0x7f48880654f0 msgr2=0x7f4888065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:32.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 -- 172.21.15.71:0/3337662734 shutdown_connections 2023-08-25T09:49:32.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.812+0000 7f489061f640 1 -- 172.21.15.71:0/3337662734 wait complete. 2023-08-25T09:49:32.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.814+0000 7f489061f640 1 Processor -- start 2023-08-25T09:49:32.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.814+0000 7f489061f640 1 -- start start 2023-08-25T09:49:32.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f489061f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 0x7f48881ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:32.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f489061f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48880c3a40 0x7f4888142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:32.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f489061f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48881509d0 0x7f48881efd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:32.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f489061f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888147b80 con 0x7f48880c3660 2023-08-25T09:49:32.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f489061f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48881f0a10 con 0x7f48881509d0 2023-08-25T09:49:32.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f489061f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4888155660 con 0x7f48880c3a40 2023-08-25T09:49:32.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f488e3bb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 0x7f48881ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:32.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f488e3bb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 0x7f48881ea7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54548/0 (socket says 172.21.15.71:54548) 2023-08-25T09:49:32.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.815+0000 7f488e3bb640 1 -- 172.21.15.71:0/361519531 learned_addr learned my addr 172.21.15.71:0/361519531 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:32.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f488e3bb640 1 -- 172.21.15.71:0/361519531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48880c3a40 msgr2=0x7f4888142f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:32.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f488e3bb640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48880c3a40 0x7f4888142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f488e3bb640 1 -- 172.21.15.71:0/361519531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48881509d0 msgr2=0x7f48881efd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:32.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f488e3bb640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48881509d0 0x7f48881efd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:32.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f488e3bb640 1 -- 172.21.15.71:0/361519531 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48780094d0 con 0x7f48880c3660 2023-08-25T09:49:32.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f488e3bb640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 0x7f48881ea7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f488400b3f0 tx=0x7f488400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:32.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884011830 con 0x7f48880c3660 2023-08-25T09:49:32.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4884011990 con 0x7f48880c3660 2023-08-25T09:49:32.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4888155940 con 0x7f48880c3660 2023-08-25T09:49:32.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.816+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884011d40 con 0x7f48880c3660 2023-08-25T09:49:32.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.817+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48881eace0 con 0x7f48880c3660 2023-08-25T09:49:32.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.818+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4888155e60 con 0x7f48880c3660 2023-08-25T09:49:32.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.819+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f488401a400 con 0x7f48880c3660 2023-08-25T09:49:32.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.820+0000 7f487effd640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f486c06a050 0x7f486c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:32.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.820+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f48840881e0 con 0x7f48880c3660 2023-08-25T09:49:32.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.820+0000 7f488dbba640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f486c06a050 0x7f486c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:32.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.821+0000 7f488dbba640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f486c06a050 0x7f486c06c510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4878036000 tx=0x7f4878039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:32.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:32.825+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4888155e60 con 0x7f48880c3660 2023-08-25T09:49:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.016+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f4888155e60 con 0x7f48880c3660 2023-08-25T09:49:33.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.581+0000 7f487effd640 1 -- 172.21.15.71:0/361519531 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v23) v1 ==== 171+0+0 (secure 0 0 0) 0x7f4888155e60 con 0x7f48880c3660 2023-08-25T09:49:33.585 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'cephfs_data' created 2023-08-25T09:49:33.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.602+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f486c06a050 msgr2=0x7f486c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:33.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.602+0000 7f489061f640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f486c06a050 0x7f486c06c510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4878036000 tx=0x7f4878039040 comp rx=0 tx=0).stop 2023-08-25T09:49:33.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.602+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 msgr2=0x7f48881ea7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:33.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.602+0000 7f489061f640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 0x7f48881ea7a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f488400b3f0 tx=0x7f488400b8b0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 shutdown_connections 2023-08-25T09:49:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48881509d0 0x7f48881efd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f486c06a050 0x7f486c06c510 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48880c3a40 0x7f4888142f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 --2- 172.21.15.71:0/361519531 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48880c3660 0x7f48881ea7a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 >> 172.21.15.71:0/361519531 conn(0x7f48880654f0 msgr2=0x7f48880c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:33.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 shutdown_connections 2023-08-25T09:49:33.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.603+0000 7f489061f640 1 -- 172.21.15.71:0/361519531 wait complete. 2023-08-25T09:49:33.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-08-25T09:49:33.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.819+0000 7f54e59f3640 1 Processor -- start 2023-08-25T09:49:33.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.820+0000 7f54e59f3640 1 -- start start 2023-08-25T09:49:33.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54e59f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f54e0150ce0 0x7f54e00c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54e59f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f54e00c58c0 0x7f54e00c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54e59f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e00c76b0 0x7f54e00c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54e59f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e00c2ba0 con 0x7f54e0150ce0 2023-08-25T09:49:33.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54e59f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e00c1770 con 0x7f54e00c58c0 2023-08-25T09:49:33.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54e59f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e00c1f30 con 0x7f54e00c76b0 2023-08-25T09:49:33.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54deffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f54e0150ce0 0x7f54e00c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:33.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54deffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f54e0150ce0 0x7f54e00c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54562/0 (socket says 172.21.15.71:54562) 2023-08-25T09:49:33.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.821+0000 7f54deffd640 1 -- 172.21.15.71:0/1069072321 learned_addr learned my addr 172.21.15.71:0/1069072321 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:33.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54d7fff640 1 --2- 172.21.15.71:0/1069072321 >> v2:172.21.15.164:3300/0 conn(0x7f54e00c58c0 0x7f54e00c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:33.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54deffd640 1 -- 172.21.15.71:0/1069072321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e00c76b0 msgr2=0x7f54e00c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:33.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54deffd640 1 --2- 172.21.15.71:0/1069072321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e00c76b0 0x7f54e00c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54deffd640 1 -- 172.21.15.71:0/1069072321 >> v2:172.21.15.164:3300/0 conn(0x7f54e00c58c0 msgr2=0x7f54e00c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54deffd640 1 --2- 172.21.15.71:0/1069072321 >> v2:172.21.15.164:3300/0 conn(0x7f54e00c58c0 0x7f54e00c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54deffd640 1 -- 172.21.15.71:0/1069072321 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e00c6400 con 0x7f54e0150ce0 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.822+0000 7f54d7fff640 1 --2- 172.21.15.71:0/1069072321 >> v2:172.21.15.164:3300/0 conn(0x7f54e00c58c0 0x7f54e00c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.823+0000 7f54deffd640 1 --2- 172.21.15.71:0/1069072321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e0150ce0 0x7f54e00c5380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f54e00741e0 tx=0x7f54d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f4ae72f6010b95 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.823+0000 7f54de7fc640 1 -- 172.21.15.71:0/1069072321 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0004230 con 0x7f54e0150ce0 2023-08-25T09:49:33.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54de7fc640 1 -- 172.21.15.71:0/1069072321 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0004390 con 0x7f54e0150ce0 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54de7fc640 1 -- 172.21.15.71:0/1069072321 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0030840 con 0x7f54e0150ce0 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 -- 172.21.15.71:0/1069072321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e0150ce0 msgr2=0x7f54e00c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 --2- 172.21.15.71:0/1069072321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e0150ce0 0x7f54e00c5380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f54e00741e0 tx=0x7f54d002b980 comp rx=0 tx=0).stop 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 -- 172.21.15.71:0/1069072321 shutdown_connections 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 --2- 172.21.15.71:0/1069072321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e00c76b0 0x7f54e00c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 --2- 172.21.15.71:0/1069072321 >> v2:172.21.15.164:3300/0 conn(0x7f54e00c58c0 0x7f54e00c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 --2- 172.21.15.71:0/1069072321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e0150ce0 0x7f54e00c5380 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 -- 172.21.15.71:0/1069072321 >> 172.21.15.71:0/1069072321 conn(0x7f54e006bff0 msgr2=0x7f54e006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:33.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 -- 172.21.15.71:0/1069072321 shutdown_connections 2023-08-25T09:49:33.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.824+0000 7f54e59f3640 1 -- 172.21.15.71:0/1069072321 wait complete. 2023-08-25T09:49:33.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.826+0000 7f54e59f3640 1 Processor -- start 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.826+0000 7f54e59f3640 1 -- start start 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54e59f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54e00c58c0 0x7f54e006f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54e59f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 0x7f54e00c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54e59f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e0150ce0 0x7f54e00c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54e59f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e01e31b0 con 0x7f54e00c76b0 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54e59f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e006e8a0 con 0x7f54e00c58c0 2023-08-25T09:49:33.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54e59f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e01e1920 con 0x7f54e0150ce0 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 0x7f54e00c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 0x7f54e00c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54576/0 (socket says 172.21.15.71:54576) 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.827+0000 7f54d7fff640 1 -- 172.21.15.71:0/1628971770 learned_addr learned my addr 172.21.15.71:0/1628971770 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54deffd640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54e00c58c0 0x7f54e006f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54d7fff640 1 -- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e0150ce0 msgr2=0x7f54e00c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54d7fff640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e0150ce0 0x7f54e00c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54d7fff640 1 -- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54e00c58c0 msgr2=0x7f54e006f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54d7fff640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54e00c58c0 0x7f54e006f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54d7fff640 1 -- 172.21.15.71:0/1628971770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0009460 con 0x7f54e00c76b0 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54deffd640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54e00c58c0 0x7f54e006f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54d7fff640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 0x7f54e00c91e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f54c800c7f0 tx=0x7f54c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c800cea0 con 0x7f54e00c76b0 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8004230 con 0x7f54e00c76b0 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54e59f3640 1 -- 172.21.15.71:0/1628971770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e006e570 con 0x7f54e00c76b0 2023-08-25T09:49:33.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.828+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8007830 con 0x7f54e00c76b0 2023-08-25T09:49:33.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.829+0000 7f54e59f3640 1 -- 172.21.15.71:0/1628971770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e01e1bc0 con 0x7f54e00c76b0 2023-08-25T09:49:33.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.830+0000 7f54e59f3640 1 -- 172.21.15.71:0/1628971770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e006ea10 con 0x7f54e00c76b0 2023-08-25T09:49:33.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.831+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f54c8007990 con 0x7f54e00c76b0 2023-08-25T09:49:33.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.832+0000 7f54dcff9640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f54c006a0a0 0x7f54c006c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:33.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.832+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f54c80884a0 con 0x7f54e00c76b0 2023-08-25T09:49:33.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.832+0000 7f54deffd640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f54c006a0a0 0x7f54c006c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:33.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.833+0000 7f54deffd640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f54c006a0a0 0x7f54c006c560 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54d0035000 tx=0x7f54d0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:33.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:33.837+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f54c8053c60 con 0x7f54e00c76b0 2023-08-25T09:49:34.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.060+0000 7f54e59f3640 1 -- 172.21.15.71:0/1628971770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f54e0069160 con 0x7f54e00c76b0 2023-08-25T09:49:34.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.612+0000 7f54dcff9640 1 -- 172.21.15.71:0/1628971770 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f54c8057280 con 0x7f54e00c76b0 2023-08-25T09:49:34.616 INFO:teuthology.orchestra.run.smithi071.stderr:new fs with metadata pool 3 and data pool 4 2023-08-25T09:49:34.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.624+0000 7f54d5ffb640 1 -- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f54c006a0a0 msgr2=0x7f54c006c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:34.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.624+0000 7f54d5ffb640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f54c006a0a0 0x7f54c006c560 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54d0035000 tx=0x7f54d0008030 comp rx=0 tx=0).stop 2023-08-25T09:49:34.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.624+0000 7f54d5ffb640 1 -- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 msgr2=0x7f54e00c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:34.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.624+0000 7f54d5ffb640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 0x7f54e00c91e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f54c800c7f0 tx=0x7f54c800ccb0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 -- 172.21.15.71:0/1628971770 shutdown_connections 2023-08-25T09:49:34.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f54c006a0a0 0x7f54c006c560 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54e0150ce0 0x7f54e00c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54e00c76b0 0x7f54e00c91e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 --2- 172.21.15.71:0/1628971770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54e00c58c0 0x7f54e006f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 -- 172.21.15.71:0/1628971770 >> 172.21.15.71:0/1628971770 conn(0x7f54e006bff0 msgr2=0x7f54e00c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:34.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 -- 172.21.15.71:0/1628971770 shutdown_connections 2023-08-25T09:49:34.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.625+0000 7f54d5ffb640 1 -- 172.21.15.71:0/1628971770 wait complete. 2023-08-25T09:49:34.648 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-08-25T09:49:34.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-08-25T09:49:34.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.837+0000 7f6cb18e0640 1 Processor -- start 2023-08-25T09:49:34.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.837+0000 7f6cb18e0640 1 -- start start 2023-08-25T09:49:34.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb18e0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cac0c4070 0x7f6cac0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb18e0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6cac071ba0 0x7f6cac0adda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb18e0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cac0bf480 con 0x7f6cac15a020 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb18e0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cac0bf740 con 0x7f6cac071ba0 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb18e0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cac0c0200 con 0x7f6cac0c4070 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cabfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6cac071ba0 0x7f6cac0adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cabfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6cac071ba0 0x7f6cac0adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54582/0 (socket says 172.21.15.71:54582) 2023-08-25T09:49:34.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cabfff640 1 -- 172.21.15.71:0/3237335289 learned_addr learned my addr 172.21.15.71:0/3237335289 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:34.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb10df640 1 --1- 172.21.15.71:0/3237335289 >> v1:172.21.15.71:6789/0 conn(0x7f6cac15a020 0x7f6cac15a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47010/0 (socket says 172.21.15.71:47010) 2023-08-25T09:49:34.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.839+0000 7f6cb08de640 1 --2- 172.21.15.71:0/3237335289 >> v2:172.21.15.164:3300/0 conn(0x7f6cac0c4070 0x7f6cac0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:34.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.840+0000 7f6cabfff640 1 -- 172.21.15.71:0/3237335289 >> v2:172.21.15.164:3300/0 conn(0x7f6cac0c4070 msgr2=0x7f6cac0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:34.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.840+0000 7f6cabfff640 1 --2- 172.21.15.71:0/3237335289 >> v2:172.21.15.164:3300/0 conn(0x7f6cac0c4070 0x7f6cac0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.840+0000 7f6cabfff640 1 -- 172.21.15.71:0/3237335289 >> v1:172.21.15.71:6789/0 conn(0x7f6cac15a020 legacy=0x7f6cac15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:34.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.840+0000 7f6cabfff640 1 -- 172.21.15.71:0/3237335289 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac0bfc40 con 0x7f6cac071ba0 2023-08-25T09:49:34.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.841+0000 7f6cabfff640 1 --2- 172.21.15.71:0/3237335289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac071ba0 0x7f6cac0adda0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6cac06cf90 tx=0x7f6c9802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd73ae4575a55dd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:34.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.841+0000 7f6cab7fe640 1 -- 172.21.15.71:0/3237335289 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9802f850 con 0x7f6cac071ba0 2023-08-25T09:49:34.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.841+0000 7f6cab7fe640 1 -- 172.21.15.71:0/3237335289 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9802f9b0 con 0x7f6cac071ba0 2023-08-25T09:49:34.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.841+0000 7f6cab7fe640 1 -- 172.21.15.71:0/3237335289 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9802fd40 con 0x7f6cac071ba0 2023-08-25T09:49:34.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3237335289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac071ba0 msgr2=0x7f6cac0adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:34.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3237335289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac071ba0 0x7f6cac0adda0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6cac06cf90 tx=0x7f6c9802ba30 comp rx=0 tx=0).stop 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3237335289 shutdown_connections 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3237335289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac071ba0 0x7f6cac0adda0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3237335289 >> v2:172.21.15.164:3300/0 conn(0x7f6cac0c4070 0x7f6cac0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3237335289 >> 172.21.15.71:0/3237335289 conn(0x7f6cac0ca0d0 msgr2=0x7f6cac0ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3237335289 shutdown_connections 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.842+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3237335289 wait complete. 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.843+0000 7f6cb18e0640 1 Processor -- start 2023-08-25T09:49:34.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.844+0000 7f6cb18e0640 1 -- start start 2023-08-25T09:49:34.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb18e0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cac071ba0 0x7f6cac069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:34.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb18e0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 0x7f6cac1edd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:34.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb18e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cac15a020 0x7f6cac1ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb18e0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac1f1290 con 0x7f6cac0c4070 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb18e0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac1f1e10 con 0x7f6cac071ba0 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb18e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac151eb0 con 0x7f6cac15a020 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 0x7f6cac1edd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 0x7f6cac1edd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54594/0 (socket says 172.21.15.71:54594) 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 -- 172.21.15.71:0/3075029049 learned_addr learned my addr 172.21.15.71:0/3075029049 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:34.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb08de640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cac071ba0 0x7f6cac069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 -- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cac15a020 msgr2=0x7f6cac1ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cac15a020 0x7f6cac1ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 -- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cac071ba0 msgr2=0x7f6cac069ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cac071ba0 0x7f6cac069ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cabfff640 1 -- 172.21.15.71:0/3075029049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980094d0 con 0x7f6cac0c4070 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.845+0000 7f6cb08de640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cac071ba0 0x7f6cac069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.846+0000 7f6cabfff640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 0x7f6cac1edd20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6c98000c00 tx=0x7f6c98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:34.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.846+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98037d10 con 0x7f6cac0c4070 2023-08-25T09:49:34.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.846+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98037e90 con 0x7f6cac0c4070 2023-08-25T09:49:34.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.846+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac1f14e0 con 0x7f6cac0c4070 2023-08-25T09:49:34.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.846+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c980426c0 con 0x7f6cac0c4070 2023-08-25T09:49:34.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.846+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac1f1aa0 con 0x7f6cac0c4070 2023-08-25T09:49:34.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.847+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac1f1c30 con 0x7f6cac0c4070 2023-08-25T09:49:34.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.848+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6c9802d3e0 con 0x7f6cac0c4070 2023-08-25T09:49:34.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.849+0000 7f6ca97fa640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c9006a050 0x7f6c9006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:34.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.849+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f6c9803c070 con 0x7f6cac0c4070 2023-08-25T09:49:34.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.849+0000 7f6cb08de640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c9006a050 0x7f6c9006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:34.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.850+0000 7f6cb08de640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c9006a050 0x7f6c9006c510 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0006fd0 tx=0x7f6ca0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:34.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:34.854+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6c98076510 con 0x7f6cac0c4070 2023-08-25T09:49:35.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.051+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f6cac061460 con 0x7f6cac0c4070 2023-08-25T09:49:35.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.585+0000 7f6ca97fa640 1 -- 172.21.15.71:0/3075029049 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v25) v1 ==== 157+0+0 (secure 0 0 0) 0x7f6c98047070 con 0x7f6cac0c4070 2023-08-25T09:49:35.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.588+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c9006a050 msgr2=0x7f6c9006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.588+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c9006a050 0x7f6c9006c510 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0006fd0 tx=0x7f6ca0008040 comp rx=0 tx=0).stop 2023-08-25T09:49:35.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.588+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 msgr2=0x7f6cac1edd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.588+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 0x7f6cac1edd20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6c98000c00 tx=0x7f6c98004730 comp rx=0 tx=0).stop 2023-08-25T09:49:35.592 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 shutdown_connections 2023-08-25T09:49:35.592 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c9006a050 0x7f6c9006c510 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cac15a020 0x7f6cac1ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cac0c4070 0x7f6cac1edd20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 --2- 172.21.15.71:0/3075029049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cac071ba0 0x7f6cac069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 >> 172.21.15.71:0/3075029049 conn(0x7f6cac0ca0d0 msgr2=0x7f6cac15a450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:35.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 shutdown_connections 2023-08-25T09:49:35.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.589+0000 7f6cb18e0640 1 -- 172.21.15.71:0/3075029049 wait complete. 2023-08-25T09:49:35.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-08-25T09:49:35.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.832+0000 7fefa8b0c640 1 Processor -- start 2023-08-25T09:49:35.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.832+0000 7fefa8b0c640 1 -- start start 2023-08-25T09:49:35.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.834+0000 7fefa8b0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 0x7fefa40c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.834+0000 7fefa8b0c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa40c71d0 con 0x7fefa4156fb0 2023-08-25T09:49:35.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.834+0000 7fefa8b0c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa40c76f0 con 0x7fefa40c7af0 2023-08-25T09:49:35.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.834+0000 7fefa8b0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa406a800 con 0x7fefa4150ca0 2023-08-25T09:49:35.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.834+0000 7fefa2d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fefa4156fb0 0x7fefa4157420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47026/0 (socket says 172.21.15.71:47026) 2023-08-25T09:49:35.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.834+0000 7fefa2d9d640 1 -- 172.21.15.71:0/2431176797 learned_addr learned my addr 172.21.15.71:0/2431176797 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:35.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa259c640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 0x7fefa40c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:35.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343686012 0 0) 0x7fef9c001110 con 0x7fefa4156fb0 2023-08-25T09:49:35.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef90003520 con 0x7fefa4156fb0 2023-08-25T09:49:35.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3889461756 0 0) 0x7fef90003520 con 0x7fefa4156fb0 2023-08-25T09:49:35.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 msgr2=0x7fefa40c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:35.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 0x7fefa40c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 >> v1:172.21.15.164:6789/0 conn(0x7fefa40c7af0 legacy=0x7fefa40c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa406d8e0 con 0x7fefa4156fb0 2023-08-25T09:49:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fef9c001800 con 0x7fefa4156fb0 2023-08-25T09:49:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 >> v1:172.21.15.71:6789/0 conn(0x7fefa4156fb0 legacy=0x7fefa4157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 0x7fef900039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fef900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef90004d00 0x7fef90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa406a800 con 0x7fef90003f20 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9000baf0 con 0x7fef90004d00 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9000bcb0 con 0x7fefa4150ca0 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.835+0000 7fefa1d9b640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fef900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa259c640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 0x7fef900039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa2d9d640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef90004d00 0x7fef90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:35.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa1d9b640 1 -- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 msgr2=0x7fef900039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:35.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa1d9b640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa4150ca0 0x7fef900039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa1d9b640 1 -- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef90004d00 msgr2=0x7fef90005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:35.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa1d9b640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef90004d00 0x7fef90005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa1d9b640 1 -- 172.21.15.71:0/2431176797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90003520 con 0x7fef90003f20 2023-08-25T09:49:35.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.836+0000 7fefa2d9d640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef90004d00 0x7fef90005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:35.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.837+0000 7fefa1d9b640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fef900047c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef8c001750 tx=0x7fef900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e97af7ad6a7085d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.837+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0052d0 con 0x7fef90003f20 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.837+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c02e050 con 0x7fef90003f20 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.837+0000 7fefa159a640 1 -- 172.21.15.71:0/2431176797 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c003830 con 0x7fef90003f20 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa8b0c640 1 -- 172.21.15.71:0/2431176797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 msgr2=0x7fef900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/2431176797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fef900047c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef8c001750 tx=0x7fef900057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa2d9d640 1 -- 172.21.15.71:0/2431176797 reap_dead start 2023-08-25T09:49:35.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa8b0c640 1 -- 172.21.15.71:0/2431176797 shutdown_connections 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa8b0c640 1 -- 172.21.15.71:0/2431176797 >> 172.21.15.71:0/2431176797 conn(0x7fefa4065910 msgr2=0x7fefa406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa8b0c640 1 -- 172.21.15.71:0/2431176797 shutdown_connections 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.838+0000 7fefa8b0c640 1 -- 172.21.15.71:0/2431176797 wait complete. 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.839+0000 7fefa8b0c640 1 Processor -- start 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.840+0000 7fefa8b0c640 1 -- start start 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.840+0000 7fefa8b0c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fefa414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.840+0000 7fefa8b0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa40bce30 0x7fefa40bda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.840+0000 7fefa8b0c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefa40bc4a0 0x7fefa40bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa8b0c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa41ee5d0 con 0x7fef90003f20 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa8b0c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa40bc9e0 con 0x7fefa40bc4a0 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa8b0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa41ee890 con 0x7fefa40bce30 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fefa414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fefa414d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54618/0 (socket says 172.21.15.71:54618) 2023-08-25T09:49:35.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 -- 172.21.15.71:0/3647748686 learned_addr learned my addr 172.21.15.71:0/3647748686 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:35.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 -- 172.21.15.71:0/3647748686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa40bce30 msgr2=0x7fefa40bda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa40bce30 0x7fefa40bda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 -- 172.21.15.71:0/3647748686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefa40bc4a0 msgr2=0x7fefa40bdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:35.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefa40bc4a0 0x7fefa40bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:35.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 -- 172.21.15.71:0/3647748686 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa41eeb70 con 0x7fef90003f20 2023-08-25T09:49:35.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.841+0000 7fefa259c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fefa414d1e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef94007f30 tx=0x7fef94011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:35.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.842+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400e030 con 0x7fef90003f20 2023-08-25T09:49:35.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.842+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef94016470 con 0x7fef90003f20 2023-08-25T09:49:35.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.842+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94016820 con 0x7fef90003f20 2023-08-25T09:49:35.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.842+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa41eee50 con 0x7fef90003f20 2023-08-25T09:49:35.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.842+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa414f800 con 0x7fef90003f20 2023-08-25T09:49:35.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.843+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa406cf30 con 0x7fef90003f20 2023-08-25T09:49:35.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.844+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fef94016980 con 0x7fef90003f20 2023-08-25T09:49:35.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.845+0000 7fef9affd640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fef8406a050 0x7fef8406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:35.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.845+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fef9400f070 con 0x7fef90003f20 2023-08-25T09:49:35.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.845+0000 7fefa1d9b640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fef8406a050 0x7fef8406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:35.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.846+0000 7fefa1d9b640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fef8406a050 0x7fef8406c510 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef8c033000 tx=0x7fef8c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:35.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:35.850+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fef94054c60 con 0x7fef90003f20 2023-08-25T09:49:36.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:36.038+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fefa4069160 con 0x7fef90003f20 2023-08-25T09:49:37.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.638+0000 7fef9affd640 1 -- 172.21.15.71:0/3647748686 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v27) v1 ==== 228+0+0 (secure 0 0 0) 0x7fef94058280 con 0x7fef90003f20 2023-08-25T09:49:37.641 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'cephfs_data_ec' created 2023-08-25T09:49:37.647 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.644+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fef8406a050 msgr2=0x7fef8406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:37.647 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.644+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fef8406a050 0x7fef8406c510 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef8c033000 tx=0x7fef8c02f000 comp rx=0 tx=0).stop 2023-08-25T09:49:37.647 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.644+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 msgr2=0x7fefa414d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:37.647 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.644+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fefa414d1e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef94007f30 tx=0x7fef94011660 comp rx=0 tx=0).stop 2023-08-25T09:49:37.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.649+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 shutdown_connections 2023-08-25T09:49:37.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.649+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefa40bc4a0 0x7fefa40bdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.649+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fef8406a050 0x7fef8406c510 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.649+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefa40bce30 0x7fefa40bda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.649+0000 7fefa8b0c640 1 --2- 172.21.15.71:0/3647748686 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef90003f20 0x7fefa414d1e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.649+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 >> 172.21.15.71:0/3647748686 conn(0x7fefa4065910 msgr2=0x7fefa4149120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:37.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.653+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 shutdown_connections 2023-08-25T09:49:37.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.653+0000 7fefa8b0c640 1 -- 172.21.15.71:0/3647748686 wait complete. 2023-08-25T09:49:37.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-08-25T09:49:37.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.873+0000 7fd28e391640 1 Processor -- start 2023-08-25T09:49:37.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.873+0000 7fd28e391640 1 -- start start 2023-08-25T09:49:37.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28e391640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd2880af420 0x7fd288069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:37.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28e391640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd288160ce0 0x7fd288161110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:37.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28e391640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2880c0690 con 0x7fd28806b350 2023-08-25T09:49:37.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28e391640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd288161650 con 0x7fd288160ce0 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28e391640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2880bf830 con 0x7fd2880af420 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28db90640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd288160ce0 0x7fd288161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28db90640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd288160ce0 0x7fd288161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54622/0 (socket says 172.21.15.71:54622) 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.875+0000 7fd28db90640 1 -- 172.21.15.71:0/3258936665 learned_addr learned my addr 172.21.15.71:0/3258936665 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.876+0000 7fd28d38f640 1 --2- 172.21.15.71:0/3258936665 >> v2:172.21.15.164:3300/0 conn(0x7fd2880af420 0x7fd288069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.876+0000 7fd28db90640 1 -- 172.21.15.71:0/3258936665 >> v2:172.21.15.164:3300/0 conn(0x7fd2880af420 msgr2=0x7fd288069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:37.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.876+0000 7fd28db90640 1 --2- 172.21.15.71:0/3258936665 >> v2:172.21.15.164:3300/0 conn(0x7fd2880af420 0x7fd288069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.876+0000 7fd28db90640 1 -- 172.21.15.71:0/3258936665 >> v1:172.21.15.164:6789/0 conn(0x7fd28806b350 legacy=0x7fd2880c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:37.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.876+0000 7fd28db90640 1 -- 172.21.15.71:0/3258936665 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880bfb10 con 0x7fd288160ce0 2023-08-25T09:49:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.877+0000 7fd28db90640 1 --2- 172.21.15.71:0/3258936665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd288161110 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd2880c7b80 tx=0x7fd28002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b73091f5cebddc35 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.878+0000 7fd27ffff640 1 -- 172.21.15.71:0/3258936665 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28002cb20 con 0x7fd288160ce0 2023-08-25T09:49:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.878+0000 7fd27ffff640 1 -- 172.21.15.71:0/3258936665 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28002cc80 con 0x7fd288160ce0 2023-08-25T09:49:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.878+0000 7fd27ffff640 1 -- 172.21.15.71:0/3258936665 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2800343f0 con 0x7fd288160ce0 2023-08-25T09:49:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.878+0000 7fd28e391640 1 -- 172.21.15.71:0/3258936665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 msgr2=0x7fd288161110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:37.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.878+0000 7fd28e391640 1 --2- 172.21.15.71:0/3258936665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd288161110 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd2880c7b80 tx=0x7fd28002bf80 comp rx=0 tx=0).stop 2023-08-25T09:49:37.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.879+0000 7fd28e391640 1 -- 172.21.15.71:0/3258936665 shutdown_connections 2023-08-25T09:49:37.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.879+0000 7fd28e391640 1 --2- 172.21.15.71:0/3258936665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd288161110 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.879+0000 7fd28e391640 1 --2- 172.21.15.71:0/3258936665 >> v2:172.21.15.164:3300/0 conn(0x7fd2880af420 0x7fd288069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.879+0000 7fd28e391640 1 -- 172.21.15.71:0/3258936665 >> 172.21.15.71:0/3258936665 conn(0x7fd28806c280 msgr2=0x7fd28806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:37.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.879+0000 7fd28e391640 1 -- 172.21.15.71:0/3258936665 shutdown_connections 2023-08-25T09:49:37.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.879+0000 7fd28e391640 1 -- 172.21.15.71:0/3258936665 wait complete. 2023-08-25T09:49:37.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.880+0000 7fd28e391640 1 Processor -- start 2023-08-25T09:49:37.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.881+0000 7fd28e391640 1 -- start start 2023-08-25T09:49:37.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.881+0000 7fd28e391640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd28806b350 0x7fd2880ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:37.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28e391640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2880af420 0x7fd2880ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:37.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28e391640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd2880c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:37.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28e391640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28815dbb0 con 0x7fd288160ce0 2023-08-25T09:49:37.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28e391640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd288156600 con 0x7fd2880af420 2023-08-25T09:49:37.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28e391640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880c8240 con 0x7fd28806b350 2023-08-25T09:49:37.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd2880c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:37.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd2880c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54630/0 (socket says 172.21.15.71:54630) 2023-08-25T09:49:37.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 -- 172.21.15.71:0/1558406318 learned_addr learned my addr 172.21.15.71:0/1558406318 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28d38f640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd28806b350 0x7fd2880ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 -- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd28806b350 msgr2=0x7fd2880ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd28806b350 0x7fd2880ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28cb8e640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2880af420 0x7fd2880ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 -- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2880af420 msgr2=0x7fd2880ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2880af420 0x7fd2880ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:37.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.882+0000 7fd28db90640 1 -- 172.21.15.71:0/1558406318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd280009b30 con 0x7fd288160ce0 2023-08-25T09:49:37.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.883+0000 7fd28db90640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd2880c3be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd280009f40 tx=0x7fd280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:37.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.883+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280007e70 con 0x7fd288160ce0 2023-08-25T09:49:37.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.883+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd280009510 con 0x7fd288160ce0 2023-08-25T09:49:37.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.883+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28003e690 con 0x7fd288160ce0 2023-08-25T09:49:37.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.883+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2881f3cf0 con 0x7fd288160ce0 2023-08-25T09:49:37.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.883+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2880c27a0 con 0x7fd288160ce0 2023-08-25T09:49:37.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.884+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28804e420 con 0x7fd288160ce0 2023-08-25T09:49:37.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.885+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd28003e7f0 con 0x7fd288160ce0 2023-08-25T09:49:37.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.886+0000 7fd27dffb640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd26c06a080 0x7fd26c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:37.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.886+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fd2800ab810 con 0x7fd288160ce0 2023-08-25T09:49:37.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.887+0000 7fd28d38f640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd26c06a080 0x7fd26c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:37.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.887+0000 7fd28d38f640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd26c06a080 0x7fd26c06c540 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd284006fd0 tx=0x7fd284008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:37.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:37.891+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd280076c20 con 0x7fd288160ce0 2023-08-25T09:49:38.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.078+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fd28806a0e0 con 0x7fd288160ce0 2023-08-25T09:49:38.646 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.643+0000 7fd27dffb640 1 -- 172.21.15.71:0/1558406318 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v28) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd28007a240 con 0x7fd288160ce0 2023-08-25T09:49:38.647 INFO:teuthology.orchestra.run.smithi071.stderr:set pool 5 allow_ec_overwrites to true 2023-08-25T09:49:38.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.658+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd26c06a080 msgr2=0x7fd26c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:38.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.659+0000 7fd28e391640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd26c06a080 0x7fd26c06c540 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd284006fd0 tx=0x7fd284008040 comp rx=0 tx=0).stop 2023-08-25T09:49:38.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.660+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 msgr2=0x7fd2880c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:38.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.660+0000 7fd28e391640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd2880c3be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd280009f40 tx=0x7fd280004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:38.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.661+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 shutdown_connections 2023-08-25T09:49:38.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.661+0000 7fd28e391640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2880af420 0x7fd2880ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.661+0000 7fd28e391640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd26c06a080 0x7fd26c06c540 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.661+0000 7fd28e391640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd288160ce0 0x7fd2880c3be0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.661+0000 7fd28e391640 1 --2- 172.21.15.71:0/1558406318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd28806b350 0x7fd2880ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.661+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 >> 172.21.15.71:0/1558406318 conn(0x7fd28806c280 msgr2=0x7fd2880c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:38.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.662+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 shutdown_connections 2023-08-25T09:49:38.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.662+0000 7fd28e391640 1 -- 172.21.15.71:0/1558406318 wait complete. 2023-08-25T09:49:38.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-08-25T09:49:38.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.884+0000 7fe7ece46640 1 Processor -- start 2023-08-25T09:49:38.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.885+0000 7fe7ece46640 1 -- start start 2023-08-25T09:49:38.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.886+0000 7fe7ece46640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe7e80c3a40 0x7fe7e80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:38.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.886+0000 7fe7ece46640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe7e81509d0 0x7fe7e8150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:38.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.886+0000 7fe7ece46640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80c1dc0 con 0x7fe7e80c3660 2023-08-25T09:49:38.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.886+0000 7fe7ece46640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e8157a80 con 0x7fe7e81509d0 2023-08-25T09:49:38.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.886+0000 7fe7ece46640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e8151320 con 0x7fe7e80c3a40 2023-08-25T09:49:38.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.886+0000 7fe7e6d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe7e81509d0 0x7fe7e8150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:38.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7e6d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe7e81509d0 0x7fe7e8150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54642/0 (socket says 172.21.15.71:54642) 2023-08-25T09:49:38.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/673398476 learned_addr learned my addr 172.21.15.71:0/673398476 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:38.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7dffff640 1 --2- 172.21.15.71:0/673398476 >> v2:172.21.15.164:3300/0 conn(0x7fe7e80c3a40 0x7fe7e80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:38.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/673398476 >> v2:172.21.15.164:3300/0 conn(0x7fe7e80c3a40 msgr2=0x7fe7e80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:38.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7e6d9d640 1 --2- 172.21.15.71:0/673398476 >> v2:172.21.15.164:3300/0 conn(0x7fe7e80c3a40 0x7fe7e80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/673398476 >> v1:172.21.15.71:6789/0 conn(0x7fe7e80c3660 legacy=0x7fe7e806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:38.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.887+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/673398476 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e8156e00 con 0x7fe7e81509d0 2023-08-25T09:49:38.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.888+0000 7fe7e6d9d640 1 --2- 172.21.15.71:0/673398476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8150de0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8067520 tx=0x7fe7e002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10b1b42bc2b9b1a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:38.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.888+0000 7fe7e5d9b640 1 -- 172.21.15.71:0/673398476 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e002cbb0 con 0x7fe7e81509d0 2023-08-25T09:49:38.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.888+0000 7fe7e5d9b640 1 -- 172.21.15.71:0/673398476 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e002cd10 con 0x7fe7e81509d0 2023-08-25T09:49:38.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.888+0000 7fe7e5d9b640 1 -- 172.21.15.71:0/673398476 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0034610 con 0x7fe7e81509d0 2023-08-25T09:49:38.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 -- 172.21.15.71:0/673398476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 msgr2=0x7fe7e8150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:38.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 --2- 172.21.15.71:0/673398476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8150de0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8067520 tx=0x7fe7e002c010 comp rx=0 tx=0).stop 2023-08-25T09:49:38.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 -- 172.21.15.71:0/673398476 shutdown_connections 2023-08-25T09:49:38.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 --2- 172.21.15.71:0/673398476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8150de0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 --2- 172.21.15.71:0/673398476 >> v2:172.21.15.164:3300/0 conn(0x7fe7e80c3a40 0x7fe7e80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 -- 172.21.15.71:0/673398476 >> 172.21.15.71:0/673398476 conn(0x7fe7e80654f0 msgr2=0x7fe7e8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:38.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 -- 172.21.15.71:0/673398476 shutdown_connections 2023-08-25T09:49:38.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.889+0000 7fe7ece46640 1 -- 172.21.15.71:0/673398476 wait complete. 2023-08-25T09:49:38.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.891+0000 7fe7ece46640 1 Processor -- start 2023-08-25T09:49:38.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.891+0000 7fe7ece46640 1 -- start start 2023-08-25T09:49:38.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7ece46640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7e80c3660 0x7fe7e81ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:38.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7ece46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7e80c3a40 0x7fe7e80c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7ece46640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7ece46640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81eecd0 con 0x7fe7e81509d0 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7ece46640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8152c90 con 0x7fe7e80c3a40 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7ece46640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e815b020 con 0x7fe7e80c3660 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7e6d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7e6d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54644/0 (socket says 172.21.15.71:54644) 2023-08-25T09:49:38.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.892+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/1439996159 learned_addr learned my addr 172.21.15.71:0/1439996159 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7e80c3660 msgr2=0x7fe7e81ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e659c640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7e80c3660 0x7fe7e81ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7dffff640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7e80c3a40 0x7fe7e80c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e6d9d640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7e80c3660 0x7fe7e81ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7e80c3a40 msgr2=0x7fe7e80c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e6d9d640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7e80c3a40 0x7fe7e80c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e6d9d640 1 -- 172.21.15.71:0/1439996159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0009b30 con 0x7fe7e81509d0 2023-08-25T09:49:38.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e659c640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7e80c3660 0x7fe7e81ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:38.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7e6d9d640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8153120 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0036040 tx=0x7fe7e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:38.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0009510 con 0x7fe7e81509d0 2023-08-25T09:49:38.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e003e450 con 0x7fe7e81509d0 2023-08-25T09:49:38.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e815aa60 con 0x7fe7e81509d0 2023-08-25T09:49:38.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.893+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e003e800 con 0x7fe7e81509d0 2023-08-25T09:49:38.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.894+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e8152640 con 0x7fe7e81509d0 2023-08-25T09:49:38.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.895+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e80c29c0 con 0x7fe7e81509d0 2023-08-25T09:49:38.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.896+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe7e003e960 con 0x7fe7e81509d0 2023-08-25T09:49:38.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.897+0000 7fe7deffd640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7c806a050 0x7fe7c806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:38.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.897+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6781+0+0 (secure 0 0 0) 0x7fe7e00ab670 con 0x7fe7e81509d0 2023-08-25T09:49:38.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.898+0000 7fe7e659c640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7c806a050 0x7fe7c806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:38.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.898+0000 7fe7e659c640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7c806a050 0x7fe7c806c510 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8000f60 tx=0x7fe7d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:38.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:38.902+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe7e00769f0 con 0x7fe7e81509d0 2023-08-25T09:49:39.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.127+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fe7e8069160 con 0x7fe7e81509d0 2023-08-25T09:49:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.696+0000 7fe7deffd640 1 -- 172.21.15.71:0/1439996159 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v5) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe7e007a010 con 0x7fe7e81509d0 2023-08-25T09:49:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:added data pool 5 to fsmap 2023-08-25T09:49:39.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.699+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7c806a050 msgr2=0x7fe7c806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7c806a050 0x7fe7c806c510 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8000f60 tx=0x7fe7d8009040 comp rx=0 tx=0).stop 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 msgr2=0x7fe7e8153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8153120 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0036040 tx=0x7fe7e0004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 shutdown_connections 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7e80c3a40 0x7fe7e80c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7c806a050 0x7fe7c806c510 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe7e81509d0 0x7fe7e8153120 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 --2- 172.21.15.71:0/1439996159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7e80c3660 0x7fe7e81ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 >> 172.21.15.71:0/1439996159 conn(0x7fe7e80654f0 msgr2=0x7fe7e80c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:39.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 shutdown_connections 2023-08-25T09:49:39.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.700+0000 7fe7ece46640 1 -- 172.21.15.71:0/1439996159 wait complete. 2023-08-25T09:49:39.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:49:39.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.926+0000 7f062c8a7640 1 Processor -- start 2023-08-25T09:49:39.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.927+0000 7f062c8a7640 1 -- start start 2023-08-25T09:49:39.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.928+0000 7f062c8a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 0x7f06280cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.928+0000 7f062c8a7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06280b1830 con 0x7f06281531e0 2023-08-25T09:49:39.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.928+0000 7f062c8a7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0628159100 con 0x7f06280c9e00 2023-08-25T09:49:39.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.928+0000 7f062c8a7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06281593e0 con 0x7f06280cb5b0 2023-08-25T09:49:39.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.928+0000 7f062659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f06281531e0 0x7f062806a560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47038/0 (socket says 172.21.15.71:47038) 2023-08-25T09:49:39.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.928+0000 7f062659c640 1 -- 172.21.15.71:0/489811163 learned_addr learned my addr 172.21.15.71:0/489811163 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:39.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f0626d9d640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 0x7f06280cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711730071 0 0) 0x7f06280b1830 con 0x7f06281531e0 2023-08-25T09:49:39.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0614003520 con 0x7f06281531e0 2023-08-25T09:49:39.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1187442394 0 0) 0x7f0614003520 con 0x7f06281531e0 2023-08-25T09:49:39.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 msgr2=0x7f06280cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:39.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 0x7f06280cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 >> v1:172.21.15.164:6789/0 conn(0x7f06280c9e00 legacy=0x7f06280ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06281596c0 con 0x7f06281531e0 2023-08-25T09:49:39.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f06180019f0 con 0x7f06281531e0 2023-08-25T09:49:39.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 >> v1:172.21.15.71:6789/0 conn(0x7f06281531e0 legacy=0x7f062806a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0614003a10 0x7f0614003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06140047c0 0x7f0614004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 0x7f0614005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06281593e0 con 0x7f06140047c0 2023-08-25T09:49:39.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061400ba80 con 0x7f0614003a10 2023-08-25T09:49:39.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.929+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061400bc40 con 0x7f06280cb5b0 2023-08-25T09:49:39.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0625d9b640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06140047c0 0x7f0614004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f062659c640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0614003a10 0x7f0614003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0626d9d640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 0x7f0614005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0625d9b640 1 -- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 msgr2=0x7f0614005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:39.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0625d9b640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06280cb5b0 0x7f0614005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0625d9b640 1 -- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0614003a10 msgr2=0x7f0614003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:39.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0625d9b640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0614003a10 0x7f0614003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.930+0000 7f0625d9b640 1 -- 172.21.15.71:0/489811163 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0614003520 con 0x7f06140047c0 2023-08-25T09:49:39.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.931+0000 7f0625d9b640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06140047c0 0x7f0614004c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0610001750 tx=0x7f06140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d0225257bd4133 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:39.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.931+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06100052d0 con 0x7f06140047c0 2023-08-25T09:49:39.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061002e050 con 0x7f06140047c0 2023-08-25T09:49:39.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062559a640 1 -- 172.21.15.71:0/489811163 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06100035f0 con 0x7f06140047c0 2023-08-25T09:49:39.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062c8a7640 1 -- 172.21.15.71:0/489811163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06140047c0 msgr2=0x7f0614004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062c8a7640 1 --2- 172.21.15.71:0/489811163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06140047c0 0x7f0614004c40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0610001750 tx=0x7f06140057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f0626d9d640 1 -- 172.21.15.71:0/489811163 reap_dead start 2023-08-25T09:49:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062c8a7640 1 -- 172.21.15.71:0/489811163 shutdown_connections 2023-08-25T09:49:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062c8a7640 1 -- 172.21.15.71:0/489811163 >> 172.21.15.71:0/489811163 conn(0x7f062806b240 msgr2=0x7f062806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062c8a7640 1 -- 172.21.15.71:0/489811163 shutdown_connections 2023-08-25T09:49:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.932+0000 7f062c8a7640 1 -- 172.21.15.71:0/489811163 wait complete. 2023-08-25T09:49:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.934+0000 7f062c8a7640 1 Processor -- start 2023-08-25T09:49:39.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.934+0000 7f062c8a7640 1 -- start start 2023-08-25T09:49:39.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f062c8a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06140047c0 0x7f06281f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f062c8a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 0x7f062815ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f062c8a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f062815ab60 0x7f0628146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f062c8a7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f062815b6a0 con 0x7f062815bb30 2023-08-25T09:49:39.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f062c8a7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0628157770 con 0x7f06140047c0 2023-08-25T09:49:39.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f062c8a7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f062815a090 con 0x7f062815ab60 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f0625d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 0x7f062815ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f0625d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 0x7f062815ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54668/0 (socket says 172.21.15.71:54668) 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.935+0000 7f0625d9b640 1 -- 172.21.15.71:0/2265304109 learned_addr learned my addr 172.21.15.71:0/2265304109 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0626d9d640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f062815ab60 0x7f0628146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f062659c640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06140047c0 0x7f06281f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0625d9b640 1 -- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f062815ab60 msgr2=0x7f0628146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0625d9b640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f062815ab60 0x7f0628146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0625d9b640 1 -- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06140047c0 msgr2=0x7f06281f1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0625d9b640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06140047c0 0x7f06281f1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0625d9b640 1 -- 172.21.15.71:0/2265304109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062815a370 con 0x7f062815bb30 2023-08-25T09:49:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f0625d9b640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 0x7f062815ca50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0610008fd0 tx=0x7f0610001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06100076c0 con 0x7f062815bb30 2023-08-25T09:49:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0610007a30 con 0x7f062815bb30 2023-08-25T09:49:39.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.936+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0610006c00 con 0x7f062815bb30 2023-08-25T09:49:39.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.937+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f062815b330 con 0x7f062815bb30 2023-08-25T09:49:39.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.937+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06280c4860 con 0x7f062815bb30 2023-08-25T09:49:39.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.938+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0628069160 con 0x7f062815bb30 2023-08-25T09:49:39.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.939+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f06100035f0 con 0x7f062815bb30 2023-08-25T09:49:39.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.940+0000 7f061effd640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f060806a050 0x7f060806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:39.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.940+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f06100aaa80 con 0x7f062815bb30 2023-08-25T09:49:39.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.940+0000 7f062659c640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f060806a050 0x7f060806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:39.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.940+0000 7f062659c640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f060806a050 0x7f060806c510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f06180013d0 tx=0x7f0618009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:39.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:39.944+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0610079900 con 0x7f062815bb30 2023-08-25T09:49:40.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.170+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06280cb060 con 0x7f062815bb30 2023-08-25T09:49:40.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.171+0000 7f061effd640 1 -- 172.21.15.71:0/2265304109 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+6459 (secure 0 0 0) 0x7f061004b090 con 0x7f062815bb30 2023-08-25T09:49:40.173 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:40.173 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:49:39.690256+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:49:40.174 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 5 2023-08-25T09:49:40.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f060806a050 msgr2=0x7f060806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f060806a050 0x7f060806c510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f06180013d0 tx=0x7f0618009040 comp rx=0 tx=0).stop 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 msgr2=0x7f062815ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 0x7f062815ca50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0610008fd0 tx=0x7f0610001120 comp rx=0 tx=0).stop 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 shutdown_connections 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f060806a050 0x7f060806c510 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f062815ab60 0x7f0628146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f062815bb30 0x7f062815ca50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 --2- 172.21.15.71:0/2265304109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06140047c0 0x7f06281f1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 >> 172.21.15.71:0/2265304109 conn(0x7f062806b240 msgr2=0x7f062806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:40.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 shutdown_connections 2023-08-25T09:49:40.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.174+0000 7f062c8a7640 1 -- 172.21.15.71:0/2265304109 wait complete. 2023-08-25T09:49:40.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:40.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.400+0000 7f3b72445640 1 Processor -- start 2023-08-25T09:49:40.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.400+0000 7f3b72445640 1 -- start start 2023-08-25T09:49:40.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.401+0000 7f3b72445640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b6c069850 0x7f3b6c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.401+0000 7f3b72445640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b6c0c4fb0 0x7f3b6c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b72445640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6c157bf0 con 0x7f3b6c1542d0 2023-08-25T09:49:40.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b72445640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c0c4b60 con 0x7f3b6c069850 2023-08-25T09:49:40.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b72445640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6c06a130 con 0x7f3b6c0c4fb0 2023-08-25T09:49:40.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b70c42640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b6c069850 0x7f3b6c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b71443640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3b6c1542d0 0x7f3b6c0c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34014/0 (socket says 172.21.15.71:34014) 2023-08-25T09:49:40.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b70c42640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b6c069850 0x7f3b6c1576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55918/0 (socket says 172.21.15.71:55918) 2023-08-25T09:49:40.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b71443640 1 -- 172.21.15.71:0/3189813145 learned_addr learned my addr 172.21.15.71:0/3189813145 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:40.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b70c42640 1 -- 172.21.15.71:0/3189813145 >> v2:172.21.15.164:3300/0 conn(0x7f3b6c0c4fb0 msgr2=0x7f3b6c0c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:40.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b71c44640 1 --2- 172.21.15.71:0/3189813145 >> v2:172.21.15.164:3300/0 conn(0x7f3b6c0c4fb0 0x7f3b6c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b70c42640 1 --2- 172.21.15.71:0/3189813145 >> v2:172.21.15.164:3300/0 conn(0x7f3b6c0c4fb0 0x7f3b6c0c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b70c42640 1 -- 172.21.15.71:0/3189813145 >> v1:172.21.15.71:6789/0 conn(0x7f3b6c1542d0 legacy=0x7f3b6c0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b70c42640 1 -- 172.21.15.71:0/3189813145 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6c0c5980 con 0x7f3b6c069850 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.402+0000 7f3b63fff640 1 -- 172.21.15.71:0/3189813145 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637120203 0 0) 0x7f3b68001110 con 0x7f3b6c1542d0 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.403+0000 7f3b70c42640 1 --2- 172.21.15.71:0/3189813145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1576b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0c1800 tx=0x7f3b5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782b7e197e1b4361 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b63fff640 1 -- 172.21.15.71:0/3189813145 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c031830 con 0x7f3b6c069850 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b63fff640 1 -- 172.21.15.71:0/3189813145 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c031990 con 0x7f3b6c069850 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b63fff640 1 -- 172.21.15.71:0/3189813145 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c031d20 con 0x7f3b6c069850 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 -- 172.21.15.71:0/3189813145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 msgr2=0x7f3b6c1576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 --2- 172.21.15.71:0/3189813145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1576b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0c1800 tx=0x7f3b5c02b8d0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 -- 172.21.15.71:0/3189813145 shutdown_connections 2023-08-25T09:49:40.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 --2- 172.21.15.71:0/3189813145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1576b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 --2- 172.21.15.71:0/3189813145 >> v2:172.21.15.164:3300/0 conn(0x7f3b6c0c4fb0 0x7f3b6c0c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 -- 172.21.15.71:0/3189813145 >> 172.21.15.71:0/3189813145 conn(0x7f3b6c06c280 msgr2=0x7f3b6c06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:40.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.404+0000 7f3b72445640 1 -- 172.21.15.71:0/3189813145 shutdown_connections 2023-08-25T09:49:40.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.405+0000 7f3b72445640 1 -- 172.21.15.71:0/3189813145 wait complete. 2023-08-25T09:49:40.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.406+0000 7f3b72445640 1 Processor -- start 2023-08-25T09:49:40.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.406+0000 7f3b72445640 1 -- start start 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.407+0000 7f3b72445640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.407+0000 7f3b72445640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b6c0c4fb0 0x7f3b6c1ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.407+0000 7f3b72445640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b6c1542d0 0x7f3b6c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.407+0000 7f3b72445640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c14ba10 con 0x7f3b6c069850 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.407+0000 7f3b72445640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c157960 con 0x7f3b6c1542d0 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.407+0000 7f3b72445640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6c1ee430 con 0x7f3b6c0c4fb0 2023-08-25T09:49:40.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55924/0 (socket says 172.21.15.71:55924) 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 -- 172.21.15.71:0/4038214124 learned_addr learned my addr 172.21.15.71:0/4038214124 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b70c42640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b6c0c4fb0 0x7f3b6c1ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71c44640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b6c1542d0 0x7f3b6c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 -- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b6c0c4fb0 msgr2=0x7f3b6c1ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b6c0c4fb0 0x7f3b6c1ee8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 -- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b6c1542d0 msgr2=0x7f3b6c157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b6c1542d0 0x7f3b6c157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 -- 172.21.15.71:0/4038214124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009400 con 0x7f3b6c069850 2023-08-25T09:49:40.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.408+0000 7f3b71443640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1ef270 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800c780 tx=0x7f3b6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:40.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.409+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b680072d0 con 0x7f3b6c069850 2023-08-25T09:49:40.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.409+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b68007450 con 0x7f3b6c069850 2023-08-25T09:49:40.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.409+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6c1ede00 con 0x7f3b6c069850 2023-08-25T09:49:40.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.409+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b680047e0 con 0x7f3b6c069850 2023-08-25T09:49:40.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.409+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6c14a570 con 0x7f3b6c069850 2023-08-25T09:49:40.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.410+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6c04e420 con 0x7f3b6c069850 2023-08-25T09:49:40.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.411+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3b68004960 con 0x7f3b6c069850 2023-08-25T09:49:40.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.412+0000 7f3b61ffb640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b5006a050 0x7f3b5006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.412+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3b68087c60 con 0x7f3b6c069850 2023-08-25T09:49:40.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.412+0000 7f3b70c42640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b5006a050 0x7f3b5006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.413+0000 7f3b70c42640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b5006a050 0x7f3b5006c510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c036000 tx=0x7f3b5c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:40.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.416+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3b68053060 con 0x7f3b6c069850 2023-08-25T09:49:40.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.613+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b6c157bf0 con 0x7f3b6c069850 2023-08-25T09:49:40.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.615+0000 7f3b61ffb640 1 -- 172.21.15.71:0/4038214124 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7f3b6c157bf0 con 0x7f3b6c069850 2023-08-25T09:49:40.618 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:40.618 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:40.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b5006a050 msgr2=0x7f3b5006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b5006a050 0x7f3b5006c510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c036000 tx=0x7f3b5c03c040 comp rx=0 tx=0).stop 2023-08-25T09:49:40.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 msgr2=0x7f3b6c1ef270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1ef270 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800c780 tx=0x7f3b6800cc40 comp rx=0 tx=0).stop 2023-08-25T09:49:40.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 shutdown_connections 2023-08-25T09:49:40.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b6c1542d0 0x7f3b6c157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b5006a050 0x7f3b5006c510 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b6c0c4fb0 0x7f3b6c1ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 --2- 172.21.15.71:0/4038214124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b6c069850 0x7f3b6c1ef270 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 >> 172.21.15.71:0/4038214124 conn(0x7f3b6c06c280 msgr2=0x7f3b6c152890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:40.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 shutdown_connections 2023-08-25T09:49:40.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.618+0000 7f3b72445640 1 -- 172.21.15.71:0/4038214124 wait complete. 2023-08-25T09:49:40.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:40.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.852+0000 7fed396d9640 1 Processor -- start 2023-08-25T09:49:40.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.852+0000 7fed396d9640 1 -- start start 2023-08-25T09:49:40.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed396d9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed340c3660 0x7fed340c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed396d9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed341509d0 0x7fed34150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed396d9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed340c1dc0 con 0x7fed340c3a40 2023-08-25T09:49:40.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed396d9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed34157a80 con 0x7fed340c3660 2023-08-25T09:49:40.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed396d9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed34151360 con 0x7fed341509d0 2023-08-25T09:49:40.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed32ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed340c3660 0x7fed340c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed32ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed340c3660 0x7fed340c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55930/0 (socket says 172.21.15.71:55930) 2023-08-25T09:49:40.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed32ffd640 1 -- 172.21.15.71:0/3794219614 learned_addr learned my addr 172.21.15.71:0/3794219614 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.855+0000 7fed32ffd640 1 -- 172.21.15.71:0/3794219614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed341509d0 msgr2=0x7fed34150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.855+0000 7fed337fe640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed341509d0 0x7fed34150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.854+0000 7fed31ffb640 1 -- 172.21.15.71:0/3794219614 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739428496 0 0) 0x7fed340c1dc0 con 0x7fed340c3a40 2023-08-25T09:49:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.855+0000 7fed32ffd640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed341509d0 0x7fed34150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.855+0000 7fed32ffd640 1 -- 172.21.15.71:0/3794219614 >> v1:172.21.15.71:6789/0 conn(0x7fed340c3a40 legacy=0x7fed3406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.855+0000 7fed32ffd640 1 -- 172.21.15.71:0/3794219614 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed34156e00 con 0x7fed340c3660 2023-08-25T09:49:40.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.855+0000 7fed337fe640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed341509d0 0x7fed34150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:40.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.856+0000 7fed32ffd640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed340c1710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fed24009e50 tx=0x7fed2402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a611eb512ca3e80 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:40.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.856+0000 7fed31ffb640 1 -- 172.21.15.71:0/3794219614 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2402cb90 con 0x7fed340c3660 2023-08-25T09:49:40.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.856+0000 7fed31ffb640 1 -- 172.21.15.71:0/3794219614 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2402ccf0 con 0x7fed340c3660 2023-08-25T09:49:40.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.856+0000 7fed31ffb640 1 -- 172.21.15.71:0/3794219614 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed24034610 con 0x7fed340c3660 2023-08-25T09:49:40.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 -- 172.21.15.71:0/3794219614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 msgr2=0x7fed340c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed340c1710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fed24009e50 tx=0x7fed2402bff0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 -- 172.21.15.71:0/3794219614 shutdown_connections 2023-08-25T09:49:40.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed341509d0 0x7fed34150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 --2- 172.21.15.71:0/3794219614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed340c1710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 -- 172.21.15.71:0/3794219614 >> 172.21.15.71:0/3794219614 conn(0x7fed340654f0 msgr2=0x7fed34065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:40.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 -- 172.21.15.71:0/3794219614 shutdown_connections 2023-08-25T09:49:40.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.857+0000 7fed396d9640 1 -- 172.21.15.71:0/3794219614 wait complete. 2023-08-25T09:49:40.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.859+0000 7fed396d9640 1 Processor -- start 2023-08-25T09:49:40.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.859+0000 7fed396d9640 1 -- start start 2023-08-25T09:49:40.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed396d9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed34155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed396d9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed340c3a40 0x7fed3414c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed396d9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed341509d0 0x7fed34158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed396d9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3415a120 con 0x7fed340c3660 2023-08-25T09:49:40.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed396d9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3414dfe0 con 0x7fed341509d0 2023-08-25T09:49:40.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed396d9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340c1200 con 0x7fed340c3a40 2023-08-25T09:49:40.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed34155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed34155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55938/0 (socket says 172.21.15.71:55938) 2023-08-25T09:49:40.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 -- 172.21.15.71:0/1917194634 learned_addr learned my addr 172.21.15.71:0/1917194634 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:40.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed327fc640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed340c3a40 0x7fed3414c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed337fe640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed341509d0 0x7fed34158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 -- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed340c3a40 msgr2=0x7fed3414c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed340c3a40 0x7fed3414c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 -- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed341509d0 msgr2=0x7fed34158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:40.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed341509d0 0x7fed34158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:40.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed32ffd640 1 -- 172.21.15.71:0/1917194634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed24009b30 con 0x7fed340c3660 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.860+0000 7fed337fe640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed341509d0 0x7fed34158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.861+0000 7fed32ffd640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed34155dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fed24009e50 tx=0x7fed24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.861+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2403e450 con 0x7fed340c3660 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.861+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2403e5b0 con 0x7fed340c3660 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.861+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2403e940 con 0x7fed340c3660 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.861+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed34157f30 con 0x7fed340c3660 2023-08-25T09:49:40.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.861+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3414e470 con 0x7fed340c3660 2023-08-25T09:49:40.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.862+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed340c29c0 con 0x7fed340c3660 2023-08-25T09:49:40.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.863+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fed2403eaa0 con 0x7fed340c3660 2023-08-25T09:49:40.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.864+0000 7fed1b7fe640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2006a050 0x7fed2006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:40.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.864+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fed240ab790 con 0x7fed340c3660 2023-08-25T09:49:40.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.865+0000 7fed327fc640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2006a050 0x7fed2006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:40.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.865+0000 7fed327fc640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2006a050 0x7fed2006c510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed340c6250 tx=0x7fed28009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:40.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:40.869+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fed24076c10 con 0x7fed340c3660 2023-08-25T09:49:41.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.058+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed34069160 con 0x7fed340c3660 2023-08-25T09:49:41.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.060+0000 7fed1b7fe640 1 -- 172.21.15.71:0/1917194634 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7fed2402ce60 con 0x7fed340c3660 2023-08-25T09:49:41.062 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:41.062 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:41.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2006a050 msgr2=0x7fed2006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2006a050 0x7fed2006c510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed340c6250 tx=0x7fed28009040 comp rx=0 tx=0).stop 2023-08-25T09:49:41.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 msgr2=0x7fed34155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed34155dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fed24009e50 tx=0x7fed24004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 shutdown_connections 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed341509d0 0x7fed34158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2006a050 0x7fed2006c510 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed340c3a40 0x7fed3414c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 --2- 172.21.15.71:0/1917194634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed340c3660 0x7fed34155dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.062+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 >> 172.21.15.71:0/1917194634 conn(0x7fed340654f0 msgr2=0x7fed340c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:41.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.063+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 shutdown_connections 2023-08-25T09:49:41.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.063+0000 7fed396d9640 1 -- 172.21.15.71:0/1917194634 wait complete. 2023-08-25T09:49:41.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:49:41.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.316+0000 7f14f0eec640 1 Processor -- start 2023-08-25T09:49:41.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.317+0000 7f14f0eec640 1 -- start start 2023-08-25T09:49:41.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.318+0000 7f14f0eec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f14ec0c7740 0x7f14ec06a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.318+0000 7f14f0eec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 0x7f14ec1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.318+0000 7f14f0eec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec158e40 con 0x7f14ec1530c0 2023-08-25T09:49:41.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.318+0000 7f14f0eec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ec06bc80 con 0x7f14ec0c7740 2023-08-25T09:49:41.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.318+0000 7f14f0eec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1534a0 con 0x7f14ec06aea0 2023-08-25T09:49:41.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.319+0000 7f14ea59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f14ec1530c0 0x7f14ec0c6e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34034/0 (socket says 172.21.15.71:34034) 2023-08-25T09:49:41.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.319+0000 7f14ea59c640 1 -- 172.21.15.71:0/940458524 learned_addr learned my addr 172.21.15.71:0/940458524 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:41.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.319+0000 7f14ead9d640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 0x7f14ec1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.319+0000 7f14e3fff640 1 --2- 172.21.15.71:0/940458524 >> v2:172.21.15.164:3300/0 conn(0x7f14ec0c7740 0x7f14ec06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.319+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722378573 0 0) 0x7f14ec158e40 con 0x7f14ec1530c0 2023-08-25T09:49:41.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.319+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d8003520 con 0x7f14ec1530c0 2023-08-25T09:49:41.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2970168952 0 0) 0x7f14d8003520 con 0x7f14ec1530c0 2023-08-25T09:49:41.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 msgr2=0x7f14ec1592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 0x7f14ec1592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 >> v2:172.21.15.164:3300/0 conn(0x7f14ec0c7740 msgr2=0x7f14ec06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 --2- 172.21.15.71:0/940458524 >> v2:172.21.15.164:3300/0 conn(0x7f14ec0c7740 0x7f14ec06a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec15c330 con 0x7f14ec1530c0 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f14dc001670 con 0x7f14ec1530c0 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 >> v1:172.21.15.71:6789/0 conn(0x7f14ec1530c0 legacy=0x7f14ec0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14d80039e0 0x7f14d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 0x7f14d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14d8004d00 0x7f14d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1534a0 con 0x7f14d8004d00 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d800ba40 con 0x7f14d80039e0 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d800bc00 con 0x7f14ec06aea0 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14ead9d640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14d8004d00 0x7f14d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14ea59c640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14d80039e0 0x7f14d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.320+0000 7f14e3fff640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 0x7f14d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.321+0000 7f14ead9d640 1 -- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 msgr2=0x7f14d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.321+0000 7f14ead9d640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14ec06aea0 0x7f14d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.321+0000 7f14ead9d640 1 -- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14d80039e0 msgr2=0x7f14d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.321+0000 7f14ead9d640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14d80039e0 0x7f14d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.321+0000 7f14ead9d640 1 -- 172.21.15.71:0/940458524 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8003520 con 0x7f14d8004d00 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.321+0000 7f14ea59c640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14d80039e0 0x7f14d8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:41.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14ead9d640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14d8004d00 0x7f14d8005180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009fc0 tx=0x7f14d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb033a9496075b1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e402cc20 con 0x7f14d8004d00 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e400f040 con 0x7f14d8004d00 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14e9d9b640 1 -- 172.21.15.71:0/940458524 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4034440 con 0x7f14d8004d00 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14f0eec640 1 -- 172.21.15.71:0/940458524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14d8004d00 msgr2=0x7f14d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14f0eec640 1 --2- 172.21.15.71:0/940458524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14d8004d00 0x7f14d8005180 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009fc0 tx=0x7f14d80057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.322+0000 7f14ead9d640 1 -- 172.21.15.71:0/940458524 reap_dead start 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.323+0000 7f14f0eec640 1 -- 172.21.15.71:0/940458524 shutdown_connections 2023-08-25T09:49:41.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.323+0000 7f14f0eec640 1 -- 172.21.15.71:0/940458524 >> 172.21.15.71:0/940458524 conn(0x7f14ec0654f0 msgr2=0x7f14ec065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:41.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.323+0000 7f14f0eec640 1 -- 172.21.15.71:0/940458524 shutdown_connections 2023-08-25T09:49:41.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.323+0000 7f14f0eec640 1 -- 172.21.15.71:0/940458524 wait complete. 2023-08-25T09:49:41.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.324+0000 7f14f0eec640 1 Processor -- start 2023-08-25T09:49:41.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 -- start start 2023-08-25T09:49:41.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14d8004d00 0x7f14ec1f1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ec1f1780 0x7f14ec14ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 0x7f14ec157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec15c190 con 0x7f14ec14b380 2023-08-25T09:49:41.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec0cd210 con 0x7f14ec1f1780 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.325+0000 7f14f0eec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec1f1ee0 con 0x7f14d8004d00 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 0x7f14ec157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 0x7f14ec157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55958/0 (socket says 172.21.15.71:55958) 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 -- 172.21.15.71:0/2609656928 learned_addr learned my addr 172.21.15.71:0/2609656928 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ea59c640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14d8004d00 0x7f14ec1f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14e3fff640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ec1f1780 0x7f14ec14ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 -- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14d8004d00 msgr2=0x7f14ec1f1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14d8004d00 0x7f14ec1f1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 -- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ec1f1780 msgr2=0x7f14ec14ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ec1f1780 0x7f14ec14ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.326+0000 7f14ead9d640 1 -- 172.21.15.71:0/2609656928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec15bef0 con 0x7f14ec14b380 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.327+0000 7f14ead9d640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 0x7f14ec157ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f14e4037040 tx=0x7f14e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.327+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4043070 con 0x7f14ec14b380 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.327+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e40349c0 con 0x7f14ec14b380 2023-08-25T09:49:41.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.327+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4034d70 con 0x7f14ec14b380 2023-08-25T09:49:41.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.327+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec1f1c00 con 0x7f14ec14b380 2023-08-25T09:49:41.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.327+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec0ca4d0 con 0x7f14ec14b380 2023-08-25T09:49:41.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.328+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec069160 con 0x7f14ec14b380 2023-08-25T09:49:41.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.329+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f14e403e440 con 0x7f14ec14b380 2023-08-25T09:49:41.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.330+0000 7f14e2ffd640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a080 0x7f14cc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.331+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f14e40aa3b0 con 0x7f14ec14b380 2023-08-25T09:49:41.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.331+0000 7f14ea59c640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a080 0x7f14cc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.331+0000 7f14ea59c640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a080 0x7f14cc06c540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f14dc005a10 tx=0x7f14dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:41.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.335+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f14e4075710 con 0x7f14ec14b380 2023-08-25T09:49:41.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.566+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14ec06f250 con 0x7f14ec14b380 2023-08-25T09:49:41.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.567+0000 7f14e2ffd640 1 -- 172.21.15.71:0/2609656928 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+6457 (secure 0 0 0) 0x7f14e4078d30 con 0x7f14ec14b380 2023-08-25T09:49:41.569 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:41.569 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:49:40.498555+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4306},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-08-25T09:49:41.570 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 6 2023-08-25T09:49:41.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a080 msgr2=0x7f14cc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a080 0x7f14cc06c540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f14dc005a10 tx=0x7f14dc009040 comp rx=0 tx=0).stop 2023-08-25T09:49:41.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 msgr2=0x7f14ec157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 0x7f14ec157ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f14e4037040 tx=0x7f14e4009750 comp rx=0 tx=0).stop 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 shutdown_connections 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14ec1f1780 0x7f14ec14ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a080 0x7f14cc06c540 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14ec14b380 0x7f14ec157ba0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 --2- 172.21.15.71:0/2609656928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14d8004d00 0x7f14ec1f1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 >> 172.21.15.71:0/2609656928 conn(0x7f14ec0654f0 msgr2=0x7f14ec1519b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:41.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 shutdown_connections 2023-08-25T09:49:41.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.570+0000 7f14f0eec640 1 -- 172.21.15.71:0/2609656928 wait complete. 2023-08-25T09:49:41.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:41.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.805+0000 7f1f7ea44640 1 Processor -- start 2023-08-25T09:49:41.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.806+0000 7f1f7ea44640 1 -- start start 2023-08-25T09:49:41.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.807+0000 7f1f7ea44640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1f780af290 0x7f1f7806b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.807+0000 7f1f7ea44640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1f78071760 0x7f1f7806a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.807+0000 7f1f7ea44640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f7806ad90 0x7f1f7815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.807+0000 7f1f7ea44640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780bf500 con 0x7f1f780af290 2023-08-25T09:49:41.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.807+0000 7f1f7ea44640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f780c0480 con 0x7f1f78071760 2023-08-25T09:49:41.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.807+0000 7f1f7ea44640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78069850 con 0x7f1f7806ad90 2023-08-25T09:49:41.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1f780af290 0x7f1f7806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1f780af290 0x7f1f7806b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55962/0 (socket says 172.21.15.71:55962) 2023-08-25T09:49:41.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 -- 172.21.15.71:0/804433662 learned_addr learned my addr 172.21.15.71:0/804433662 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:41.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 -- 172.21.15.71:0/804433662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f7806ad90 msgr2=0x7f1f7815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:41.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 --2- 172.21.15.71:0/804433662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f7806ad90 0x7f1f7815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7d241640 1 --2- 172.21.15.71:0/804433662 >> v2:172.21.15.164:3300/0 conn(0x7f1f78071760 0x7f1f7806a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 -- 172.21.15.71:0/804433662 >> v2:172.21.15.164:3300/0 conn(0x7f1f78071760 msgr2=0x7f1f7806a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:41.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 --2- 172.21.15.71:0/804433662 >> v2:172.21.15.164:3300/0 conn(0x7f1f78071760 0x7f1f7806a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7da42640 1 -- 172.21.15.71:0/804433662 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78069ad0 con 0x7f1f780af290 2023-08-25T09:49:41.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.808+0000 7f1f7d241640 1 --2- 172.21.15.71:0/804433662 >> v2:172.21.15.164:3300/0 conn(0x7f1f78071760 0x7f1f7806a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:41.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.809+0000 7f1f7da42640 1 --2- 172.21.15.71:0/804433662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f780af290 0x7f1f7806b300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1f7815a0e0 tx=0x7f1f7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b153d6812902c75f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:41.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ca40640 1 -- 172.21.15.71:0/804433662 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74031870 con 0x7f1f780af290 2023-08-25T09:49:41.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ca40640 1 -- 172.21.15.71:0/804433662 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f740319d0 con 0x7f1f780af290 2023-08-25T09:49:41.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ca40640 1 -- 172.21.15.71:0/804433662 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74031d60 con 0x7f1f780af290 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 -- 172.21.15.71:0/804433662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f780af290 msgr2=0x7f1f7806b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/804433662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f780af290 0x7f1f7806b300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1f7815a0e0 tx=0x7f1f7402b910 comp rx=0 tx=0).stop 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 -- 172.21.15.71:0/804433662 shutdown_connections 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/804433662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f7806ad90 0x7f1f7815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/804433662 >> v2:172.21.15.164:3300/0 conn(0x7f1f78071760 0x7f1f7806a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/804433662 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f780af290 0x7f1f7806b300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 -- 172.21.15.71:0/804433662 >> 172.21.15.71:0/804433662 conn(0x7f1f7806c280 msgr2=0x7f1f7806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 -- 172.21.15.71:0/804433662 shutdown_connections 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.810+0000 7f1f7ea44640 1 -- 172.21.15.71:0/804433662 wait complete. 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.812+0000 7f1f7ea44640 1 Processor -- start 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.812+0000 7f1f7ea44640 1 -- start start 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7ea44640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 0x7f1f780c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7ea44640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f78071760 0x7f1f781ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7ea44640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780af290 0x7f1f7814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7ea44640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78153b70 con 0x7f1f7806ad90 2023-08-25T09:49:41.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7ea44640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f78153890 con 0x7f1f780af290 2023-08-25T09:49:41.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7ea44640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f781f1eb0 con 0x7f1f78071760 2023-08-25T09:49:41.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.813+0000 7f1f7da42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 0x7f1f780c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 0x7f1f780c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55978/0 (socket says 172.21.15.71:55978) 2023-08-25T09:49:41.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 -- 172.21.15.71:0/1774698797 learned_addr learned my addr 172.21.15.71:0/1774698797 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:41.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7d241640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f78071760 0x7f1f781ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 -- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f78071760 msgr2=0x7f1f781ee8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7e243640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780af290 0x7f1f7814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f78071760 0x7f1f781ee8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 -- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780af290 msgr2=0x7f1f7814ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780af290 0x7f1f7814ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 -- 172.21.15.71:0/1774698797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f740094d0 con 0x7f1f7806ad90 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7d241640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f78071760 0x7f1f781ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f7da42640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 0x7f1f780c98e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1f7402b950 tx=0x7f1f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.814+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74031860 con 0x7f1f7806ad90 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.815+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f740443f0 con 0x7f1f7806ad90 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.815+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f740447a0 con 0x7f1f7806ad90 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.815+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f781f2190 con 0x7f1f7806ad90 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.815+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f781f2750 con 0x7f1f7806ad90 2023-08-25T09:49:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.816+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f780c8840 con 0x7f1f7806ad90 2023-08-25T09:49:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.817+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1f7404d450 con 0x7f1f7806ad90 2023-08-25T09:49:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.817+0000 7f1f6e7fc640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f5c06a050 0x7f1f5c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.818+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1f7402d040 con 0x7f1f7806ad90 2023-08-25T09:49:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.818+0000 7f1f7d241640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f5c06a050 0x7f1f5c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.818+0000 7f1f7d241640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f5c06a050 0x7f1f5c06c510 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1f68006fd0 tx=0x7f1f68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:41.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:41.822+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1f74076400 con 0x7f1f7806ad90 2023-08-25T09:49:42.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.009+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f7806c990 con 0x7f1f7806ad90 2023-08-25T09:49:42.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.011+0000 7f1f6e7fc640 1 -- 172.21.15.71:0/1774698797 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7f1f74035070 con 0x7f1f7806ad90 2023-08-25T09:49:42.013 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:42.013 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:42.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f5c06a050 msgr2=0x7f1f5c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f5c06a050 0x7f1f5c06c510 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1f68006fd0 tx=0x7f1f68008040 comp rx=0 tx=0).stop 2023-08-25T09:49:42.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 msgr2=0x7f1f780c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 0x7f1f780c98e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1f7402b950 tx=0x7f1f740046c0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 shutdown_connections 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f780af290 0x7f1f7814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f5c06a050 0x7f1f5c06c510 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f78071760 0x7f1f781ee8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 --2- 172.21.15.71:0/1774698797 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f7806ad90 0x7f1f780c98e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 >> 172.21.15.71:0/1774698797 conn(0x7f1f7806c280 msgr2=0x7f1f781563a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 shutdown_connections 2023-08-25T09:49:42.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.014+0000 7f1f7ea44640 1 -- 172.21.15.71:0/1774698797 wait complete. 2023-08-25T09:49:42.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-08-25T09:49:34.063814+0000', 'modified': '2023-08-25T09:49:40.498555+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4306}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4306': {'gid': 4306, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.164:6837/881271092', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 881271092}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 881271092}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-08-25T09:49:42.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-08-25T09:49:42.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-08-25T09:49:42.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.859+0000 7fd1cc457640 1 Processor -- start 2023-08-25T09:49:42.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.860+0000 7fd1cc457640 1 -- start start 2023-08-25T09:49:42.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1cc457640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd1c4150ce0 0x7fd1c40c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1cc457640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1c41613f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1cc457640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c40c2ba0 con 0x7fd1c40c58c0 2023-08-25T09:49:42.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1cc457640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c40c1770 con 0x7fd1c4150ce0 2023-08-25T09:49:42.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1cc457640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c40c1f30 con 0x7fd1c40c76b0 2023-08-25T09:49:42.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1c99f2640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd1c40c58c0 0x7fd1c40c34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34050/0 (socket says 172.21.15.71:34050) 2023-08-25T09:49:42.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.861+0000 7fd1c99f2640 1 -- 172.21.15.71:0/774667841 learned_addr learned my addr 172.21.15.71:0/774667841 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:42.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1c41613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/774667841 >> v2:172.21.15.164:3300/0 conn(0x7fd1c4150ce0 0x7fd1c40c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955944833 0 0) 0x7fd1b4001110 con 0x7fd1c40c58c0 2023-08-25T09:49:42.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b0003520 con 0x7fd1c40c58c0 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580159428 0 0) 0x7fd1b0003520 con 0x7fd1c40c58c0 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 msgr2=0x7fd1c41613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1c41613f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 >> v2:172.21.15.164:3300/0 conn(0x7fd1c4150ce0 msgr2=0x7fd1c40c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 --2- 172.21.15.71:0/774667841 >> v2:172.21.15.164:3300/0 conn(0x7fd1c4150ce0 0x7fd1c40c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c40c6400 con 0x7fd1c40c58c0 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fd1b4001800 con 0x7fd1c40c58c0 2023-08-25T09:49:42.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 >> v1:172.21.15.71:6789/0 conn(0x7fd1c40c58c0 legacy=0x7fd1c40c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1b0004d00 0x7fd1b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c40c1770 con 0x7fd1b0003970 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c40c1f30 con 0x7fd1b0004d00 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.862+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b000b980 con 0x7fd1c40c76b0 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 msgr2=0x7fd1b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1b0004d00 msgr2=0x7fd1b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1b0004d00 0x7fd1b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/774667841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0003520 con 0x7fd1b0003970 2023-08-25T09:49:42.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.863+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c76b0 0x7fd1b00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:42.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.864+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1b0003d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009ab0 tx=0x7fd1b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50432a65121d491f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:42.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.864+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0005990 con 0x7fd1b0003970 2023-08-25T09:49:42.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.864+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0005af0 con 0x7fd1b0003970 2023-08-25T09:49:42.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.864+0000 7fd1c91f1640 1 -- 172.21.15.71:0/774667841 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0005ea0 con 0x7fd1b0003970 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1cc457640 1 -- 172.21.15.71:0/774667841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 msgr2=0x7fd1b0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1cc457640 1 --2- 172.21.15.71:0/774667841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1b0003d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009ab0 tx=0x7fd1b00057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/774667841 reap_dead start 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1cc457640 1 -- 172.21.15.71:0/774667841 shutdown_connections 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1cc457640 1 -- 172.21.15.71:0/774667841 >> 172.21.15.71:0/774667841 conn(0x7fd1c406bff0 msgr2=0x7fd1c406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1cc457640 1 -- 172.21.15.71:0/774667841 shutdown_connections 2023-08-25T09:49:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.865+0000 7fd1cc457640 1 -- 172.21.15.71:0/774667841 wait complete. 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.867+0000 7fd1cc457640 1 Processor -- start 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.867+0000 7fd1cc457640 1 -- start start 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1cc457640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1c40caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1cc457640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c84a0 0x7fd1c414c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1cc457640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1c41ef7b0 0x7fd1c413fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1cc457640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c406a460 con 0x7fd1b0003970 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1cc457640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c41480f0 con 0x7fd1c41ef7b0 2023-08-25T09:49:42.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1cc457640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c40cb020 con 0x7fd1c40c84a0 2023-08-25T09:49:42.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1ca1f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1c40caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1ca1f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1c40caae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55992/0 (socket says 172.21.15.71:55992) 2023-08-25T09:49:42.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/127354031 learned_addr learned my addr 172.21.15.71:0/127354031 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:42.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.868+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c84a0 0x7fd1c414c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c84a0 msgr2=0x7fd1c414c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c84a0 0x7fd1c414c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1c41ef7b0 msgr2=0x7fd1c413fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1c41ef7b0 0x7fd1c413fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/127354031 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c41492e0 con 0x7fd1b0003970 2023-08-25T09:49:42.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c84a0 0x7fd1c414c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:42.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1c40caae0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd1c002c2b0 tx=0x7fd1c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:42.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1baffd640 1 -- 172.21.15.71:0/127354031 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0005e70 con 0x7fd1b0003970 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1baffd640 1 -- 172.21.15.71:0/127354031 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0004030 con 0x7fd1b0003970 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1baffd640 1 -- 172.21.15.71:0/127354031 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00043e0 con 0x7fd1b0003970 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c4149640 con 0x7fd1b0003970 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.869+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c414a7d0 con 0x7fd1b0003970 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.871+0000 7fd1baffd640 1 -- 172.21.15.71:0/127354031 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd1c0003610 con 0x7fd1b0003970 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.872+0000 7fd1baffd640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd1a806a050 0x7fd1a806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.872+0000 7fd1baffd640 1 -- 172.21.15.71:0/127354031 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd1c00aa7d0 con 0x7fd1b0003970 2023-08-25T09:49:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.873+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd1a806a050 0x7fd1a806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.873+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd1a806a050 0x7fd1a806c510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4001930 tx=0x7fd1b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.873+0000 7fd1cefca740 1 Processor -- start 2023-08-25T09:49:42.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 -- start start 2023-08-25T09:49:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 0x563f893fd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f8932b190 con 0x563f893bc630 2023-08-25T09:49:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89328880 con 0x563f893bdbd0 2023-08-25T09:49:42.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.874+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f89324170 con 0x563f89407a50 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 0x563f893fd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 0x563f893fd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56006/0 (socket says 172.21.15.71:56006) 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/427948237 learned_addr learned my addr 172.21.15.71:0/427948237 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 msgr2=0x563f893be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893be4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 msgr2=0x563f893bdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.875+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/427948237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f893227f0 con 0x563f893bc630 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.876+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 0x563f893fd550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40c4040 tx=0x7fd1bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab30896d525d3d4b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.876+0000 7fd1a7fff640 1 -- 172.21.15.71:0/427948237 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc019070 con 0x563f893bc630 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.876+0000 7fd1a7fff640 1 -- 172.21.15.71:0/427948237 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00ed50 con 0x563f893bc630 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.876+0000 7fd1a7fff640 1 -- 172.21.15.71:0/427948237 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc010040 con 0x563f893bc630 2023-08-25T09:49:42.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.876+0000 7fd1cefca740 1 -- 172.21.15.71:0/427948237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 msgr2=0x563f893fd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.876+0000 7fd1cefca740 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 0x563f893fd550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40c4040 tx=0x7fd1bc00b040 comp rx=0 tx=0).stop 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 -- 172.21.15.71:0/427948237 shutdown_connections 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f893bc630 0x563f893fd550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 --2- 172.21.15.71:0/427948237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 -- 172.21.15.71:0/427948237 >> 172.21.15.71:0/427948237 conn(0x563f893ba340 msgr2=0x563f893bcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:42.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 -- 172.21.15.71:0/427948237 shutdown_connections 2023-08-25T09:49:42.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 -- 172.21.15.71:0/427948237 wait complete. 2023-08-25T09:49:42.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.877+0000 7fd1cefca740 1 -- start start 2023-08-25T09:49:42.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.878+0000 7fd1cefca740 1 Processor -- start 2023-08-25T09:49:42.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89400230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f893feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f893fb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f892c3d20 con 0x563f89407a50 2023-08-25T09:49:42.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f892c39c0 con 0x563f894118c0 2023-08-25T09:49:42.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f89089f20 con 0x563f893bdbd0 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1cefca740 10 client.0 _ll_register_callbacks cb 0x563f892db630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f893feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f893feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56022/0 (socket says 172.21.15.71:56022) 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 -- 172.21.15.71:0/1456664481 learned_addr learned my addr 172.21.15.71:0/1456664481 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89400230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 -- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 msgr2=0x563f89400230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89400230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 -- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 msgr2=0x563f893fb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f893fb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.879+0000 7fd1c99f2640 1 -- 172.21.15.71:0/1456664481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc009c70 con 0x563f89407a50 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f893feb10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40c5d00 tx=0x7fd1b4001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0aabe16f0b7bc5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1a57fa640 1 -- 172.21.15.71:0/1456664481 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400c050 con 0x563f89407a50 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1a57fa640 1 -- 172.21.15.71:0/1456664481 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd1b4013690 con 0x563f89407a50 2023-08-25T09:49:42.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f892f8540 con 0x563f89407a50 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1a57fa640 1 -- 172.21.15.71:0/1456664481 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1b4013850 con 0x563f89407a50 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1cefca740 10 client.4461 fetch_fsmap learned FSMap version 6 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x563f8931bf30 con 0x563f89407a50 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.880+0000 7fd1a57fa640 1 -- 172.21.15.71:0/1456664481 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563f892f8540 con 0x563f89407a50 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1a57fa640 1 -- 172.21.15.71:0/1456664481 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 6) v1 ==== 4452+0+0 (secure 0 0 0) 0x7fd1b40139b0 con 0x563f89407a50 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1cefca740 10 client.4461 fetch_fsmap finished waiting for FSMap version 6 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1cefca740 10 client.4461 resolve_mds: resolved name 'f' to daemon mds.f 2023-08-25T09:49:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1cefca740 20 client.4461 populate_metadata read hostname 'smithi071' 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1cefca740 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1cefca740 4 client.4461 mds_command: new command op to 4306 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 --> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f8931d150 con 0x563f893fbae0 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.881+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.882+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 msgr2=0x563f8940e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.882+0000 7fd1ca1f3640 1 -- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 msgr2=0x563f8940e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.882+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T09:49:42.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:42.882+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T09:49:43.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:43.083+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:43.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:43.085+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6dea7216be5d0aa server_cookie=73300925609293bd in_seq=0 out_seq=0 2023-08-25T09:49:43.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:43.085+0000 7fd1a57fa640 10 client.4461 ms_handle_connect on v2:172.21.15.164:6836/881271092 2023-08-25T09:49:45.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.508+0000 7fd1a57fa640 1 -- 172.21.15.71:0/1456664481 <== mds.0 v2:172.21.15.164:6836/881271092 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563f8931d150 con 0x563f893fbae0 2023-08-25T09:49:45.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.508+0000 7fd1a57fa640 10 client.4461 handle_command_reply: tid=0 2023-08-25T09:49:45.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.508+0000 7fd1cefca740 1 client.4461 shutdown 2023-08-25T09:49:45.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.509+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 msgr2=0x563f893feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.509+0000 7fd1cefca740 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f893feb10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40c5d00 tx=0x7fd1b4001b90 comp rx=0 tx=0).stop 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 shutdown_connections 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f893fbae0 0x563f8940e9f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1a57fa640 0 client.4461 ms_handle_reset on v2:172.21.15.164:6836/881271092 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f893fb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f893feb10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 --2- 172.21.15.71:0/1456664481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89400230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 >> 172.21.15.71:0/1456664481 conn(0x563f893bc630 msgr2=0x563f893e72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:45.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 shutdown_connections 2023-08-25T09:49:45.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.510+0000 7fd1cefca740 1 -- 172.21.15.71:0/1456664481 wait complete. 2023-08-25T09:49:45.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.511+0000 7fd1cefca740 20 client.4461 trim_cache size 0 max 16384 2023-08-25T09:49:45.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.533+0000 7fd1cefca740 1 Processor -- start 2023-08-25T09:49:45.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.533+0000 7fd1cefca740 1 -- start start 2023-08-25T09:49:45.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 0x563f893e6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1cefca740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f893ad310 con 0x563f894118c0 2023-08-25T09:49:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f893fff20 con 0x563f893bdbd0 2023-08-25T09:49:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f893b9760 con 0x563f89407a50 2023-08-25T09:49:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1ca9f4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 0x563f893e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1ca9f4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 0x563f893e6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56030/0 (socket says 172.21.15.71:56030) 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/4117996060 learned_addr learned my addr 172.21.15.71:0/4117996060 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.535+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 msgr2=0x563f893e5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893e5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 msgr2=0x563f893bce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca9f4640 1 -- 172.21.15.71:0/4117996060 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f892c2a10 con 0x563f894118c0 2023-08-25T09:49:45.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 0x563f893e6170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd1c406bda0 tx=0x7fd1bc011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d17633740892892 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:45.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1b8ff9640 1 -- 172.21.15.71:0/4117996060 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc010030 con 0x563f894118c0 2023-08-25T09:49:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1b8ff9640 1 -- 172.21.15.71:0/4117996060 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc01e670 con 0x563f894118c0 2023-08-25T09:49:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.536+0000 7fd1b8ff9640 1 -- 172.21.15.71:0/4117996060 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc019040 con 0x563f894118c0 2023-08-25T09:49:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 -- 172.21.15.71:0/4117996060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 msgr2=0x563f893e6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 0x563f893e6170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd1c406bda0 tx=0x7fd1bc011000 comp rx=0 tx=0).stop 2023-08-25T09:49:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 -- 172.21.15.71:0/4117996060 shutdown_connections 2023-08-25T09:49:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f894118c0 0x563f893e6170 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f89407a50 0x563f893e5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 --2- 172.21.15.71:0/4117996060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f893bdbd0 0x563f893bce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 -- 172.21.15.71:0/4117996060 >> 172.21.15.71:0/4117996060 conn(0x563f893bc630 msgr2=0x563f893e7320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 -- 172.21.15.71:0/4117996060 shutdown_connections 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.537+0000 7fd1cefca740 1 -- 172.21.15.71:0/4117996060 wait complete. 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.538+0000 7fd1cefca740 1 -- start start 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.538+0000 7fd1cefca740 1 Processor -- start 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.539+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89406fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.539+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f8946c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1cefca740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f89465500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1cefca740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f894074e0 con 0x563f89407a50 2023-08-25T09:49:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f8946b940 con 0x563f894118c0 2023-08-25T09:49:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1cefca740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f8946d0f0 con 0x563f893bdbd0 2023-08-25T09:49:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1cefca740 10 client.0 _ll_register_callbacks cb 0x7fd1b400e8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:49:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f8946c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f8946c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56038/0 (socket says 172.21.15.71:56038) 2023-08-25T09:49:45.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 -- 172.21.15.71:0/960701335 learned_addr learned my addr 172.21.15.71:0/960701335 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:45.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89406fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f89465500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 -- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 msgr2=0x563f89406fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89406fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 -- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 msgr2=0x563f89465500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f89465500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1c99f2640 1 -- 172.21.15.71:0/960701335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f893227f0 con 0x563f89407a50 2023-08-25T09:49:45.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.540+0000 7fd1ca9f4640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f89465500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:45.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1c99f2640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f8946c350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e600 tx=0x7fd1b4007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4da7141cf1cede7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:45.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:49:45.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1a57fa640 1 -- 172.21.15.71:0/960701335 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400c040 con 0x563f89407a50 2023-08-25T09:49:45.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1bc00ed50 con 0x563f89407a50 2023-08-25T09:49:45.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1a57fa640 1 -- 172.21.15.71:0/960701335 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd1b40138a0 con 0x563f89407a50 2023-08-25T09:49:45.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1a57fa640 1 -- 172.21.15.71:0/960701335 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1b4005c80 con 0x563f89407a50 2023-08-25T09:49:45.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1cefca740 10 client.4467 fetch_fsmap learned FSMap version 6 2023-08-25T09:49:45.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.541+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x563f892e7de0 con 0x563f89407a50 2023-08-25T09:49:45.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1a57fa640 1 -- 172.21.15.71:0/960701335 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00ed50 con 0x563f89407a50 2023-08-25T09:49:45.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1a57fa640 1 -- 172.21.15.71:0/960701335 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 6) v1 ==== 4452+0+0 (secure 0 0 0) 0x7fd1b4040470 con 0x563f89407a50 2023-08-25T09:49:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1cefca740 10 client.4467 fetch_fsmap finished waiting for FSMap version 6 2023-08-25T09:49:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1cefca740 10 client.4467 resolve_mds: resolved name 'f' to daemon mds.f 2023-08-25T09:49:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1cefca740 20 client.4467 populate_metadata read hostname 'smithi071' 2023-08-25T09:49:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1cefca740 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f8946ef30 0x563f8946f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1cefca740 4 client.4467 mds_command: new command op to 4306 tid=0[{"prefix": "status"}] 2023-08-25T09:49:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 --> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563f88fd35e0 con 0x563f8946ef30 2023-08-25T09:49:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.542+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f8946ef30 0x563f8946f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.543+0000 7fd1ca1f3640 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f8946ef30 0x563f8946f310 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=add369da6e4a2827 server_cookie=c7ff3ff16d8532f in_seq=0 out_seq=0 2023-08-25T09:49:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.543+0000 7fd1a57fa640 10 client.4467 ms_handle_connect on v2:172.21.15.164:6836/881271092 2023-08-25T09:49:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1a57fa640 1 -- 172.21.15.71:0/960701335 <== mds.0 v2:172.21.15.164:6836/881271092 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x563f88fd35e0 con 0x563f8946ef30 2023-08-25T09:49:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1a57fa640 10 client.4467 handle_command_reply: tid=0 2023-08-25T09:49:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 client.4467 shutdown 2023-08-25T09:49:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 msgr2=0x563f8946c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f8946c350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e600 tx=0x7fd1b4007000 comp rx=0 tx=0).stop 2023-08-25T09:49:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 shutdown_connections 2023-08-25T09:49:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x563f8946ef30 0x563f8946f310 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563f894118c0 0x563f89465500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563f89407a50 0x563f8946c350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 --2- 172.21.15.71:0/960701335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x563f893bdbd0 0x563f89406fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1a57fa640 0 client.4467 ms_handle_reset on v2:172.21.15.164:6836/881271092 2023-08-25T09:49:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 >> 172.21.15.71:0/960701335 conn(0x563f893bc630 msgr2=0x563f893e5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:45.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 shutdown_connections 2023-08-25T09:49:45.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.544+0000 7fd1cefca740 1 -- 172.21.15.71:0/960701335 wait complete. 2023-08-25T09:49:45.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cefca740 20 client.4467 trim_cache size 0 max 16384 2023-08-25T09:49:45.562 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:49:45.562 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "d3b06f22-7372-48e3-a060-a2382f01af3f", 2023-08-25T09:49:45.563 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 0, 2023-08-25T09:49:45.563 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4306, 2023-08-25T09:49:45.563 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2023-08-25T09:49:45.564 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2023-08-25T09:49:45.564 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2023-08-25T09:49:45.564 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 10.932145564000001, 2023-08-25T09:49:45.564 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 6, 2023-08-25T09:49:45.565 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 29, 2023-08-25T09:49:45.565 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 29, 2023-08-25T09:49:45.565 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 17.778347902 2023-08-25T09:49:45.566 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:49:45.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd1a806a050 msgr2=0x7fd1a806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd1a806a050 0x7fd1a806c510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4001930 tx=0x7fd1b4009040 comp rx=0 tx=0).stop 2023-08-25T09:49:45.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 msgr2=0x7fd1c40caae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1c40caae0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd1c002c2b0 tx=0x7fd1c0004730 comp rx=0 tx=0).stop 2023-08-25T09:49:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 shutdown_connections 2023-08-25T09:49:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1c41ef7b0 0x7fd1c413fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd1a806a050 0x7fd1a806c510 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1c40c84a0 0x7fd1c414c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 --2- 172.21.15.71:0/127354031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1b0003970 0x7fd1c40caae0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 >> 172.21.15.71:0/127354031 conn(0x7fd1c406bff0 msgr2=0x7fd1c406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:45.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 shutdown_connections 2023-08-25T09:49:45.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.545+0000 7fd1cc457640 1 -- 172.21.15.71:0/127354031 wait complete. 2023-08-25T09:49:45.569 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-08-25T09:49:45.570 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:45.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-08-25T09:49:45.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.780+0000 7f79d484b640 1 Processor -- start 2023-08-25T09:49:45.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.780+0000 7f79d484b640 1 -- start start 2023-08-25T09:49:45.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79d484b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f79d0159490 0x7f79d00ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79d484b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00b1af0 con 0x7f79d00cb570 2023-08-25T09:49:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79d484b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00b0fb0 con 0x7f79d00cb190 2023-08-25T09:49:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79d484b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d00c98f0 con 0x7f79d0159490 2023-08-25T09:49:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79c7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f79d00cb570 0x7f79d0158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34052/0 (socket says 172.21.15.71:34052) 2023-08-25T09:49:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79c7fff640 1 -- 172.21.15.71:0/1503771465 learned_addr learned my addr 172.21.15.71:0/1503771465 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79cffff640 1 --2- 172.21.15.71:0/1503771465 >> v2:172.21.15.164:3300/0 conn(0x7f79d0159490 0x7f79d00ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.782+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730029546 0 0) 0x7f79d00b1af0 con 0x7f79d00cb570 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc003520 con 0x7f79d00cb570 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 136117125 0 0) 0x7f79bc003520 con 0x7f79d00cb570 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 >> v2:172.21.15.164:3300/0 conn(0x7f79d0159490 msgr2=0x7f79d00ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 --2- 172.21.15.71:0/1503771465 >> v2:172.21.15.164:3300/0 conn(0x7f79d0159490 0x7f79d00ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 >> v1:172.21.15.164:6789/0 conn(0x7f79d00cb190 legacy=0x7f79d00c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d00b1530 con 0x7f79d00cb570 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f79b8001670 con 0x7f79d00cb570 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 >> v1:172.21.15.71:6789/0 conn(0x7f79d00cb570 legacy=0x7f79d0158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0039e0 0x7f79bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79bc0047c0 0x7f79bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79bc0052a0 0x7f79bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00c98f0 con 0x7f79bc0052a0 2023-08-25T09:49:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc00c160 con 0x7f79bc0039e0 2023-08-25T09:49:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79bc00c320 con 0x7f79bc0047c0 2023-08-25T09:49:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.783+0000 7f79cffff640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79bc0052a0 0x7f79bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cf7fe640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0039e0 0x7f79bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79c7fff640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79bc0047c0 0x7f79bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cffff640 1 -- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79bc0047c0 msgr2=0x7f79bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cffff640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79bc0047c0 0x7f79bc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cffff640 1 -- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0039e0 msgr2=0x7f79bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cffff640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0039e0 0x7f79bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cffff640 1 -- 172.21.15.71:0/1503771465 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc003520 con 0x7f79bc0052a0 2023-08-25T09:49:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.784+0000 7f79cf7fe640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0039e0 0x7f79bc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.785+0000 7f79cffff640 1 -- 172.21.15.71:0/1503771465 reap_dead start 2023-08-25T09:49:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.785+0000 7f79cffff640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79bc0052a0 0x7f79bc00b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79d006b9d0 tx=0x7f79bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a4da14a941fae server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.785+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c802cca0 con 0x7f79bc0052a0 2023-08-25T09:49:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.785+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c802ce00 con 0x7f79bc0052a0 2023-08-25T09:49:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.785+0000 7f79ceffd640 1 -- 172.21.15.71:0/1503771465 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8003bd0 con 0x7f79bc0052a0 2023-08-25T09:49:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 -- 172.21.15.71:0/1503771465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79bc0052a0 msgr2=0x7f79bc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79bc0052a0 0x7f79bc00b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f79d006b9d0 tx=0x7f79bc0057f0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 -- 172.21.15.71:0/1503771465 shutdown_connections 2023-08-25T09:49:45.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 --2- 172.21.15.71:0/1503771465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79bc0052a0 0x7f79bc00b980 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 -- 172.21.15.71:0/1503771465 >> 172.21.15.71:0/1503771465 conn(0x7f79d006ae20 msgr2=0x7f79d006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 -- 172.21.15.71:0/1503771465 shutdown_connections 2023-08-25T09:49:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.786+0000 7f79d484b640 1 -- 172.21.15.71:0/1503771465 wait complete. 2023-08-25T09:49:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.788+0000 7f79d484b640 1 Processor -- start 2023-08-25T09:49:45.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.788+0000 7f79d484b640 1 -- start start 2023-08-25T09:49:45.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79d484b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0052a0 0x7f79d014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79d484b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5b30 0x7f79d0151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79d484b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 0x7f79d01548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79d484b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d01501a0 con 0x7f79d0151d70 2023-08-25T09:49:45.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79d484b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d015b0d0 con 0x7f79bc0052a0 2023-08-25T09:49:45.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79d484b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d01ee7d0 con 0x7f79d00c5b30 2023-08-25T09:49:45.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 0x7f79d01548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 0x7f79d01548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56054/0 (socket says 172.21.15.71:56054) 2023-08-25T09:49:45.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 -- 172.21.15.71:0/2115147482 learned_addr learned my addr 172.21.15.71:0/2115147482 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:45.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 -- 172.21.15.71:0/2115147482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5b30 msgr2=0x7f79d0151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5b30 0x7f79d0151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 -- 172.21.15.71:0/2115147482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0052a0 msgr2=0x7f79d014fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0052a0 0x7f79d014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.789+0000 7f79cffff640 1 -- 172.21.15.71:0/2115147482 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d015b370 con 0x7f79d0151d70 2023-08-25T09:49:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.790+0000 7f79cffff640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 0x7f79d01548f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f79c802c710 tx=0x7f79c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.790+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8039070 con 0x7f79d0151d70 2023-08-25T09:49:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.790+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c8003e10 con 0x7f79d0151d70 2023-08-25T09:49:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.790+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c8004490 con 0x7f79d0151d70 2023-08-25T09:49:45.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.790+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d015b650 con 0x7f79d0151d70 2023-08-25T09:49:45.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.790+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d0159e20 con 0x7f79d0151d70 2023-08-25T09:49:45.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.791+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d015b7e0 con 0x7f79d0151d70 2023-08-25T09:49:45.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.792+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f79c803d070 con 0x7f79d0151d70 2023-08-25T09:49:45.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.793+0000 7f79ccff9640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a080 0x7f79b006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:45.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.793+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f79c80a9f60 con 0x7f79d0151d70 2023-08-25T09:49:45.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.794+0000 7f79cf7fe640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a080 0x7f79b006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:45.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.794+0000 7f79cf7fe640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a080 0x7f79b006c540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79c0000f60 tx=0x7f79c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:45.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:45.797+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f79c80753d0 con 0x7f79d0151d70 2023-08-25T09:49:46.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.024+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f79d004e420 con 0x7f79d0151d70 2023-08-25T09:49:46.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.025+0000 7f79ccff9640 1 -- 172.21.15.71:0/2115147482 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f79c8044030 con 0x7f79d0151d70 2023-08-25T09:49:46.027 INFO:teuthology.orchestra.run.smithi071.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-08-25T09:49:46.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.027+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a080 msgr2=0x7f79b006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:46.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.027+0000 7f79d484b640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a080 0x7f79b006c540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79c0000f60 tx=0x7f79c0009040 comp rx=0 tx=0).stop 2023-08-25T09:49:46.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.027+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 msgr2=0x7f79d01548f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:46.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 0x7f79d01548f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f79c802c710 tx=0x7f79c8004ab0 comp rx=0 tx=0).stop 2023-08-25T09:49:46.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 shutdown_connections 2023-08-25T09:49:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a080 0x7f79b006c540 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d0151d70 0x7f79d01548f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5b30 0x7f79d0151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:46.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 --2- 172.21.15.71:0/2115147482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79bc0052a0 0x7f79d014fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:46.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 >> 172.21.15.71:0/2115147482 conn(0x7f79d006ae20 msgr2=0x7f79d00b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:46.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 shutdown_connections 2023-08-25T09:49:46.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:46.028+0000 7f79d484b640 1 -- 172.21.15.71:0/2115147482 wait complete. 2023-08-25T09:49:46.048 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-08-25T09:49:46.049 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-08-25T09:49:46.049 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi071.front.sepia.ceph.com 2023-08-25T09:49:46.049 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-08-25T09:49:46.049 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:46.050 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-08-25T09:49:46.051 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-08-25T09:49:46.051 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-08-25T09:49:46.051 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-08-25T09:49:46.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:46.052 DEBUG:teuthology.orchestra.run.smithi071:> ip addr 2023-08-25T09:49:46.107 INFO:teuthology.orchestra.run.smithi071.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-08-25T09:49:46.107 INFO:teuthology.orchestra.run.smithi071.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-08-25T09:49:46.108 INFO:teuthology.orchestra.run.smithi071.stdout: inet 127.0.0.1/8 scope host lo 2023-08-25T09:49:46.108 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2023-08-25T09:49:46.108 INFO:teuthology.orchestra.run.smithi071.stdout: inet6 ::1/128 scope host 2023-08-25T09:49:46.108 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2023-08-25T09:49:46.109 INFO:teuthology.orchestra.run.smithi071.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T09:49:46.109 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:d8:aa:4a brd ff:ff:ff:ff:ff:ff 2023-08-25T09:49:46.109 INFO:teuthology.orchestra.run.smithi071.stdout: altname enp7s0f0 2023-08-25T09:49:46.109 INFO:teuthology.orchestra.run.smithi071.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T09:49:46.109 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:d8:aa:4b brd ff:ff:ff:ff:ff:ff 2023-08-25T09:49:46.109 INFO:teuthology.orchestra.run.smithi071.stdout: altname enp7s0f1 2023-08-25T09:49:46.110 INFO:teuthology.orchestra.run.smithi071.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T09:49:46.110 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:8f:d7:64 brd ff:ff:ff:ff:ff:ff 2023-08-25T09:49:46.110 INFO:teuthology.orchestra.run.smithi071.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-08-25T09:49:46.110 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:8f:d7:65 brd ff:ff:ff:ff:ff:ff 2023-08-25T09:49:46.110 INFO:teuthology.orchestra.run.smithi071.stdout: inet 172.21.15.71/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-08-25T09:49:46.111 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft 42332sec preferred_lft 42332sec 2023-08-25T09:49:46.111 INFO:teuthology.orchestra.run.smithi071.stdout: inet6 fe80::ec4:7aff:fe8f:d765/64 scope link noprefixroute 2023-08-25T09:49:46.111 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2023-08-25T09:49:46.111 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-08-25T09:49:46.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:46.112 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:49:46.113 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:49:46.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link add name ceph-brx type bridge 2023-08-25T09:49:46.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip addr flush dev ceph-brx 2023-08-25T09:49:46.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set ceph-brx up 2023-08-25T09:49:46.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-08-25T09:49:46.114 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:49:46.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:46.579 DEBUG:teuthology.orchestra.run.smithi071:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-08-25T09:49:46.656 INFO:teuthology.orchestra.run.smithi071.stdout:1 2023-08-25T09:49:46.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:46.660 DEBUG:teuthology.orchestra.run.smithi071:> route 2023-08-25T09:49:46.715 INFO:teuthology.orchestra.run.smithi071.stdout:Kernel IP routing table 2023-08-25T09:49:46.716 INFO:teuthology.orchestra.run.smithi071.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-08-25T09:49:46.716 INFO:teuthology.orchestra.run.smithi071.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-08-25T09:49:46.716 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-08-25T09:49:46.717 INFO:teuthology.orchestra.run.smithi071.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-08-25T09:49:46.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:46.718 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:49:46.718 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:49:46.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-08-25T09:49:46.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-08-25T09:49:46.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-08-25T09:49:46.719 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:49:47.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:47.530 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list 2023-08-25T09:49:47.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:47.587 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list-id 2023-08-25T09:49:47.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:47.641 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:49:47.641 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:49:47.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-08-25T09:49:47.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-08-25T09:49:47.642 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:49:47.791 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-08-25T09:49:47.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:47.792 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:49:47.792 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:49:47.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-08-25T09:49:47.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-08-25T09:49:47.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-08-25T09:49:47.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-08-25T09:49:47.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-08-25T09:49:47.794 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:49:48.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:48.087 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:49:48.087 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:49:48.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set brx.0 up 2023-08-25T09:49:48.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set dev brx.0 master ceph-brx 2023-08-25T09:49:48.088 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:49:48.260 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-08-25T09:49:48.260 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-08-25T09:49:48.261 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:48.308 INFO:teuthology.orchestra.run.smithi071.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-08-25T09:49:48.308 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-08-25T09:49:48.309 DEBUG:teuthology.orchestra.run.smithi071:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:48.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe fuse 2023-08-25T09:49:48.451 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-08-25T09:49:48.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-08-25T09:49:48.548 INFO:teuthology.orchestra.run.smithi071.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-08-25T09:49:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-08-25T09:49:48.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:48.554 DEBUG:teuthology.orchestra.run.smithi071:> ls /sys/fs/fuse/connections 2023-08-25T09:49:48.611 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-08-25T09:49:48.611 DEBUG:teuthology.orchestra.run.smithi071:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-08-25T09:49:48.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe fuse 2023-08-25T09:49:48.694 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-08-25T09:49:48.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-08-25T09:49:48.755 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T09:49:48.752+0000 7f5fc1b18640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-08-25T09:49:48.770 INFO:teuthology.orchestra.run.smithi071.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-08-25T09:49:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-08-25T09:49:48.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:48.774 DEBUG:teuthology.orchestra.run.smithi071:> ls /sys/fs/fuse/connections 2023-08-25T09:49:48.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi071.stderr:2023-08-25T09:49:48.806+0000 7f84bf352480 -1 init, newargv = 0x55ad426a2cd0 newargc=15 2023-08-25T09:49:48.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi071.stderr:ceph-fuse[94699]: starting ceph client 2023-08-25T09:49:48.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi071.stderr:ceph-fuse[94699]: starting fuse 2023-08-25T09:49:49.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:49:49.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:49:49.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe fuse 2023-08-25T09:49:50.023 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-08-25T09:49:50.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-08-25T09:49:50.114 INFO:teuthology.orchestra.run.smithi071.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-08-25T09:49:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-08-25T09:49:50.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:50.118 DEBUG:teuthology.orchestra.run.smithi071:> ls /sys/fs/fuse/connections 2023-08-25T09:49:50.168 INFO:teuthology.orchestra.run.smithi071.stdout:39 2023-08-25T09:49:50.168 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-08-25T09:49:50.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo stdin-killer -- python3 -c ' 2023-08-25T09:49:50.169 DEBUG:teuthology.orchestra.run.smithi071:> import glob 2023-08-25T09:49:50.169 DEBUG:teuthology.orchestra.run.smithi071:> import re 2023-08-25T09:49:50.169 DEBUG:teuthology.orchestra.run.smithi071:> import os 2023-08-25T09:49:50.170 DEBUG:teuthology.orchestra.run.smithi071:> import subprocess 2023-08-25T09:49:50.170 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:49:50.170 DEBUG:teuthology.orchestra.run.smithi071:> def _find_admin_socket(client_name): 2023-08-25T09:49:50.171 DEBUG:teuthology.orchestra.run.smithi071:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-08-25T09:49:50.171 DEBUG:teuthology.orchestra.run.smithi071:> files = glob.glob(asok_path) 2023-08-25T09:49:50.171 DEBUG:teuthology.orchestra.run.smithi071:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-08-25T09:49:50.171 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:49:50.172 DEBUG:teuthology.orchestra.run.smithi071:> # Given a non-glob path, it better be there 2023-08-25T09:49:50.172 DEBUG:teuthology.orchestra.run.smithi071:> if "*" not in asok_path: 2023-08-25T09:49:50.172 DEBUG:teuthology.orchestra.run.smithi071:> assert(len(files) == 1) 2023-08-25T09:49:50.172 DEBUG:teuthology.orchestra.run.smithi071:> return files[0] 2023-08-25T09:49:50.173 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:49:50.173 DEBUG:teuthology.orchestra.run.smithi071:> for f in files: 2023-08-25T09:49:50.173 DEBUG:teuthology.orchestra.run.smithi071:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-08-25T09:49:50.174 DEBUG:teuthology.orchestra.run.smithi071:> if os.path.exists("/proc/{0}".format(pid)): 2023-08-25T09:49:50.174 DEBUG:teuthology.orchestra.run.smithi071:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-08-25T09:49:50.174 DEBUG:teuthology.orchestra.run.smithi071:> contents = proc_f.read() 2023-08-25T09:49:50.174 DEBUG:teuthology.orchestra.run.smithi071:> if mountpoint in contents: 2023-08-25T09:49:50.175 DEBUG:teuthology.orchestra.run.smithi071:> return f 2023-08-25T09:49:50.175 DEBUG:teuthology.orchestra.run.smithi071:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-08-25T09:49:50.175 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:49:50.175 DEBUG:teuthology.orchestra.run.smithi071:> print(_find_admin_socket("client.admin")) 2023-08-25T09:49:50.176 DEBUG:teuthology.orchestra.run.smithi071:> ' 2023-08-25T09:49:50.354 INFO:teuthology.orchestra.run.smithi071.stdout:/var/run/ceph/ceph-client.admin.94699.asok 2023-08-25T09:49:50.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-08-25T09:49:50.372 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.94699.asok 2023-08-25T09:49:50.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:50.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.94699.asok status 2023-08-25T09:49:50.616 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:49:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "metadata": { 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_sha1": "ba6b39e4f4f77fb95c747099b53db0467d435f56", 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_version": "ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev)", 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "entity_id": "admin", 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "hostname": "smithi071", 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "pid": "94699", 2023-08-25T09:49:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "root": "/" 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "dentry_count": 0, 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "dentry_pinned_count": 0, 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4479, 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "inst": { 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "name": { 2023-08-25T09:49:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "client", 2023-08-25T09:49:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "num": 4479 2023-08-25T09:49:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:49:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": { 2023-08-25T09:49:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "v1", 2023-08-25T09:49:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": "192.168.0.1:0", 2023-08-25T09:49:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: "nonce": 12903202 2023-08-25T09:49:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-08-25T09:49:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:49:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": { 2023-08-25T09:49:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "v1", 2023-08-25T09:49:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": "192.168.0.1:0", 2023-08-25T09:49:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "nonce": 12903202 2023-08-25T09:49:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:49:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "inst_str": "client.4479 192.168.0.1:0/12903202", 2023-08-25T09:49:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "addr_str": "192.168.0.1:0/12903202", 2023-08-25T09:49:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "inode_count": 1, 2023-08-25T09:49:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_epoch": 6, 2023-08-25T09:49:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_epoch": 29, 2023-08-25T09:49:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_epoch_barrier": 0, 2023-08-25T09:49:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "blocklisted": false, 2023-08-25T09:49:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs" 2023-08-25T09:49:50.623 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:49:50.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:50.634 DEBUG:teuthology.orchestra.run.smithi071:> stat --file-system '--printf=%T 2023-08-25T09:49:50.634 DEBUG:teuthology.orchestra.run.smithi071:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:50.689 INFO:teuthology.orchestra.run.smithi071.stdout:fuseblk 2023-08-25T09:49:50.689 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:50.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:50.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:50.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:50.790 DEBUG:teuthology.orchestra.run.smithi071:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-08-25T09:49:51.215 INFO:teuthology.orchestra.run.smithi071.stdout:# file: . 2023-08-25T09:49:51.215 INFO:teuthology.orchestra.run.smithi071.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-08-25T09:49:51.216 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:51.217 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:51.217 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T09:49:51.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:51.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.493+0000 7fc13910f640 1 Processor -- start 2023-08-25T09:49:51.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.493+0000 7fc13910f640 1 -- start start 2023-08-25T09:49:51.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13910f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13910f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc134159830 0x7fc134069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13910f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc134159c80 con 0x7fc13406bed0 2023-08-25T09:49:51.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13910f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13415ca50 con 0x7fc134159830 2023-08-25T09:49:51.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13910f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13414fc80 con 0x7fc13406a530 2023-08-25T09:49:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc134159830 0x7fc134069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc134159830 0x7fc134069850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43482/0 (socket says 172.21.15.71:43482) 2023-08-25T09:49:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.495+0000 7fc13359e640 1 -- 172.21.15.71:0/2648423740 learned_addr learned my addr 172.21.15.71:0/2648423740 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.496+0000 7fc132d9d640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.496+0000 7fc13359e640 1 -- 172.21.15.71:0/2648423740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 msgr2=0x7fc13415cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:51.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.496+0000 7fc13359e640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13415cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.496+0000 7fc13359e640 1 -- 172.21.15.71:0/2648423740 >> v1:172.21.15.164:6789/0 conn(0x7fc13406bed0 legacy=0x7fc1340c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.496+0000 7fc13359e640 1 -- 172.21.15.71:0/2648423740 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13406dae0 con 0x7fc134159830 2023-08-25T09:49:51.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.496+0000 7fc132d9d640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13415cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:51.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.497+0000 7fc13359e640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc134069850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009e50 tx=0x7fc12c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8373a4a02ce8822 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:51.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.497+0000 7fc13259c640 1 -- 172.21.15.71:0/2648423740 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c02ca80 con 0x7fc134159830 2023-08-25T09:49:51.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.497+0000 7fc13259c640 1 -- 172.21.15.71:0/2648423740 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c02cbe0 con 0x7fc134159830 2023-08-25T09:49:51.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.497+0000 7fc13259c640 1 -- 172.21.15.71:0/2648423740 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c0343f0 con 0x7fc134159830 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 -- 172.21.15.71:0/2648423740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 msgr2=0x7fc134069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc134069850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009e50 tx=0x7fc12c02bee0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 -- 172.21.15.71:0/2648423740 shutdown_connections 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc134069850 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 --2- 172.21.15.71:0/2648423740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13415cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 -- 172.21.15.71:0/2648423740 >> 172.21.15.71:0/2648423740 conn(0x7fc1340654f0 msgr2=0x7fc134065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:51.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 -- 172.21.15.71:0/2648423740 shutdown_connections 2023-08-25T09:49:51.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.498+0000 7fc13910f640 1 -- 172.21.15.71:0/2648423740 wait complete. 2023-08-25T09:49:51.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.500+0000 7fc13910f640 1 Processor -- start 2023-08-25T09:49:51.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.500+0000 7fc13910f640 1 -- start start 2023-08-25T09:49:51.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13910f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13414e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13910f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc13406bed0 0x7fc1341f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13910f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc1341f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13910f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134154860 con 0x7fc134159830 2023-08-25T09:49:51.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13910f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134150950 con 0x7fc13406bed0 2023-08-25T09:49:51.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13910f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1341ee5d0 con 0x7fc13406a530 2023-08-25T09:49:51.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc1341f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc1341f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43486/0 (socket says 172.21.15.71:43486) 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.501+0000 7fc13359e640 1 -- 172.21.15.71:0/128884168 learned_addr learned my addr 172.21.15.71:0/128884168 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc132d9d640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13414e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc12a59c640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc13406bed0 0x7fc1341f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13359e640 1 -- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 msgr2=0x7fc13414e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13359e640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13414e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13359e640 1 -- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc13406bed0 msgr2=0x7fc1341f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13359e640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc13406bed0 0x7fc1341f1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13359e640 1 -- 172.21.15.71:0/128884168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c009b30 con 0x7fc134159830 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13359e640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc1341f2100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009e50 tx=0x7fc12c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c009510 con 0x7fc134159830 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12c03e450 con 0x7fc134159830 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c03e800 con 0x7fc134159830 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.502+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1341506b0 con 0x7fc134159830 2023-08-25T09:49:51.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.503+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1341eeac0 con 0x7fc134159830 2023-08-25T09:49:51.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.504+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13415c020 con 0x7fc134159830 2023-08-25T09:49:51.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.505+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc12c02cd50 con 0x7fc134159830 2023-08-25T09:49:51.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.506+0000 7fc12bfff640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc11406a050 0x7fc11406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.506+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc12c0ab420 con 0x7fc134159830 2023-08-25T09:49:51.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.506+0000 7fc132d9d640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc11406a050 0x7fc11406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.507+0000 7fc132d9d640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc11406a050 0x7fc11406c510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc124006fd0 tx=0x7fc124008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:51.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.511+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc12c0766f0 con 0x7fc134159830 2023-08-25T09:49:51.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.702+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc134070a00 con 0x7fc134159830 2023-08-25T09:49:51.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.704+0000 7fc12bfff640 1 -- 172.21.15.71:0/128884168 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7fc12c079d10 con 0x7fc134159830 2023-08-25T09:49:51.707 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:51.707 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:51.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc11406a050 msgr2=0x7fc11406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc11406a050 0x7fc11406c510 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc124006fd0 tx=0x7fc124008040 comp rx=0 tx=0).stop 2023-08-25T09:49:51.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 msgr2=0x7fc1341f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc1341f2100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009e50 tx=0x7fc12c004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 shutdown_connections 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc13406bed0 0x7fc1341f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc11406a050 0x7fc11406c510 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc134159830 0x7fc1341f2100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 --2- 172.21.15.71:0/128884168 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc13406a530 0x7fc13414e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.707+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 >> 172.21.15.71:0/128884168 conn(0x7fc1340654f0 msgr2=0x7fc13415a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:51.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.708+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 shutdown_connections 2023-08-25T09:49:51.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.708+0000 7fc13910f640 1 -- 172.21.15.71:0/128884168 wait complete. 2023-08-25T09:49:51.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:49:51.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:49:51.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.951+0000 7fab74a5d640 1 Processor -- start 2023-08-25T09:49:51.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.952+0000 7fab74a5d640 1 -- start start 2023-08-25T09:49:51.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.953+0000 7fab74a5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.953+0000 7fab74a5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fab700ad2f0 0x7fab70155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.953+0000 7fab74a5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700c1060 con 0x7fab700c76d0 2023-08-25T09:49:51.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.953+0000 7fab74a5d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700adf70 con 0x7fab700ad2f0 2023-08-25T09:49:51.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.953+0000 7fab74a5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700af730 con 0x7fab700c72f0 2023-08-25T09:49:51.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:59166/0 (socket says 172.21.15.71:59166) 2023-08-25T09:49:51.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 -- 172.21.15.71:0/4226048820 learned_addr learned my addr 172.21.15.71:0/4226048820 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:51.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6ffff640 1 --2- 172.21.15.71:0/4226048820 >> v2:172.21.15.164:3300/0 conn(0x7fab700ad2f0 0x7fab70155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 -- 172.21.15.71:0/4226048820 >> v2:172.21.15.164:3300/0 conn(0x7fab700ad2f0 msgr2=0x7fab70155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:51.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 --2- 172.21.15.71:0/4226048820 >> v2:172.21.15.164:3300/0 conn(0x7fab700ad2f0 0x7fab70155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 -- 172.21.15.71:0/4226048820 >> v1:172.21.15.164:6789/0 conn(0x7fab700c76d0 legacy=0x7fab70152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.954+0000 7fab6f7fe640 1 -- 172.21.15.71:0/4226048820 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7006b4a0 con 0x7fab700c72f0 2023-08-25T09:49:51.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.956+0000 7fab6f7fe640 1 --2- 172.21.15.71:0/4226048820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700acdb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fab60009870 tx=0x7fab6002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=229e499101671db9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:51.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.956+0000 7fab6e7fc640 1 -- 172.21.15.71:0/4226048820 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6002f840 con 0x7fab700c72f0 2023-08-25T09:49:51.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.956+0000 7fab6e7fc640 1 -- 172.21.15.71:0/4226048820 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6002f9a0 con 0x7fab700c72f0 2023-08-25T09:49:51.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.956+0000 7fab6e7fc640 1 -- 172.21.15.71:0/4226048820 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6002fd30 con 0x7fab700c72f0 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 -- 172.21.15.71:0/4226048820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 msgr2=0x7fab700acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4226048820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700acdb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fab60009870 tx=0x7fab6002ba20 comp rx=0 tx=0).stop 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 -- 172.21.15.71:0/4226048820 shutdown_connections 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4226048820 >> v2:172.21.15.164:3300/0 conn(0x7fab700ad2f0 0x7fab70155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4226048820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700acdb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 -- 172.21.15.71:0/4226048820 >> 172.21.15.71:0/4226048820 conn(0x7fab7006c280 msgr2=0x7fab7006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:51.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 -- 172.21.15.71:0/4226048820 shutdown_connections 2023-08-25T09:49:51.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.957+0000 7fab74a5d640 1 -- 172.21.15.71:0/4226048820 wait complete. 2023-08-25T09:49:51.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.959+0000 7fab74a5d640 1 Processor -- start 2023-08-25T09:49:51.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.959+0000 7fab74a5d640 1 -- start start 2023-08-25T09:49:51.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab74a5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab700ad2f0 0x7fab700ab470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab74a5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab74a5d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 0x7fab700c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab74a5d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700aa080 con 0x7fab700c76d0 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab74a5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c8af0 con 0x7fab700ad2f0 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab74a5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700a9630 con 0x7fab700c72f0 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 0x7fab700c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 0x7fab700c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43500/0 (socket says 172.21.15.71:43500) 2023-08-25T09:49:51.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 -- 172.21.15.71:0/4206496832 learned_addr learned my addr 172.21.15.71:0/4206496832 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:51.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6f7fe640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab700ad2f0 0x7fab700ab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6effd640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 -- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 msgr2=0x7fab700c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 -- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab700ad2f0 msgr2=0x7fab700ab470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab700ad2f0 0x7fab700ab470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.960+0000 7fab6ffff640 1 -- 172.21.15.71:0/4206496832 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600094d0 con 0x7fab700c76d0 2023-08-25T09:49:51.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.961+0000 7fab6ffff640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 0x7fab700c9ad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab6800efc0 tx=0x7fab6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:51.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.961+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68009ca0 con 0x7fab700c76d0 2023-08-25T09:49:51.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.961+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6800f040 con 0x7fab700c76d0 2023-08-25T09:49:51.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.961+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab680146c0 con 0x7fab700c76d0 2023-08-25T09:49:51.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.961+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab700a9910 con 0x7fab700c76d0 2023-08-25T09:49:51.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.961+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab700c9070 con 0x7fab700c76d0 2023-08-25T09:49:51.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.963+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700c9290 con 0x7fab700c76d0 2023-08-25T09:49:51.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.963+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fab68014820 con 0x7fab700c76d0 2023-08-25T09:49:51.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.964+0000 7fab57fff640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:51.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.964+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fab680896d0 con 0x7fab700c76d0 2023-08-25T09:49:51.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.965+0000 7fab6f7fe640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:51.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.965+0000 7fab6f7fe640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab60036000 tx=0x7fab60008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:51.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:51.969+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fab68054b60 con 0x7fab700c76d0 2023-08-25T09:49:52.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.157+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab7004e420 con 0x7fab700c76d0 2023-08-25T09:49:52.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.158+0000 7fab57fff640 1 -- 172.21.15.71:0/4206496832 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab68058180 con 0x7fab700c76d0 2023-08-25T09:49:52.160 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:49:52.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 msgr2=0x7fab5c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab60036000 tx=0x7fab60008030 comp rx=0 tx=0).stop 2023-08-25T09:49:52.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 msgr2=0x7fab700c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 0x7fab700c9ad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab6800efc0 tx=0x7fab6800c490 comp rx=0 tx=0).stop 2023-08-25T09:49:52.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 shutdown_connections 2023-08-25T09:49:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab700c76d0 0x7fab700c9ad0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab700c72f0 0x7fab700c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.160+0000 7fab74a5d640 1 --2- 172.21.15.71:0/4206496832 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab700ad2f0 0x7fab700ab470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.161+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 >> 172.21.15.71:0/4206496832 conn(0x7fab7006c280 msgr2=0x7fab700c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.161+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 shutdown_connections 2023-08-25T09:49:52.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.161+0000 7fab74a5d640 1 -- 172.21.15.71:0/4206496832 wait complete. 2023-08-25T09:49:52.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:49:52.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.397+0000 7fd9bd871640 1 Processor -- start 2023-08-25T09:49:52.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.397+0000 7fd9bd871640 1 -- start start 2023-08-25T09:49:52.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.399+0000 7fd9bd871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.399+0000 7fd9bd871640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd9b81509d0 0x7fd9b8150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.399+0000 7fd9bd871640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b80c1dc0 con 0x7fd9b80c3660 2023-08-25T09:49:52.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.399+0000 7fd9bd871640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b8157a80 con 0x7fd9b81509d0 2023-08-25T09:49:52.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.399+0000 7fd9bd871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b81513a0 con 0x7fd9b80c3a40 2023-08-25T09:49:52.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.399+0000 7fd9b6ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd9b80c3660 0x7fd9b806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53806/0 (socket says 172.21.15.71:53806) 2023-08-25T09:49:52.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b6ffd640 1 -- 172.21.15.71:0/3125597317 learned_addr learned my addr 172.21.15.71:0/3125597317 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:52.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b77fe640 1 --2- 172.21.15.71:0/3125597317 >> v2:172.21.15.71:3300/0 conn(0x7fd9b81509d0 0x7fd9b8150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9affff640 1 --2- 172.21.15.71:0/3125597317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b67fc640 1 -- 172.21.15.71:0/3125597317 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013677322 0 0) 0x7fd9b80c1dc0 con 0x7fd9b80c3660 2023-08-25T09:49:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b67fc640 1 -- 172.21.15.71:0/3125597317 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a4003520 con 0x7fd9b80c3660 2023-08-25T09:49:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b77fe640 1 -- 172.21.15.71:0/3125597317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 msgr2=0x7fd9b80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b77fe640 1 --2- 172.21.15.71:0/3125597317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b77fe640 1 -- 172.21.15.71:0/3125597317 >> v1:172.21.15.71:6789/0 conn(0x7fd9b80c3660 legacy=0x7fd9b806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.400+0000 7fd9b77fe640 1 -- 172.21.15.71:0/3125597317 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b8156e00 con 0x7fd9b81509d0 2023-08-25T09:49:52.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.401+0000 7fd9b77fe640 1 --2- 172.21.15.71:0/3125597317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8150e60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a477504f98989b5f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:52.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9b67fc640 1 -- 172.21.15.71:0/3125597317 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b002cce0 con 0x7fd9b81509d0 2023-08-25T09:49:52.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9b67fc640 1 -- 172.21.15.71:0/3125597317 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b002ce40 con 0x7fd9b81509d0 2023-08-25T09:49:52.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9b67fc640 1 -- 172.21.15.71:0/3125597317 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0034660 con 0x7fd9b81509d0 2023-08-25T09:49:52.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 -- 172.21.15.71:0/3125597317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 msgr2=0x7fd9b8150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 --2- 172.21.15.71:0/3125597317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8150e60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b002bf30 comp rx=0 tx=0).stop 2023-08-25T09:49:52.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 -- 172.21.15.71:0/3125597317 shutdown_connections 2023-08-25T09:49:52.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 --2- 172.21.15.71:0/3125597317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8150e60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 --2- 172.21.15.71:0/3125597317 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 -- 172.21.15.71:0/3125597317 >> 172.21.15.71:0/3125597317 conn(0x7fd9b80654f0 msgr2=0x7fd9b8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:52.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 -- 172.21.15.71:0/3125597317 shutdown_connections 2023-08-25T09:49:52.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.402+0000 7fd9bd871640 1 -- 172.21.15.71:0/3125597317 wait complete. 2023-08-25T09:49:52.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.404+0000 7fd9bd871640 1 Processor -- start 2023-08-25T09:49:52.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.404+0000 7fd9bd871640 1 -- start start 2023-08-25T09:49:52.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.405+0000 7fd9bd871640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9b80c3660 0x7fd9b81ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.405+0000 7fd9bd871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b81531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.405+0000 7fd9bd871640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.405+0000 7fd9bd871640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b8152120 con 0x7fd9b81509d0 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.405+0000 7fd9bd871640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b8152d30 con 0x7fd9b80c3660 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.405+0000 7fd9bd871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b81f0000 con 0x7fd9b80c3a40 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43512/0 (socket says 172.21.15.71:43512) 2023-08-25T09:49:52.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 -- 172.21.15.71:0/1495620449 learned_addr learned my addr 172.21.15.71:0/1495620449 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:52.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b6ffd640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9b80c3660 0x7fd9b81ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9affff640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b81531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 -- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 msgr2=0x7fd9b81531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b81531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 -- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9b80c3660 msgr2=0x7fd9b81ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9b80c3660 0x7fd9b81ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 -- 172.21.15.71:0/1495620449 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b0009a70 con 0x7fd9b81509d0 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9affff640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b81531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.406+0000 7fd9b77fe640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8153700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.407+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b003d600 con 0x7fd9b81509d0 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.407+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b003d780 con 0x7fd9b81509d0 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.407+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b003db30 con 0x7fd9b81509d0 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.407+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b81f02e0 con 0x7fd9b81509d0 2023-08-25T09:49:52.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.407+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b81529c0 con 0x7fd9b81509d0 2023-08-25T09:49:52.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.408+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b80c29c0 con 0x7fd9b81509d0 2023-08-25T09:49:52.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.409+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd9b003b040 con 0x7fd9b81509d0 2023-08-25T09:49:52.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.410+0000 7fd9b4ff9640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd99806a050 0x7fd99806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.410+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd9b00aaea0 con 0x7fd9b81509d0 2023-08-25T09:49:52.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.410+0000 7fd9b6ffd640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd99806a050 0x7fd99806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.411+0000 7fd9b6ffd640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd99806a050 0x7fd99806c510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80011e0 tx=0x7fd9a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:52.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.414+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd9b0076170 con 0x7fd9b81509d0 2023-08-25T09:49:52.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.605+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd9b8152b50 con 0x7fd9b81509d0 2023-08-25T09:49:52.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.606+0000 7fd9b4ff9640 1 -- 172.21.15.71:0/1495620449 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd9b004e020 con 0x7fd9b81509d0 2023-08-25T09:49:52.608 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:49:52.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd99806a050 msgr2=0x7fd99806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd99806a050 0x7fd99806c510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80011e0 tx=0x7fd9a8009040 comp rx=0 tx=0).stop 2023-08-25T09:49:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 msgr2=0x7fd9b8153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8153700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b0004a40 comp rx=0 tx=0).stop 2023-08-25T09:49:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 shutdown_connections 2023-08-25T09:49:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd99806a050 0x7fd99806c510 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd9b81509d0 0x7fd9b8153700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9b80c3a40 0x7fd9b81531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 --2- 172.21.15.71:0/1495620449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9b80c3660 0x7fd9b81ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 >> 172.21.15.71:0/1495620449 conn(0x7fd9b80654f0 msgr2=0x7fd9b80c8860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:52.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 shutdown_connections 2023-08-25T09:49:52.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.609+0000 7fd9bd871640 1 -- 172.21.15.71:0/1495620449 wait complete. 2023-08-25T09:49:52.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:49:52.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.840+0000 7fcd1b59e640 1 Processor -- start 2023-08-25T09:49:52.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.841+0000 7fcd1b59e640 1 -- start start 2023-08-25T09:49:52.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.842+0000 7fcd1b59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcd1c0c76d0 0x7fcd1c152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.842+0000 7fcd1b59e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd1c06aed0 0x7fcd1c1532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.842+0000 7fcd1b59e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0c1060 con 0x7fcd1c0c72f0 2023-08-25T09:49:52.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.842+0000 7fcd1b59e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0adf70 con 0x7fcd1c06aed0 2023-08-25T09:49:52.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.842+0000 7fcd1b59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0af730 con 0x7fcd1c0c76d0 2023-08-25T09:49:52.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd1c06aed0 0x7fcd1c1532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd1c06aed0 0x7fcd1c1532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43528/0 (socket says 172.21.15.71:43528) 2023-08-25T09:49:52.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 -- 172.21.15.71:0/3947868940 learned_addr learned my addr 172.21.15.71:0/3947868940 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:52.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd13fff640 1 --2- 172.21.15.71:0/3947868940 >> v2:172.21.15.164:3300/0 conn(0x7fcd1c0c76d0 0x7fcd1c152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 -- 172.21.15.71:0/3947868940 >> v2:172.21.15.164:3300/0 conn(0x7fcd1c0c76d0 msgr2=0x7fcd1c152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:52.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 --2- 172.21.15.71:0/3947868940 >> v2:172.21.15.164:3300/0 conn(0x7fcd1c0c76d0 0x7fcd1c152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 -- 172.21.15.71:0/3947868940 >> v1:172.21.15.164:6789/0 conn(0x7fcd1c0c72f0 legacy=0x7fcd1c0acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.843+0000 7fcd1ad9d640 1 -- 172.21.15.71:0/3947868940 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c06b4a0 con 0x7fcd1c06aed0 2023-08-25T09:49:52.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.844+0000 7fcd1ad9d640 1 --2- 172.21.15.71:0/3947868940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c1532e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0c7b80 tx=0x7fcd1402e210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc531afcf6d8127f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:52.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd19d9b640 1 -- 172.21.15.71:0/3947868940 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1400b0d0 con 0x7fcd1c06aed0 2023-08-25T09:49:52.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd19d9b640 1 -- 172.21.15.71:0/3947868940 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1400b230 con 0x7fcd1c06aed0 2023-08-25T09:49:52.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd19d9b640 1 -- 172.21.15.71:0/3947868940 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14009600 con 0x7fcd1c06aed0 2023-08-25T09:49:52.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd1b59e640 1 -- 172.21.15.71:0/3947868940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 msgr2=0x7fcd1c1532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/3947868940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c1532e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c0c7b80 tx=0x7fcd1402e210 comp rx=0 tx=0).stop 2023-08-25T09:49:52.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd1b59e640 1 -- 172.21.15.71:0/3947868940 shutdown_connections 2023-08-25T09:49:52.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/3947868940 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c1532e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/3947868940 >> v2:172.21.15.164:3300/0 conn(0x7fcd1c0c76d0 0x7fcd1c152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.845+0000 7fcd1b59e640 1 -- 172.21.15.71:0/3947868940 >> 172.21.15.71:0/3947868940 conn(0x7fcd1c06c280 msgr2=0x7fcd1c06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:52.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.846+0000 7fcd1b59e640 1 -- 172.21.15.71:0/3947868940 shutdown_connections 2023-08-25T09:49:52.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.846+0000 7fcd1b59e640 1 -- 172.21.15.71:0/3947868940 wait complete. 2023-08-25T09:49:52.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.847+0000 7fcd1b59e640 1 Processor -- start 2023-08-25T09:49:52.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.847+0000 7fcd1b59e640 1 -- start start 2023-08-25T09:49:52.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.848+0000 7fcd1b59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c0a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.848+0000 7fcd1b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0c72f0 0x7fcd1c1ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.848+0000 7fcd1b59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0c76d0 0x7fcd1c0aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.848+0000 7fcd1b59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0aa9e0 con 0x7fcd1c06aed0 2023-08-25T09:49:52.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.848+0000 7fcd1b59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c1ede30 con 0x7fcd1c0c76d0 2023-08-25T09:49:52.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.848+0000 7fcd1b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0a9f20 con 0x7fcd1c0c72f0 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c0a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c0a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43534/0 (socket says 172.21.15.71:43534) 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 -- 172.21.15.71:0/2845846274 learned_addr learned my addr 172.21.15.71:0/2845846274 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1ad9d640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0c76d0 0x7fcd1c0aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd13fff640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0c72f0 0x7fcd1c1ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 -- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0c72f0 msgr2=0x7fcd1c1ee610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0c72f0 0x7fcd1c1ee610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 -- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0c76d0 msgr2=0x7fcd1c0aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0c76d0 0x7fcd1c0aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 -- 172.21.15.71:0/2845846274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1400bd40 con 0x7fcd1c06aed0 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.849+0000 7fcd1a59c640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c0a99e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c450 tx=0x7fcd0c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.850+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0052d0 con 0x7fcd1c06aed0 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.850+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0c005430 con 0x7fcd1c06aed0 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.850+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c012870 con 0x7fcd1c06aed0 2023-08-25T09:49:52.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.850+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c1f1870 con 0x7fcd1c06aed0 2023-08-25T09:49:52.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.850+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c1f1e30 con 0x7fcd1c06aed0 2023-08-25T09:49:52.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.851+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c04e420 con 0x7fcd1c06aed0 2023-08-25T09:49:52.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.852+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcd0c0129d0 con 0x7fcd1c06aed0 2023-08-25T09:49:52.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.853+0000 7fcd12ffd640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fccfc06a050 0x7fccfc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:52.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.853+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fcd0c088470 con 0x7fcd1c06aed0 2023-08-25T09:49:52.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.853+0000 7fcd13fff640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fccfc06a050 0x7fccfc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:52.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.854+0000 7fcd13fff640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fccfc06a050 0x7fccfc06c510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd04006fd0 tx=0x7fcd04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:52.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:52.858+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fcd0c0537d0 con 0x7fcd1c06aed0 2023-08-25T09:49:53.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.047+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd1c150560 con 0x7fcd1c06aed0 2023-08-25T09:49:53.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.049+0000 7fcd12ffd640 1 -- 172.21.15.71:0/2845846274 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd0c056df0 con 0x7fcd1c06aed0 2023-08-25T09:49:53.051 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:49:53.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fccfc06a050 msgr2=0x7fccfc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fccfc06a050 0x7fccfc06c510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd04006fd0 tx=0x7fcd04008040 comp rx=0 tx=0).stop 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 msgr2=0x7fcd1c0a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c0a99e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c450 tx=0x7fcd0c00c910 comp rx=0 tx=0).stop 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 shutdown_connections 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd1c0c76d0 0x7fcd1c0aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fccfc06a050 0x7fccfc06c510 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd1c0c72f0 0x7fcd1c1ee610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 --2- 172.21.15.71:0/2845846274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd1c06aed0 0x7fcd1c0a99e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.051+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 >> 172.21.15.71:0/2845846274 conn(0x7fcd1c06c280 msgr2=0x7fcd1c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:53.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.052+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 shutdown_connections 2023-08-25T09:49:53.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.052+0000 7fcd1b59e640 1 -- 172.21.15.71:0/2845846274 wait complete. 2023-08-25T09:49:53.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:49:53.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.287+0000 7f74178e7640 1 Processor -- start 2023-08-25T09:49:53.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.288+0000 7f74178e7640 1 -- start start 2023-08-25T09:49:53.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.289+0000 7f74178e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74100c3660 0x7f74100c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.289+0000 7f74178e7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f74100c3a40 0x7f741006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f74178e7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100c1dc0 con 0x7f74101509d0 2023-08-25T09:49:53.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f74178e7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7410157a80 con 0x7f74100c3a40 2023-08-25T09:49:53.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f74178e7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741006cf70 con 0x7f74100c3660 2023-08-25T09:49:53.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f74100c3a40 0x7f741006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f74100c3a40 0x7f741006ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43536/0 (socket says 172.21.15.71:43536) 2023-08-25T09:49:53.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 -- 172.21.15.71:0/1680797834 learned_addr learned my addr 172.21.15.71:0/1680797834 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:53.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7415683640 1 --2- 172.21.15.71:0/1680797834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74100c3660 0x7f74100c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 -- 172.21.15.71:0/1680797834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74100c3660 msgr2=0x7f74100c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:53.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 --2- 172.21.15.71:0/1680797834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74100c3660 0x7f74100c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 -- 172.21.15.71:0/1680797834 >> v1:172.21.15.164:6789/0 conn(0x7f74101509d0 legacy=0x7f7410150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:53.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.290+0000 7f7414e82640 1 -- 172.21.15.71:0/1680797834 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7410156e00 con 0x7f74100c3a40 2023-08-25T09:49:53.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.292+0000 7f7414e82640 1 --2- 172.21.15.71:0/1680797834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3a40 0x7f741006ca30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7400009870 tx=0x7f740002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3140d3f569349b49 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:53.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.292+0000 7f7407fff640 1 -- 172.21.15.71:0/1680797834 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400030840 con 0x7f74100c3a40 2023-08-25T09:49:53.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.292+0000 7f7407fff640 1 -- 172.21.15.71:0/1680797834 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000309a0 con 0x7f74100c3a40 2023-08-25T09:49:53.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.292+0000 7f7407fff640 1 -- 172.21.15.71:0/1680797834 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400030d30 con 0x7f74100c3a40 2023-08-25T09:49:53.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 -- 172.21.15.71:0/1680797834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3a40 msgr2=0x7f741006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 --2- 172.21.15.71:0/1680797834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3a40 0x7f741006ca30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7400009870 tx=0x7f740002b980 comp rx=0 tx=0).stop 2023-08-25T09:49:53.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 -- 172.21.15.71:0/1680797834 shutdown_connections 2023-08-25T09:49:53.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 --2- 172.21.15.71:0/1680797834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3a40 0x7f741006ca30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 --2- 172.21.15.71:0/1680797834 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74100c3660 0x7f74100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 -- 172.21.15.71:0/1680797834 >> 172.21.15.71:0/1680797834 conn(0x7f74100654f0 msgr2=0x7f7410065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:53.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 -- 172.21.15.71:0/1680797834 shutdown_connections 2023-08-25T09:49:53.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.293+0000 7f74178e7640 1 -- 172.21.15.71:0/1680797834 wait complete. 2023-08-25T09:49:53.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.294+0000 7f74178e7640 1 Processor -- start 2023-08-25T09:49:53.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.295+0000 7f74178e7640 1 -- start start 2023-08-25T09:49:53.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.295+0000 7f74178e7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 0x7f7410154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.295+0000 7f74178e7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74100c3a40 0x7f74101eefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.295+0000 7f74178e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74101509d0 0x7f7410069850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.295+0000 7f74178e7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74101f2aa0 con 0x7f74100c3660 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.295+0000 7f74178e7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74101ef500 con 0x7f74100c3a40 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f74178e7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74101f1c40 con 0x7f74101509d0 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 0x7f7410154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 0x7f7410154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43538/0 (socket says 172.21.15.71:43538) 2023-08-25T09:49:53.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 -- 172.21.15.71:0/1272653420 learned_addr learned my addr 172.21.15.71:0/1272653420 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:53.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 -- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74101509d0 msgr2=0x7f7410069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74101509d0 0x7f7410069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7414e82640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74100c3a40 0x7f74101eefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 -- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74100c3a40 msgr2=0x7f74101eefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74100c3a40 0x7f74101eefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 -- 172.21.15.71:0/1272653420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74000094d0 con 0x7f74100c3660 2023-08-25T09:49:53.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7414e82640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74100c3a40 0x7f74101eefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.296+0000 7f7415683640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 0x7f7410154030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d5f0 tx=0x7f740c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.297+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c0125f0 con 0x7f74100c3660 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.297+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740c012750 con 0x7f74100c3660 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.297+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c012b00 con 0x7f74100c3660 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.297+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74101f1f20 con 0x7f74100c3660 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.297+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74101f24e0 con 0x7f74100c3660 2023-08-25T09:49:53.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.298+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74100c29c0 con 0x7f74100c3660 2023-08-25T09:49:53.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.299+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f740c012c60 con 0x7f74100c3660 2023-08-25T09:49:53.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.300+0000 7f7405ffb640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73f406a050 0x7f73f406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.300+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f740c088de0 con 0x7f74100c3660 2023-08-25T09:49:53.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.300+0000 7f7414e82640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73f406a050 0x7f73f406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.301+0000 7f7414e82640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73f406a050 0x7f73f406c510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7400036000 tx=0x7f740003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:53.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.304+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f740c054140 con 0x7f74100c3660 2023-08-25T09:49:53.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.492+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7410069160 con 0x7f74100c3660 2023-08-25T09:49:53.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.493+0000 7f7405ffb640 1 -- 172.21.15.71:0/1272653420 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f740c057760 con 0x7f74100c3660 2023-08-25T09:49:53.495 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:49:53.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.495+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73f406a050 msgr2=0x7f73f406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.495+0000 7f74178e7640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73f406a050 0x7f73f406c510 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7400036000 tx=0x7f740003b040 comp rx=0 tx=0).stop 2023-08-25T09:49:53.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.495+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 msgr2=0x7f7410154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.495+0000 7f74178e7640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 0x7f7410154030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f740c00d5f0 tx=0x7f740c00dab0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 shutdown_connections 2023-08-25T09:49:53.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f74100c3a40 0x7f74101eefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73f406a050 0x7f73f406c510 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74101509d0 0x7f7410069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 --2- 172.21.15.71:0/1272653420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74100c3660 0x7f7410154030 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 >> 172.21.15.71:0/1272653420 conn(0x7f74100654f0 msgr2=0x7f74100c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:53.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 shutdown_connections 2023-08-25T09:49:53.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.496+0000 7f74178e7640 1 -- 172.21.15.71:0/1272653420 wait complete. 2023-08-25T09:49:53.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:49:53.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.731+0000 7efde9b42640 1 Processor -- start 2023-08-25T09:49:53.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.731+0000 7efde9b42640 1 -- start start 2023-08-25T09:49:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde9b42640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efde40c3a40 0x7efde40c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde9b42640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efde41509d0 0x7efde4150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde9b42640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde40c1dc0 con 0x7efde40c3660 2023-08-25T09:49:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde9b42640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde4157a80 con 0x7efde40c3a40 2023-08-25T09:49:53.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde9b42640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde4151320 con 0x7efde41509d0 2023-08-25T09:49:53.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde37fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efde40c3660 0x7efde406ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53814/0 (socket says 172.21.15.71:53814) 2023-08-25T09:49:53.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde37fe640 1 -- 172.21.15.71:0/2300442575 learned_addr learned my addr 172.21.15.71:0/2300442575 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:53.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.733+0000 7efde2ffd640 1 --2- 172.21.15.71:0/2300442575 >> v2:172.21.15.71:3300/0 conn(0x7efde40c3a40 0x7efde40c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde3fff640 1 --2- 172.21.15.71:0/2300442575 >> v2:172.21.15.164:3300/0 conn(0x7efde41509d0 0x7efde4150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde2ffd640 1 -- 172.21.15.71:0/2300442575 >> v2:172.21.15.164:3300/0 conn(0x7efde41509d0 msgr2=0x7efde4150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:53.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde2ffd640 1 --2- 172.21.15.71:0/2300442575 >> v2:172.21.15.164:3300/0 conn(0x7efde41509d0 0x7efde4150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde2ffd640 1 -- 172.21.15.71:0/2300442575 >> v1:172.21.15.71:6789/0 conn(0x7efde40c3660 legacy=0x7efde406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde27fc640 1 -- 172.21.15.71:0/2300442575 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178775841 0 0) 0x7efde40c1dc0 con 0x7efde40c3660 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde2ffd640 1 -- 172.21.15.71:0/2300442575 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4156e00 con 0x7efde40c3a40 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.734+0000 7efde3fff640 1 --2- 172.21.15.71:0/2300442575 >> v2:172.21.15.164:3300/0 conn(0x7efde41509d0 0x7efde4150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.735+0000 7efde2ffd640 1 --2- 172.21.15.71:0/2300442575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3a40 0x7efde40c1710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efde40663f0 tx=0x7efdd802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4d85105781a2af server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.735+0000 7efde27fc640 1 -- 172.21.15.71:0/2300442575 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8030610 con 0x7efde40c3a40 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.735+0000 7efde27fc640 1 -- 172.21.15.71:0/2300442575 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd8030770 con 0x7efde40c3a40 2023-08-25T09:49:53.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.735+0000 7efde27fc640 1 -- 172.21.15.71:0/2300442575 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8030b00 con 0x7efde40c3a40 2023-08-25T09:49:53.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 -- 172.21.15.71:0/2300442575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3a40 msgr2=0x7efde40c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 --2- 172.21.15.71:0/2300442575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3a40 0x7efde40c1710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efde40663f0 tx=0x7efdd802b960 comp rx=0 tx=0).stop 2023-08-25T09:49:53.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 -- 172.21.15.71:0/2300442575 shutdown_connections 2023-08-25T09:49:53.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 --2- 172.21.15.71:0/2300442575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3a40 0x7efde40c1710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 --2- 172.21.15.71:0/2300442575 >> v2:172.21.15.164:3300/0 conn(0x7efde41509d0 0x7efde4150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 -- 172.21.15.71:0/2300442575 >> 172.21.15.71:0/2300442575 conn(0x7efde40654f0 msgr2=0x7efde4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:53.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 -- 172.21.15.71:0/2300442575 shutdown_connections 2023-08-25T09:49:53.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.736+0000 7efde9b42640 1 -- 172.21.15.71:0/2300442575 wait complete. 2023-08-25T09:49:53.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.738+0000 7efde9b42640 1 Processor -- start 2023-08-25T09:49:53.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.738+0000 7efde9b42640 1 -- start start 2023-08-25T09:49:53.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde9b42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 0x7efde41effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde9b42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efde40c3a40 0x7efde415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde9b42640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efde41509d0 0x7efde41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde9b42640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4151a20 con 0x7efde40c3660 2023-08-25T09:49:53.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde9b42640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde41f0500 con 0x7efde41509d0 2023-08-25T09:49:53.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde9b42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde4159e50 con 0x7efde40c3a40 2023-08-25T09:49:53.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 0x7efde41effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 0x7efde41effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43560/0 (socket says 172.21.15.71:43560) 2023-08-25T09:49:53.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.739+0000 7efde37fe640 1 -- 172.21.15.71:0/1589067159 learned_addr learned my addr 172.21.15.71:0/1589067159 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:53.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde2ffd640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efde40c3a40 0x7efde415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde37fe640 1 -- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efde40c3a40 msgr2=0x7efde415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde3fff640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efde41509d0 0x7efde41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde37fe640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efde40c3a40 0x7efde415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde37fe640 1 -- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efde41509d0 msgr2=0x7efde41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde37fe640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efde41509d0 0x7efde41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde37fe640 1 -- 172.21.15.71:0/1589067159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009400 con 0x7efde40c3660 2023-08-25T09:49:53.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde3fff640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efde41509d0 0x7efde41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:53.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde37fe640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 0x7efde41effc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efdd400c4d0 tx=0x7efdd400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:53.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd40074e0 con 0x7efde40c3660 2023-08-25T09:49:53.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd4007660 con 0x7efde40c3660 2023-08-25T09:49:53.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4004a20 con 0x7efde40c3660 2023-08-25T09:49:53.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.740+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde4158ee0 con 0x7efde40c3660 2023-08-25T09:49:53.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.741+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde4159520 con 0x7efde40c3660 2023-08-25T09:49:53.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.742+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde40c29c0 con 0x7efde40c3660 2023-08-25T09:49:53.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.743+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efdd4004b80 con 0x7efde40c3660 2023-08-25T09:49:53.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.743+0000 7efdcbfff640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efdd006a050 0x7efdd006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:53.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.744+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7efdd4087980 con 0x7efde40c3660 2023-08-25T09:49:53.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.744+0000 7efde2ffd640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efdd006a050 0x7efdd006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:53.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.744+0000 7efde2ffd640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efdd006a050 0x7efdd006c510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdd8036000 tx=0x7efdd803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:53.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.748+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efdd4052e10 con 0x7efde40c3660 2023-08-25T09:49:53.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.938+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7efde406e100 con 0x7efde40c3660 2023-08-25T09:49:53.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.939+0000 7efdcbfff640 1 -- 172.21.15.71:0/1589067159 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7efdd4018020 con 0x7efde40c3660 2023-08-25T09:49:53.941 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:49:53.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efdd006a050 msgr2=0x7efdd006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efdd006a050 0x7efdd006c510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdd8036000 tx=0x7efdd803b040 comp rx=0 tx=0).stop 2023-08-25T09:49:53.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 msgr2=0x7efde41effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:53.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 0x7efde41effc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efdd400c4d0 tx=0x7efdd400c990 comp rx=0 tx=0).stop 2023-08-25T09:49:53.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 shutdown_connections 2023-08-25T09:49:53.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efde41509d0 0x7efde41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efdd006a050 0x7efdd006c510 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efde40c3a40 0x7efde415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 --2- 172.21.15.71:0/1589067159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efde40c3660 0x7efde41effc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:53.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 >> 172.21.15.71:0/1589067159 conn(0x7efde40654f0 msgr2=0x7efde4070830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:53.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 shutdown_connections 2023-08-25T09:49:53.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:53.942+0000 7efde9b42640 1 -- 172.21.15.71:0/1589067159 wait complete. 2023-08-25T09:49:53.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T09:49:54.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.178+0000 7fa3b28f9640 1 Processor -- start 2023-08-25T09:49:54.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.178+0000 7fa3b28f9640 1 -- start start 2023-08-25T09:49:54.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3b28f9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa3ac150ce0 0x7fa3ac0c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3b28f9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c58c0 0x7fa3ac0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3b28f9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa3ac0c76b0 0x7fa3ac0c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3b28f9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c2ba0 con 0x7fa3ac150ce0 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3b28f9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c1770 con 0x7fa3ac0c76b0 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3b28f9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c1f30 con 0x7fa3ac0c58c0 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3abfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa3ac150ce0 0x7fa3ac0c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3abfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa3ac150ce0 0x7fa3ac0c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43576/0 (socket says 172.21.15.71:43576) 2023-08-25T09:49:54.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3abfff640 1 -- 172.21.15.71:0/3819166478 learned_addr learned my addr 172.21.15.71:0/3819166478 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:54.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3ab7fe640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c58c0 0x7fa3ac0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.180+0000 7fa3abfff640 1 -- 172.21.15.71:0/3819166478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c58c0 msgr2=0x7fa3ac0c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:54.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3abfff640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c58c0 0x7fa3ac0c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3abfff640 1 -- 172.21.15.71:0/3819166478 >> v2:172.21.15.164:3300/0 conn(0x7fa3ac0c76b0 msgr2=0x7fa3ac0c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:54.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3b0e96640 1 --2- 172.21.15.71:0/3819166478 >> v2:172.21.15.164:3300/0 conn(0x7fa3ac0c76b0 0x7fa3ac0c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3abfff640 1 --2- 172.21.15.71:0/3819166478 >> v2:172.21.15.164:3300/0 conn(0x7fa3ac0c76b0 0x7fa3ac0c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3abfff640 1 -- 172.21.15.71:0/3819166478 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0c6400 con 0x7fa3ac150ce0 2023-08-25T09:49:54.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3ab7fe640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c58c0 0x7fa3ac0c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:54.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.181+0000 7fa3b0e96640 1 --2- 172.21.15.71:0/3819166478 >> v2:172.21.15.164:3300/0 conn(0x7fa3ac0c76b0 0x7fa3ac0c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:54.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.182+0000 7fa3abfff640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac0c5380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0c8140 tx=0x7fa39c02b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97229086602f78d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:54.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.182+0000 7fa3aaffd640 1 -- 172.21.15.71:0/3819166478 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c004230 con 0x7fa3ac150ce0 2023-08-25T09:49:54.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.182+0000 7fa3aaffd640 1 -- 172.21.15.71:0/3819166478 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39c004390 con 0x7fa3ac150ce0 2023-08-25T09:49:54.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.182+0000 7fa3aaffd640 1 -- 172.21.15.71:0/3819166478 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39c030830 con 0x7fa3ac150ce0 2023-08-25T09:49:54.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 -- 172.21.15.71:0/3819166478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 msgr2=0x7fa3ac0c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:54.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac0c5380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0c8140 tx=0x7fa39c02b970 comp rx=0 tx=0).stop 2023-08-25T09:49:54.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 -- 172.21.15.71:0/3819166478 shutdown_connections 2023-08-25T09:49:54.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/3819166478 >> v2:172.21.15.164:3300/0 conn(0x7fa3ac0c76b0 0x7fa3ac0c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c58c0 0x7fa3ac0c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/3819166478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac0c5380 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 -- 172.21.15.71:0/3819166478 >> 172.21.15.71:0/3819166478 conn(0x7fa3ac06bff0 msgr2=0x7fa3ac06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:54.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 -- 172.21.15.71:0/3819166478 shutdown_connections 2023-08-25T09:49:54.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.183+0000 7fa3b28f9640 1 -- 172.21.15.71:0/3819166478 wait complete. 2023-08-25T09:49:54.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.185+0000 7fa3b28f9640 1 Processor -- start 2023-08-25T09:49:54.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.185+0000 7fa3b28f9640 1 -- start start 2023-08-25T09:49:54.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b28f9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3ac0c58c0 0x7fa3ac155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b28f9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c76b0 0x7fa3ac155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b28f9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac1eee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b28f9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0c8c50 con 0x7fa3ac150ce0 2023-08-25T09:49:54.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b28f9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac154e30 con 0x7fa3ac0c58c0 2023-08-25T09:49:54.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b28f9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac1571f0 con 0x7fa3ac0c76b0 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b0e96640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac1eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b0e96640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac1eee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43592/0 (socket says 172.21.15.71:43592) 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b0e96640 1 -- 172.21.15.71:0/2224739645 learned_addr learned my addr 172.21.15.71:0/2224739645 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3ab7fe640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c76b0 0x7fa3ac155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b0e96640 1 -- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c76b0 msgr2=0x7fa3ac155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b0e96640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c76b0 0x7fa3ac155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3abfff640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3ac0c58c0 0x7fa3ac155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.186+0000 7fa3b0e96640 1 -- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3ac0c58c0 msgr2=0x7fa3ac155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3b0e96640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3ac0c58c0 0x7fa3ac155250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3b0e96640 1 -- 172.21.15.71:0/2224739645 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c009460 con 0x7fa3ac150ce0 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3b0e96640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac1eee30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400cea0 tx=0x7fa3a400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4003b90 con 0x7fa3ac150ce0 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a4003d10 con 0x7fa3ac150ce0 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4004260 con 0x7fa3ac150ce0 2023-08-25T09:49:54.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac1574d0 con 0x7fa3ac150ce0 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.187+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac14ac40 con 0x7fa3ac150ce0 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.188+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac1548b0 con 0x7fa3ac150ce0 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.189+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa3a40043c0 con 0x7fa3ac150ce0 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.190+0000 7fa3a8ff9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa39806a080 0x7fa39806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.190+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa3a4089210 con 0x7fa3ac150ce0 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.191+0000 7fa3abfff640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa39806a080 0x7fa39806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:54.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.191+0000 7fa3abfff640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa39806a080 0x7fa39806c540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa39c035000 tx=0x7fa39c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:54.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.195+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa3ac1548b0 con 0x7fa3ac150ce0 2023-08-25T09:49:54.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.399+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa3ac154f70 con 0x7fa3ac150ce0 2023-08-25T09:49:54.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.400+0000 7fa3a8ff9640 1 -- 172.21.15.71:0/2224739645 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa3ac154f70 con 0x7fa3ac150ce0 2023-08-25T09:49:54.402 INFO:teuthology.orchestra.run.smithi071.stderr:listed 0 entries 2023-08-25T09:49:54.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa39806a080 msgr2=0x7fa39806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:54.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa39806a080 0x7fa39806c540 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa39c035000 tx=0x7fa39c008030 comp rx=0 tx=0).stop 2023-08-25T09:49:54.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 msgr2=0x7fa3ac1eee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:54.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac1eee30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400cea0 tx=0x7fa3a400cf90 comp rx=0 tx=0).stop 2023-08-25T09:49:54.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 shutdown_connections 2023-08-25T09:49:54.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa39806a080 0x7fa39806c540 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3ac150ce0 0x7fa3ac1eee30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3ac0c76b0 0x7fa3ac155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 --2- 172.21.15.71:0/2224739645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3ac0c58c0 0x7fa3ac155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:54.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 >> 172.21.15.71:0/2224739645 conn(0x7fa3ac06bff0 msgr2=0x7fa3ac14f860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:54.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 shutdown_connections 2023-08-25T09:49:54.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.403+0000 7fa3b28f9640 1 -- 172.21.15.71:0/2224739645 wait complete. 2023-08-25T09:49:54.422 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:49:54.422 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T09:49:54.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:54.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.745+0000 7fa7bf671640 1 Processor -- start 2023-08-25T09:49:55.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.745+0000 7fa7bf671640 1 -- start start 2023-08-25T09:49:55.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bf671640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa7b81531b0 0x7fa7b8069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bf671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bf671640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80c56c0 con 0x7fa7b80c3660 2023-08-25T09:49:55.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bf671640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b815d340 con 0x7fa7b81531b0 2023-08-25T09:49:55.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bf671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80c5910 con 0x7fa7b815c570 2023-08-25T09:49:55.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bd40d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa7b81531b0 0x7fa7b8069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bd40d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa7b81531b0 0x7fa7b8069850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43594/0 (socket says 172.21.15.71:43594) 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.747+0000 7fa7bd40d640 1 -- 172.21.15.71:0/2401756111 learned_addr learned my addr 172.21.15.71:0/2401756111 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.748+0000 7fa7bdc0e640 1 --2- 172.21.15.71:0/2401756111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.748+0000 7fa7bd40d640 1 -- 172.21.15.71:0/2401756111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 msgr2=0x7fa7b815c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.748+0000 7fa7bd40d640 1 --2- 172.21.15.71:0/2401756111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b815c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.748+0000 7fa7bd40d640 1 -- 172.21.15.71:0/2401756111 >> v1:172.21.15.164:6789/0 conn(0x7fa7b80c3660 legacy=0x7fa7b80c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.748+0000 7fa7bd40d640 1 -- 172.21.15.71:0/2401756111 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80c2e60 con 0x7fa7b81531b0 2023-08-25T09:49:55.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.749+0000 7fa7bd40d640 1 --2- 172.21.15.71:0/2401756111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b81531b0 0x7fa7b8069850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009870 tx=0x7fa7b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec06d46138aacb47 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.749+0000 7fa7affff640 1 -- 172.21.15.71:0/2401756111 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4030840 con 0x7fa7b81531b0 2023-08-25T09:49:55.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.749+0000 7fa7affff640 1 -- 172.21.15.71:0/2401756111 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b40309a0 con 0x7fa7b81531b0 2023-08-25T09:49:55.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.749+0000 7fa7affff640 1 -- 172.21.15.71:0/2401756111 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4030d30 con 0x7fa7b81531b0 2023-08-25T09:49:55.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 -- 172.21.15.71:0/2401756111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b81531b0 msgr2=0x7fa7b8069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 --2- 172.21.15.71:0/2401756111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b81531b0 0x7fa7b8069850 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009870 tx=0x7fa7b402b980 comp rx=0 tx=0).stop 2023-08-25T09:49:55.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 -- 172.21.15.71:0/2401756111 shutdown_connections 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 --2- 172.21.15.71:0/2401756111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 --2- 172.21.15.71:0/2401756111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b81531b0 0x7fa7b8069850 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 -- 172.21.15.71:0/2401756111 >> 172.21.15.71:0/2401756111 conn(0x7fa7b80ca740 msgr2=0x7fa7b80cab50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 -- 172.21.15.71:0/2401756111 shutdown_connections 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.750+0000 7fa7bf671640 1 -- 172.21.15.71:0/2401756111 wait complete. 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.752+0000 7fa7bf671640 1 Processor -- start 2023-08-25T09:49:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.752+0000 7fa7bf671640 1 -- start start 2023-08-25T09:49:55.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bf671640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 0x7fa7b814ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bf671640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b81531b0 0x7fa7b8155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bf671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b8155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bf671640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b814dd30 con 0x7fa7b80c3660 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bf671640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81552d0 con 0x7fa7b81531b0 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bf671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b815b4c0 con 0x7fa7b815c570 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 0x7fa7b814ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 0x7fa7b814ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43610/0 (socket says 172.21.15.71:43610) 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 -- 172.21.15.71:0/3237482961 learned_addr learned my addr 172.21.15.71:0/3237482961 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bdc0e640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b8155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bcc0c640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b81531b0 0x7fa7b8155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 -- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 msgr2=0x7fa7b8155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b8155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 -- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b81531b0 msgr2=0x7fa7b8155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b81531b0 0x7fa7b8155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bd40d640 1 -- 172.21.15.71:0/3237482961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b40094d0 con 0x7fa7b80c3660 2023-08-25T09:49:55.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.753+0000 7fa7bcc0c640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b81531b0 0x7fa7b8155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:55.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.754+0000 7fa7bd40d640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 0x7fa7b814ee30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009870 tx=0x7fa7b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.754+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4039c90 con 0x7fa7b80c3660 2023-08-25T09:49:55.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.754+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b4039df0 con 0x7fa7b80c3660 2023-08-25T09:49:55.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.754+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b806ebf0 con 0x7fa7b80c3660 2023-08-25T09:49:55.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.754+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4043620 con 0x7fa7b80c3660 2023-08-25T09:49:55.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.754+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b806f1b0 con 0x7fa7b80c3660 2023-08-25T09:49:55.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.755+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b806f340 con 0x7fa7b80c3660 2023-08-25T09:49:55.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.756+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa7b4043780 con 0x7fa7b80c3660 2023-08-25T09:49:55.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.757+0000 7fa7adffb640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa79c06a080 0x7fa79c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.757+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fa7b403e070 con 0x7fa7b80c3660 2023-08-25T09:49:55.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.757+0000 7fa7bcc0c640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa79c06a080 0x7fa79c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.758+0000 7fa7bcc0c640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa79c06a080 0x7fa79c06c540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7b806a1a0 tx=0x7fa7a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.761+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa7b4076d60 con 0x7fa7b80c3660 2023-08-25T09:49:55.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.955+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7b8069160 con 0x7fa7b80c3660 2023-08-25T09:49:55.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.957+0000 7fa7adffb640 1 -- 172.21.15.71:0/3237482961 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7fa7b4034070 con 0x7fa7b80c3660 2023-08-25T09:49:55.091 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:55.091 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:55.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa79c06a080 msgr2=0x7fa79c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa79c06a080 0x7fa79c06c540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7b806a1a0 tx=0x7fa7a8008040 comp rx=0 tx=0).stop 2023-08-25T09:49:55.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 msgr2=0x7fa7b814ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 0x7fa7b814ee30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009870 tx=0x7fa7b4004730 comp rx=0 tx=0).stop 2023-08-25T09:49:55.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 shutdown_connections 2023-08-25T09:49:55.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b81531b0 0x7fa7b8155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa79c06a080 0x7fa79c06c540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b815c570 0x7fa7b8155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 --2- 172.21.15.71:0/3237482961 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa7b80c3660 0x7fa7b814ee30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 >> 172.21.15.71:0/3237482961 conn(0x7fa7b80ca740 msgr2=0x7fa7b80c9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:55.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 shutdown_connections 2023-08-25T09:49:55.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:54.961+0000 7fa7bf671640 1 -- 172.21.15.71:0/3237482961 wait complete. 2023-08-25T09:49:55.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:49:55.098 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:49:55.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.321+0000 7f11407a4640 1 Processor -- start 2023-08-25T09:49:55.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.321+0000 7f11407a4640 1 -- start start 2023-08-25T09:49:55.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.323+0000 7f11407a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1138150d60 0x7f11380c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.323+0000 7f11407a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 0x7f11380c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.323+0000 7f11407a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11380c7690 con 0x7f113806e400 2023-08-25T09:49:55.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.323+0000 7f11407a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1138151140 con 0x7f1138150d60 2023-08-25T09:49:55.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.323+0000 7f11407a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c7f30 con 0x7f11381572f0 2023-08-25T09:49:55.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 0x7f11380c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 0x7f11380c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:59278/0 (socket says 172.21.15.71:59278) 2023-08-25T09:49:55.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 -- 172.21.15.71:0/2895730401 learned_addr learned my addr 172.21.15.71:0/2895730401 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:55.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113e540640 1 --2- 172.21.15.71:0/2895730401 >> v2:172.21.15.164:3300/0 conn(0x7f1138150d60 0x7f11380c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 -- 172.21.15.71:0/2895730401 >> v2:172.21.15.164:3300/0 conn(0x7f1138150d60 msgr2=0x7f11380c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:55.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 --2- 172.21.15.71:0/2895730401 >> v2:172.21.15.164:3300/0 conn(0x7f1138150d60 0x7f11380c8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 -- 172.21.15.71:0/2895730401 >> v1:172.21.15.164:6789/0 conn(0x7f113806e400 legacy=0x7f1138156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113ed41640 1 -- 172.21.15.71:0/2895730401 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113806f2d0 con 0x7f11381572f0 2023-08-25T09:49:55.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.324+0000 7f113e540640 1 --2- 172.21.15.71:0/2895730401 >> v2:172.21.15.164:3300/0 conn(0x7f1138150d60 0x7f11380c8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:49:55.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.326+0000 7f113ed41640 1 --2- 172.21.15.71:0/2895730401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 0x7f11380c9410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1130009e50 tx=0x7f113002bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abfcb3d8871683b0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.327+0000 7f113d53e640 1 -- 172.21.15.71:0/2895730401 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113002ca60 con 0x7f11381572f0 2023-08-25T09:49:55.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.327+0000 7f113d53e640 1 -- 172.21.15.71:0/2895730401 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113002cbc0 con 0x7f11381572f0 2023-08-25T09:49:55.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.327+0000 7f113d53e640 1 -- 172.21.15.71:0/2895730401 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11300343f0 con 0x7f11381572f0 2023-08-25T09:49:55.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.327+0000 7f11407a4640 1 -- 172.21.15.71:0/2895730401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 msgr2=0x7f11380c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.327+0000 7f11407a4640 1 --2- 172.21.15.71:0/2895730401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 0x7f11380c9410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1130009e50 tx=0x7f113002bec0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.327+0000 7f11407a4640 1 -- 172.21.15.71:0/2895730401 shutdown_connections 2023-08-25T09:49:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.328+0000 7f11407a4640 1 --2- 172.21.15.71:0/2895730401 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11381572f0 0x7f11380c9410 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.328+0000 7f11407a4640 1 --2- 172.21.15.71:0/2895730401 >> v2:172.21.15.164:3300/0 conn(0x7f1138150d60 0x7f11380c8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.328+0000 7f11407a4640 1 -- 172.21.15.71:0/2895730401 >> 172.21.15.71:0/2895730401 conn(0x7f11380654f0 msgr2=0x7f1138065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.328+0000 7f11407a4640 1 -- 172.21.15.71:0/2895730401 shutdown_connections 2023-08-25T09:49:55.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.328+0000 7f11407a4640 1 -- 172.21.15.71:0/2895730401 wait complete. 2023-08-25T09:49:55.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.329+0000 7f11407a4640 1 Processor -- start 2023-08-25T09:49:55.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.330+0000 7f11407a4640 1 -- start start 2023-08-25T09:49:55.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f11407a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113806e400 0x7f113814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f11407a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 0x7f1138155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f11407a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11381556b0 0x7f1138142aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f11407a4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11381ee870 con 0x7f1138150d60 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f11407a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11380c45f0 con 0x7f11381556b0 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f11407a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1138154720 con 0x7f113806e400 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f113dd3f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 0x7f1138155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f113dd3f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 0x7f1138155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43626/0 (socket says 172.21.15.71:43626) 2023-08-25T09:49:55.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f113dd3f640 1 -- 172.21.15.71:0/271328381 learned_addr learned my addr 172.21.15.71:0/271328381 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:55.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f113e540640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113806e400 0x7f113814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.331+0000 7f113ed41640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11381556b0 0x7f1138142aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f113dd3f640 1 -- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113806e400 msgr2=0x7f113814d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f113dd3f640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113806e400 0x7f113814d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f113dd3f640 1 -- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11381556b0 msgr2=0x7f1138142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f113dd3f640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11381556b0 0x7f1138142aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f113dd3f640 1 -- 172.21.15.71:0/271328381 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134009400 con 0x7f1138150d60 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f113dd3f640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 0x7f1138155170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f112800c400 tx=0x7f112800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11280052d0 con 0x7f1138150d60 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1128005430 con 0x7f1138150d60 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1130009b30 con 0x7f1138150d60 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128003830 con 0x7f1138150d60 2023-08-25T09:49:55.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.332+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11381eea80 con 0x7f1138150d60 2023-08-25T09:49:55.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.334+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11380c8530 con 0x7f1138150d60 2023-08-25T09:49:55.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.335+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1128003990 con 0x7f1138150d60 2023-08-25T09:49:55.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.336+0000 7f112effd640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f111c06a080 0x7f111c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.336+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1128087e00 con 0x7f1138150d60 2023-08-25T09:49:55.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.336+0000 7f113e540640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f111c06a080 0x7f111c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.337+0000 7f113e540640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f111c06a080 0x7f111c06c540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1134009c60 tx=0x7f1134008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.340+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1128053160 con 0x7f1138150d60 2023-08-25T09:49:55.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.529+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1138069160 con 0x7f1138150d60 2023-08-25T09:49:55.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.529+0000 7f112effd640 1 -- 172.21.15.71:0/271328381 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1128056780 con 0x7f1138150d60 2023-08-25T09:49:55.531 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:49:55.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f111c06a080 msgr2=0x7f111c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f111c06a080 0x7f111c06c540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1134009c60 tx=0x7f1134008040 comp rx=0 tx=0).stop 2023-08-25T09:49:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 msgr2=0x7f1138155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 0x7f1138155170 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f112800c400 tx=0x7f112800c8c0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 shutdown_connections 2023-08-25T09:49:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11381556b0 0x7f1138142aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f111c06a080 0x7f111c06c540 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1138150d60 0x7f1138155170 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 --2- 172.21.15.71:0/271328381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113806e400 0x7f113814d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 >> 172.21.15.71:0/271328381 conn(0x7f11380654f0 msgr2=0x7f11380c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:55.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 shutdown_connections 2023-08-25T09:49:55.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.533+0000 7f11407a4640 1 -- 172.21.15.71:0/271328381 wait complete. 2023-08-25T09:49:55.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:49:55.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.782+0000 7f1c1af66640 1 Processor -- start 2023-08-25T09:49:55.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.782+0000 7f1c1af66640 1 -- start start 2023-08-25T09:49:55.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c1af66640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1c141542d0 0x7f1c141576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c1af66640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1c14069850 0x7f1c140c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c1af66640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c140c75c0 0x7f1c140c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c1af66640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140c4b60 con 0x7f1c14069850 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c1af66640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c14157bf0 con 0x7f1c141542d0 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c1af66640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1406a130 con 0x7f1c140c75c0 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1c14069850 0x7f1c140c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1c14069850 0x7f1c140c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43642/0 (socket says 172.21.15.71:43642) 2023-08-25T09:49:55.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 -- 172.21.15.71:0/1933261560 learned_addr learned my addr 172.21.15.71:0/1933261560 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:55.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 -- 172.21.15.71:0/1933261560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c140c75c0 msgr2=0x7f1c140c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:55.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 --2- 172.21.15.71:0/1933261560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c140c75c0 0x7f1c140c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 -- 172.21.15.71:0/1933261560 >> v2:172.21.15.164:3300/0 conn(0x7f1c141542d0 msgr2=0x7f1c141576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 --2- 172.21.15.71:0/1933261560 >> v2:172.21.15.164:3300/0 conn(0x7f1c141542d0 0x7f1c141576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.784+0000 7f1c19763640 1 -- 172.21.15.71:0/1933261560 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c141546b0 con 0x7f1c14069850 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.785+0000 7f1c19763640 1 --2- 172.21.15.71:0/1933261560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c14069850 0x7f1c140c7080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c14069ee0 tx=0x7f1c0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47134078e58fdd7b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.786+0000 7f1c18f62640 1 -- 172.21.15.71:0/1933261560 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04031870 con 0x7f1c14069850 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.786+0000 7f1c18f62640 1 -- 172.21.15.71:0/1933261560 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c040319d0 con 0x7f1c14069850 2023-08-25T09:49:55.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.786+0000 7f1c18f62640 1 -- 172.21.15.71:0/1933261560 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04031d60 con 0x7f1c14069850 2023-08-25T09:49:55.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.786+0000 7f1c1af66640 1 -- 172.21.15.71:0/1933261560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c14069850 msgr2=0x7f1c140c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.786+0000 7f1c1af66640 1 --2- 172.21.15.71:0/1933261560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c14069850 0x7f1c140c7080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c14069ee0 tx=0x7f1c0402b910 comp rx=0 tx=0).stop 2023-08-25T09:49:55.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 -- 172.21.15.71:0/1933261560 shutdown_connections 2023-08-25T09:49:55.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 --2- 172.21.15.71:0/1933261560 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c140c75c0 0x7f1c140c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 --2- 172.21.15.71:0/1933261560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c14069850 0x7f1c140c7080 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 --2- 172.21.15.71:0/1933261560 >> v2:172.21.15.164:3300/0 conn(0x7f1c141542d0 0x7f1c141576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 -- 172.21.15.71:0/1933261560 >> 172.21.15.71:0/1933261560 conn(0x7f1c1406c280 msgr2=0x7f1c1406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:55.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 -- 172.21.15.71:0/1933261560 shutdown_connections 2023-08-25T09:49:55.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.787+0000 7f1c1af66640 1 -- 172.21.15.71:0/1933261560 wait complete. 2023-08-25T09:49:55.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.788+0000 7f1c1af66640 1 Processor -- start 2023-08-25T09:49:55.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 -- start start 2023-08-25T09:49:55.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c14069850 0x7f1c141f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c140c75c0 0x7f1c140ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 0x7f1c141566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14158250 con 0x7f1c141542d0 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14156260 con 0x7f1c140c75c0 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.789+0000 7f1c1af66640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140c30a0 con 0x7f1c14069850 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 0x7f1c141566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 0x7f1c141566f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43650/0 (socket says 172.21.15.71:43650) 2023-08-25T09:49:55.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 -- 172.21.15.71:0/3049130257 learned_addr learned my addr 172.21.15.71:0/3049130257 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:55.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c19763640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c140c75c0 0x7f1c140ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c19f64640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c14069850 0x7f1c141f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 -- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c14069850 msgr2=0x7f1c141f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c14069850 0x7f1c141f2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 -- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c140c75c0 msgr2=0x7f1c140ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c140c75c0 0x7f1c140ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c1a765640 1 -- 172.21.15.71:0/3049130257 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c040094d0 con 0x7f1c141542d0 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.790+0000 7f1c19f64640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c14069850 0x7f1c141f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.791+0000 7f1c1a765640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 0x7f1c141566f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00b9a0 tx=0x7f1c0c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.791+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00cba0 con 0x7f1c141542d0 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.791+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c00cd20 con 0x7f1c141542d0 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.791+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c014680 con 0x7f1c141542d0 2023-08-25T09:49:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.791+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c140c3380 con 0x7f1c141542d0 2023-08-25T09:49:55.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.791+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c140c3940 con 0x7f1c141542d0 2023-08-25T09:49:55.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.792+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c14156410 con 0x7f1c141542d0 2023-08-25T09:49:55.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.793+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1c0c01d3b0 con 0x7f1c141542d0 2023-08-25T09:49:55.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.794+0000 7f1c0a7fc640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1bf806a080 0x7f1bf806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:55.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.794+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1c0c0888a0 con 0x7f1c141542d0 2023-08-25T09:49:55.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.794+0000 7f1c19f64640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1bf806a080 0x7f1bf806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:55.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.795+0000 7f1c19f64640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1bf806a080 0x7f1bf806c540 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000a870 tx=0x7f1c10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:55.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.799+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1c14156410 con 0x7f1c141542d0 2023-08-25T09:49:55.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.986+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1c14156410 con 0x7f1c141542d0 2023-08-25T09:49:55.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.987+0000 7f1c0a7fc640 1 -- 172.21.15.71:0/3049130257 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1c0c019020 con 0x7f1c141542d0 2023-08-25T09:49:55.989 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:49:55.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1bf806a080 msgr2=0x7f1bf806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1bf806a080 0x7f1bf806c540 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000a870 tx=0x7f1c10008040 comp rx=0 tx=0).stop 2023-08-25T09:49:55.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 msgr2=0x7f1c141566f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:55.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 0x7f1c141566f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00b9a0 tx=0x7f1c0c00be60 comp rx=0 tx=0).stop 2023-08-25T09:49:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 shutdown_connections 2023-08-25T09:49:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c140c75c0 0x7f1c140ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1bf806a080 0x7f1bf806c540 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c141542d0 0x7f1c141566f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 --2- 172.21.15.71:0/3049130257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c14069850 0x7f1c141f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:55.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.989+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 >> 172.21.15.71:0/3049130257 conn(0x7f1c1406c280 msgr2=0x7f1c140af180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:55.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.990+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 shutdown_connections 2023-08-25T09:49:55.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:55.990+0000 7f1c1af66640 1 -- 172.21.15.71:0/3049130257 wait complete. 2023-08-25T09:49:56.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:49:56.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.226+0000 7f0b1828e640 1 Processor -- start 2023-08-25T09:49:56.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.226+0000 7f0b1828e640 1 -- start start 2023-08-25T09:49:56.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.228+0000 7f0b1828e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0b100c3660 0x7f0b100c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.228+0000 7f0b1828e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b1006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.228+0000 7f0b1828e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b100c1dc0 con 0x7f0b101509d0 2023-08-25T09:49:56.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.228+0000 7f0b1828e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b10157a80 con 0x7f0b100c3660 2023-08-25T09:49:56.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.228+0000 7f0b1828e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1006cf70 con 0x7f0b100c3a40 2023-08-25T09:49:56.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.229+0000 7f0b1602a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0b100c3660 0x7f0b100c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.229+0000 7f0b1682b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0b101509d0 0x7f0b10150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:33878/0 (socket says 172.21.15.71:33878) 2023-08-25T09:49:56.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.229+0000 7f0b1682b640 1 -- 172.21.15.71:0/1085615682 learned_addr learned my addr 172.21.15.71:0/1085615682 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:56.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.229+0000 7f0b15829640 1 --2- 172.21.15.71:0/1085615682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b1006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.230+0000 7f0b15028640 1 -- 172.21.15.71:0/1085615682 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771714556 0 0) 0x7f0b100c1dc0 con 0x7f0b101509d0 2023-08-25T09:49:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.230+0000 7f0b15028640 1 -- 172.21.15.71:0/1085615682 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0afc003520 con 0x7f0b101509d0 2023-08-25T09:49:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.230+0000 7f0b15829640 1 -- 172.21.15.71:0/1085615682 >> v2:172.21.15.164:3300/0 conn(0x7f0b100c3660 msgr2=0x7f0b100c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.230+0000 7f0b15829640 1 --2- 172.21.15.71:0/1085615682 >> v2:172.21.15.164:3300/0 conn(0x7f0b100c3660 0x7f0b100c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.230+0000 7f0b15829640 1 -- 172.21.15.71:0/1085615682 >> v1:172.21.15.164:6789/0 conn(0x7f0b101509d0 legacy=0x7f0b10150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.230+0000 7f0b15829640 1 -- 172.21.15.71:0/1085615682 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b10156e00 con 0x7f0b100c3a40 2023-08-25T09:49:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.231+0000 7f0b15829640 1 --2- 172.21.15.71:0/1085615682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b1006ca30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b000097a0 tx=0x7f0b0002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4764dd6140abe57 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b15028640 1 -- 172.21.15.71:0/1085615682 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00030800 con 0x7f0b100c3a40 2023-08-25T09:49:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b15028640 1 -- 172.21.15.71:0/1085615682 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b00030960 con 0x7f0b100c3a40 2023-08-25T09:49:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b15028640 1 -- 172.21.15.71:0/1085615682 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00030cd0 con 0x7f0b100c3a40 2023-08-25T09:49:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 -- 172.21.15.71:0/1085615682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 msgr2=0x7f0b1006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1085615682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b1006ca30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0b000097a0 tx=0x7f0b0002b940 comp rx=0 tx=0).stop 2023-08-25T09:49:56.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 -- 172.21.15.71:0/1085615682 shutdown_connections 2023-08-25T09:49:56.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1085615682 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b1006ca30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1085615682 >> v2:172.21.15.164:3300/0 conn(0x7f0b100c3660 0x7f0b100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 -- 172.21.15.71:0/1085615682 >> 172.21.15.71:0/1085615682 conn(0x7f0b100654f0 msgr2=0x7f0b10065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:56.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 -- 172.21.15.71:0/1085615682 shutdown_connections 2023-08-25T09:49:56.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.232+0000 7f0b1828e640 1 -- 172.21.15.71:0/1085615682 wait complete. 2023-08-25T09:49:56.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.234+0000 7f0b1828e640 1 Processor -- start 2023-08-25T09:49:56.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.234+0000 7f0b1828e640 1 -- start start 2023-08-25T09:49:56.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1828e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b100c3660 0x7f0b1014c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1828e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b101e0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1828e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 0x7f0b101f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1828e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b101f2830 con 0x7f0b101509d0 2023-08-25T09:49:56.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1828e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1014c610 con 0x7f0b100c3660 2023-08-25T09:49:56.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1828e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b101f2570 con 0x7f0b100c3a40 2023-08-25T09:49:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1682b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 0x7f0b101f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.235+0000 7f0b1682b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 0x7f0b101f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43656/0 (socket says 172.21.15.71:43656) 2023-08-25T09:49:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 -- 172.21.15.71:0/1859575224 learned_addr learned my addr 172.21.15.71:0/1859575224 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1602a640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b100c3660 0x7f0b1014c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b15829640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b101e0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 -- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 msgr2=0x7f0b101e0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b101e0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 -- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b100c3660 msgr2=0x7f0b1014c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b100c3660 0x7f0b1014c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 -- 172.21.15.71:0/1859575224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b00009400 con 0x7f0b101509d0 2023-08-25T09:49:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1602a640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b100c3660 0x7f0b1014c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b1682b640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 0x7f0b101f1af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800cb40 tx=0x7f0b0800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.236+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0800aca0 con 0x7f0b101509d0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.237+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0800ae00 con 0x7f0b101509d0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.237+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b08012620 con 0x7f0b101509d0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.237+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1006aa40 con 0x7f0b101509d0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.237+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1006b000 con 0x7f0b101509d0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.238+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b100c29c0 con 0x7f0b101509d0 2023-08-25T09:49:56.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.239+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0b08007070 con 0x7f0b101509d0 2023-08-25T09:49:56.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.240+0000 7f0b06ffd640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0af406a050 0x7f0af406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.240+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0b08088980 con 0x7f0b101509d0 2023-08-25T09:49:56.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.241+0000 7f0b1602a640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0af406a050 0x7f0af406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.241+0000 7f0b1602a640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0af406a050 0x7f0af406c510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009af0 tx=0x7f0b0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:56.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.245+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0b08053ce0 con 0x7f0b101509d0 2023-08-25T09:49:56.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.433+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0b1006a8a0 con 0x7f0b101509d0 2023-08-25T09:49:56.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.434+0000 7f0b06ffd640 1 -- 172.21.15.71:0/1859575224 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0b1006a8a0 con 0x7f0b101509d0 2023-08-25T09:49:56.436 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:49:56.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.436+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0af406a050 msgr2=0x7f0af406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.436+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0af406a050 0x7f0af406c510 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c009af0 tx=0x7f0b0c008040 comp rx=0 tx=0).stop 2023-08-25T09:49:56.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.436+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 msgr2=0x7f0b101f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.436+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 0x7f0b101f1af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800cb40 tx=0x7f0b0800a060 comp rx=0 tx=0).stop 2023-08-25T09:49:56.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 shutdown_connections 2023-08-25T09:49:56.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0af406a050 0x7f0af406c510 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b101509d0 0x7f0b101f1af0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0b100c3a40 0x7f0b101e0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 --2- 172.21.15.71:0/1859575224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0b100c3660 0x7f0b1014c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 >> 172.21.15.71:0/1859575224 conn(0x7f0b100654f0 msgr2=0x7f0b10156630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:56.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 shutdown_connections 2023-08-25T09:49:56.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.437+0000 7f0b1828e640 1 -- 172.21.15.71:0/1859575224 wait complete. 2023-08-25T09:49:56.455 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:49:56.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.672+0000 7fbf60379640 1 Processor -- start 2023-08-25T09:49:56.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.672+0000 7fbf60379640 1 -- start start 2023-08-25T09:49:56.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf60379640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbf581594d0 0x7fbf580c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf60379640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf5806cc70 0x7fbf5814f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf60379640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5806a960 con 0x7fbf580c3940 2023-08-25T09:49:56.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf60379640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5806ac20 con 0x7fbf581594d0 2023-08-25T09:49:56.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf60379640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5806af00 con 0x7fbf5806cc70 2023-08-25T09:49:56.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf5e916640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbf580c3940 0x7fbf581598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53822/0 (socket says 172.21.15.71:53822) 2023-08-25T09:49:56.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf5e916640 1 -- 172.21.15.71:0/327555252 learned_addr learned my addr 172.21.15.71:0/327555252 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:56.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.673+0000 7fbf5e115640 1 --2- 172.21.15.71:0/327555252 >> v2:172.21.15.71:3300/0 conn(0x7fbf581594d0 0x7fbf580c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5d914640 1 --2- 172.21.15.71:0/327555252 >> v2:172.21.15.164:3300/0 conn(0x7fbf5806cc70 0x7fbf5814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5e115640 1 -- 172.21.15.71:0/327555252 >> v2:172.21.15.164:3300/0 conn(0x7fbf5806cc70 msgr2=0x7fbf5814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:56.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5e115640 1 --2- 172.21.15.71:0/327555252 >> v2:172.21.15.164:3300/0 conn(0x7fbf5806cc70 0x7fbf5814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5e115640 1 -- 172.21.15.71:0/327555252 >> v1:172.21.15.71:6789/0 conn(0x7fbf580c3940 legacy=0x7fbf581598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5e115640 1 -- 172.21.15.71:0/327555252 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580c2e60 con 0x7fbf581594d0 2023-08-25T09:49:56.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5d113640 1 -- 172.21.15.71:0/327555252 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464483902 0 0) 0x7fbf5806a960 con 0x7fbf580c3940 2023-08-25T09:49:56.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.674+0000 7fbf5d914640 1 --2- 172.21.15.71:0/327555252 >> v2:172.21.15.164:3300/0 conn(0x7fbf5806cc70 0x7fbf5814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:49:56.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.675+0000 7fbf5e115640 1 --2- 172.21.15.71:0/327555252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf581594d0 0x7fbf580c4c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbf58156da0 tx=0x7fbf4802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879f0f0818adc6f8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:56.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf5d113640 1 -- 172.21.15.71:0/327555252 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4802cc70 con 0x7fbf581594d0 2023-08-25T09:49:56.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf5d113640 1 -- 172.21.15.71:0/327555252 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4802cdd0 con 0x7fbf581594d0 2023-08-25T09:49:56.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf5d113640 1 -- 172.21.15.71:0/327555252 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf48034660 con 0x7fbf581594d0 2023-08-25T09:49:56.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 -- 172.21.15.71:0/327555252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf581594d0 msgr2=0x7fbf580c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 --2- 172.21.15.71:0/327555252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf581594d0 0x7fbf580c4c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbf58156da0 tx=0x7fbf4802bec0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 -- 172.21.15.71:0/327555252 shutdown_connections 2023-08-25T09:49:56.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 --2- 172.21.15.71:0/327555252 >> v2:172.21.15.164:3300/0 conn(0x7fbf5806cc70 0x7fbf5814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 --2- 172.21.15.71:0/327555252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf581594d0 0x7fbf580c4c90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 -- 172.21.15.71:0/327555252 >> 172.21.15.71:0/327555252 conn(0x7fbf580654f0 msgr2=0x7fbf58065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:56.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 -- 172.21.15.71:0/327555252 shutdown_connections 2023-08-25T09:49:56.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.676+0000 7fbf60379640 1 -- 172.21.15.71:0/327555252 wait complete. 2023-08-25T09:49:56.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.678+0000 7fbf60379640 1 Processor -- start 2023-08-25T09:49:56.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.678+0000 7fbf60379640 1 -- start start 2023-08-25T09:49:56.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf60379640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf5806cc70 0x7fbf581e1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf60379640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 0x7fbf580c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf60379640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf581594d0 0x7fbf580c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf60379640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58158ee0 con 0x7fbf580c3940 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf60379640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf581ee9b0 con 0x7fbf5806cc70 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf60379640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf581eec90 con 0x7fbf581594d0 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf5d914640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 0x7fbf580c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf5d914640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 0x7fbf580c83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43688/0 (socket says 172.21.15.71:43688) 2023-08-25T09:49:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.679+0000 7fbf5d914640 1 -- 172.21.15.71:0/4179236605 learned_addr learned my addr 172.21.15.71:0/4179236605 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5e115640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf5806cc70 0x7fbf581e1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5d914640 1 -- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf581594d0 msgr2=0x7fbf580c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5d914640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf581594d0 0x7fbf580c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5d914640 1 -- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf5806cc70 msgr2=0x7fbf581e1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5d914640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf5806cc70 0x7fbf581e1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5d914640 1 -- 172.21.15.71:0/4179236605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf48009a70 con 0x7fbf580c3940 2023-08-25T09:49:56.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5e115640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf5806cc70 0x7fbf581e1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:56.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf5d914640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 0x7fbf580c83d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00d7d0 tx=0x7fbf4c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:56.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0096a0 con 0x7fbf580c3940 2023-08-25T09:49:56.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c009800 con 0x7fbf580c3940 2023-08-25T09:49:56.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c009b90 con 0x7fbf580c3940 2023-08-25T09:49:56.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.680+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf58146300 con 0x7fbf580c3940 2023-08-25T09:49:56.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.681+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf58158bf0 con 0x7fbf580c3940 2023-08-25T09:49:56.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.682+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf58069160 con 0x7fbf580c3940 2023-08-25T09:49:56.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.683+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbf4c009cf0 con 0x7fbf580c3940 2023-08-25T09:49:56.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.684+0000 7fbf56ffd640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf4406a050 0x7fbf4406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:56.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.684+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fbf4c088cc0 con 0x7fbf580c3940 2023-08-25T09:49:56.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.684+0000 7fbf5e115640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf4406a050 0x7fbf4406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:56.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.685+0000 7fbf5e115640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf4406a050 0x7fbf4406c510 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf48036000 tx=0x7fbf48030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:56.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.689+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbf4c054150 con 0x7fbf580c3940 2023-08-25T09:49:56.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.881+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf581eee60 con 0x7fbf580c3940 2023-08-25T09:49:56.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.882+0000 7fbf56ffd640 1 -- 172.21.15.71:0/4179236605 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf4c057770 con 0x7fbf580c3940 2023-08-25T09:49:56.884 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:49:56.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf4406a050 msgr2=0x7fbf4406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf4406a050 0x7fbf4406c510 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf48036000 tx=0x7fbf48030000 comp rx=0 tx=0).stop 2023-08-25T09:49:56.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 msgr2=0x7fbf580c83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:56.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 0x7fbf580c83d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00d7d0 tx=0x7fbf4c00dc90 comp rx=0 tx=0).stop 2023-08-25T09:49:56.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 shutdown_connections 2023-08-25T09:49:56.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf4406a050 0x7fbf4406c510 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf581594d0 0x7fbf580c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf580c3940 0x7fbf580c83d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 --2- 172.21.15.71:0/4179236605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf5806cc70 0x7fbf581e1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:56.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 >> 172.21.15.71:0/4179236605 conn(0x7fbf580654f0 msgr2=0x7fbf5814dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:56.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 shutdown_connections 2023-08-25T09:49:56.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:56.885+0000 7fbf60379640 1 -- 172.21.15.71:0/4179236605 wait complete. 2023-08-25T09:49:56.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:49:57.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.121+0000 7f82865c4640 1 Processor -- start 2023-08-25T09:49:57.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.121+0000 7f82865c4640 1 -- start start 2023-08-25T09:49:57.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f82865c4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f82800c3660 0x7f82800c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f82865c4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f82800c3a40 0x7f828006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f82865c4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800c1dc0 con 0x7f82801509d0 2023-08-25T09:49:57.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f82865c4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8280157a80 con 0x7f82800c3a40 2023-08-25T09:49:57.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f82865c4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828006cf70 con 0x7f82800c3660 2023-08-25T09:49:57.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f8284b61640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f82801509d0 0x7f8280150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53830/0 (socket says 172.21.15.71:53830) 2023-08-25T09:49:57.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f8284b61640 1 -- 172.21.15.71:0/1542849947 learned_addr learned my addr 172.21.15.71:0/1542849947 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:57.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.123+0000 7f827f7fe640 1 --2- 172.21.15.71:0/1542849947 >> v2:172.21.15.71:3300/0 conn(0x7f82800c3a40 0x7f828006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.124+0000 7f827ffff640 1 --2- 172.21.15.71:0/1542849947 >> v2:172.21.15.164:3300/0 conn(0x7f82800c3660 0x7f82800c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.124+0000 7f827effd640 1 -- 172.21.15.71:0/1542849947 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53598226 0 0) 0x7f82800c1dc0 con 0x7f82801509d0 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.124+0000 7f827f7fe640 1 -- 172.21.15.71:0/1542849947 >> v2:172.21.15.164:3300/0 conn(0x7f82800c3660 msgr2=0x7f82800c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.124+0000 7f827f7fe640 1 --2- 172.21.15.71:0/1542849947 >> v2:172.21.15.164:3300/0 conn(0x7f82800c3660 0x7f82800c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.124+0000 7f827f7fe640 1 -- 172.21.15.71:0/1542849947 >> v1:172.21.15.71:6789/0 conn(0x7f82801509d0 legacy=0x7f8280150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.124+0000 7f827f7fe640 1 -- 172.21.15.71:0/1542849947 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280156e00 con 0x7f82800c3a40 2023-08-25T09:49:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.125+0000 7f827f7fe640 1 --2- 172.21.15.71:0/1542849947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82800c3a40 0x7f828006ca30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8280066440 tx=0x7f827402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32aa611ed4f3a73b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:57.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.125+0000 7f827effd640 1 -- 172.21.15.71:0/1542849947 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274030610 con 0x7f82800c3a40 2023-08-25T09:49:57.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.125+0000 7f827effd640 1 -- 172.21.15.71:0/1542849947 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8274030770 con 0x7f82800c3a40 2023-08-25T09:49:57.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.125+0000 7f827effd640 1 -- 172.21.15.71:0/1542849947 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274030b00 con 0x7f82800c3a40 2023-08-25T09:49:57.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 -- 172.21.15.71:0/1542849947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82800c3a40 msgr2=0x7f828006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 --2- 172.21.15.71:0/1542849947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82800c3a40 0x7f828006ca30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8280066440 tx=0x7f827402b960 comp rx=0 tx=0).stop 2023-08-25T09:49:57.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 -- 172.21.15.71:0/1542849947 shutdown_connections 2023-08-25T09:49:57.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 --2- 172.21.15.71:0/1542849947 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82800c3a40 0x7f828006ca30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 --2- 172.21.15.71:0/1542849947 >> v2:172.21.15.164:3300/0 conn(0x7f82800c3660 0x7f82800c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 -- 172.21.15.71:0/1542849947 >> 172.21.15.71:0/1542849947 conn(0x7f82800654f0 msgr2=0x7f8280065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:57.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 -- 172.21.15.71:0/1542849947 shutdown_connections 2023-08-25T09:49:57.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.126+0000 7f82865c4640 1 -- 172.21.15.71:0/1542849947 wait complete. 2023-08-25T09:49:57.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.128+0000 7f82865c4640 1 Processor -- start 2023-08-25T09:49:57.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.128+0000 7f82865c4640 1 -- start start 2023-08-25T09:49:57.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f82865c4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82800c3660 0x7f8280158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f82865c4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82800c3a40 0x7f8280159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f82865c4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 0x7f828015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f82865c4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280157e20 con 0x7f82801509d0 2023-08-25T09:49:57.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f82865c4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280158c70 con 0x7f82800c3a40 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f82865c4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8280158330 con 0x7f82800c3660 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 0x7f828015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 0x7f828015a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43706/0 (socket says 172.21.15.71:43706) 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 -- 172.21.15.71:0/309800476 learned_addr learned my addr 172.21.15.71:0/309800476 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 -- 172.21.15.71:0/309800476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82800c3660 msgr2=0x7f8280158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82800c3660 0x7f8280158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 -- 172.21.15.71:0/309800476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82800c3a40 msgr2=0x7f8280159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82800c3a40 0x7f8280159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.129+0000 7f8284b61640 1 -- 172.21.15.71:0/309800476 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274009400 con 0x7f82801509d0 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.130+0000 7f8284b61640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 0x7f828015a320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f827800c7e0 tx=0x7f827800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.130+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827800a5e0 con 0x7f82801509d0 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.130+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827800a760 con 0x7f82801509d0 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.130+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827800ab10 con 0x7f82801509d0 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.130+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828006a9d0 con 0x7f82801509d0 2023-08-25T09:49:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.130+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828006ac70 con 0x7f82801509d0 2023-08-25T09:49:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.131+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82800c29c0 con 0x7f82801509d0 2023-08-25T09:49:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.132+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f827800ac70 con 0x7f82801509d0 2023-08-25T09:49:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.133+0000 7f827cff9640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f826c06a080 0x7f826c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.133+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8278087dc0 con 0x7f82801509d0 2023-08-25T09:49:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.133+0000 7f827ffff640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f826c06a080 0x7f826c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.134+0000 7f827ffff640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f826c06a080 0x7f826c06c540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8270006fd0 tx=0x7f8270008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:57.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.137+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8278053120 con 0x7f82801509d0 2023-08-25T09:49:57.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.324+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8280069160 con 0x7f82801509d0 2023-08-25T09:49:57.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.325+0000 7f827cff9640 1 -- 172.21.15.71:0/309800476 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f82780180a0 con 0x7f82801509d0 2023-08-25T09:49:57.327 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:49:57.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f826c06a080 msgr2=0x7f826c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f826c06a080 0x7f826c06c540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8270006fd0 tx=0x7f8270008040 comp rx=0 tx=0).stop 2023-08-25T09:49:57.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 msgr2=0x7f828015a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 0x7f828015a320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f827800c7e0 tx=0x7f827800cca0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 shutdown_connections 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82800c3a40 0x7f8280159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f826c06a080 0x7f826c06c540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82801509d0 0x7f828015a320 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 --2- 172.21.15.71:0/309800476 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82800c3660 0x7f8280158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 >> 172.21.15.71:0/309800476 conn(0x7f82800654f0 msgr2=0x7f8280150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:57.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 shutdown_connections 2023-08-25T09:49:57.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.328+0000 7f82865c4640 1 -- 172.21.15.71:0/309800476 wait complete. 2023-08-25T09:49:57.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T09:49:57.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.566+0000 7ff2b097f640 1 Processor -- start 2023-08-25T09:49:57.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.566+0000 7ff2b097f640 1 -- start start 2023-08-25T09:49:57.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2b097f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff2ac0709f0 0x7ff2ac06f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2b097f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac06b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2b097f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac159e70 con 0x7ff2ac14fe20 2023-08-25T09:49:57.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2b097f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2ac1598d0 con 0x7ff2ac0709f0 2023-08-25T09:49:57.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2b097f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac159b90 con 0x7ff2ac0c6e50 2023-08-25T09:49:57.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2ab7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff2ac14fe20 0x7ff2ac06a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53832/0 (socket says 172.21.15.71:53832) 2023-08-25T09:49:57.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.568+0000 7ff2ab7fe640 1 -- 172.21.15.71:0/2828014341 learned_addr learned my addr 172.21.15.71:0/2828014341 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:57.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2a3fff640 1 --2- 172.21.15.71:0/2828014341 >> v2:172.21.15.71:3300/0 conn(0x7ff2ac0709f0 0x7ff2ac06f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2abfff640 1 --2- 172.21.15.71:0/2828014341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac06b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2a3fff640 1 -- 172.21.15.71:0/2828014341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 msgr2=0x7ff2ac06b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:57.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2a3fff640 1 --2- 172.21.15.71:0/2828014341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac06b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2a3fff640 1 -- 172.21.15.71:0/2828014341 >> v1:172.21.15.71:6789/0 conn(0x7ff2ac14fe20 legacy=0x7ff2ac06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2a3fff640 1 -- 172.21.15.71:0/2828014341 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0c6090 con 0x7ff2ac0709f0 2023-08-25T09:49:57.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.569+0000 7ff2aaffd640 1 -- 172.21.15.71:0/2828014341 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720656786 0 0) 0x7ff29c001110 con 0x7ff2ac14fe20 2023-08-25T09:49:57.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.570+0000 7ff2a3fff640 1 --2- 172.21.15.71:0/2828014341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac06f3f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2940097a0 tx=0x7ff29402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f054bbf854b247fd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:57.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2aaffd640 1 -- 172.21.15.71:0/2828014341 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294030800 con 0x7ff2ac0709f0 2023-08-25T09:49:57.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2aaffd640 1 -- 172.21.15.71:0/2828014341 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff294030960 con 0x7ff2ac0709f0 2023-08-25T09:49:57.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2aaffd640 1 -- 172.21.15.71:0/2828014341 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294030cf0 con 0x7ff2ac0709f0 2023-08-25T09:49:57.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 -- 172.21.15.71:0/2828014341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 msgr2=0x7ff2ac06f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2828014341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac06f3f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2940097a0 tx=0x7ff29402b940 comp rx=0 tx=0).stop 2023-08-25T09:49:57.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 -- 172.21.15.71:0/2828014341 shutdown_connections 2023-08-25T09:49:57.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2828014341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac06b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2828014341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac06f3f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 -- 172.21.15.71:0/2828014341 >> 172.21.15.71:0/2828014341 conn(0x7ff2ac06d000 msgr2=0x7ff2ac06d410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:57.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.571+0000 7ff2b097f640 1 -- 172.21.15.71:0/2828014341 shutdown_connections 2023-08-25T09:49:57.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.572+0000 7ff2b097f640 1 -- 172.21.15.71:0/2828014341 wait complete. 2023-08-25T09:49:57.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.573+0000 7ff2b097f640 1 Processor -- start 2023-08-25T09:49:57.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.573+0000 7ff2b097f640 1 -- start start 2023-08-25T09:49:57.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.574+0000 7ff2b097f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac14e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.574+0000 7ff2b097f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.574+0000 7ff2b097f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2ac14fe20 0x7ff2ac158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.574+0000 7ff2b097f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac1555a0 con 0x7ff2ac0709f0 2023-08-25T09:49:57.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.574+0000 7ff2b097f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac0ca4c0 con 0x7ff2ac14fe20 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.574+0000 7ff2b097f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac1574a0 con 0x7ff2ac0c6e50 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac14e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac14e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43722/0 (socket says 172.21.15.71:43722) 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 -- 172.21.15.71:0/2178068790 learned_addr learned my addr 172.21.15.71:0/2178068790 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2abfff640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2ac14fe20 0x7ff2ac158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 -- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 msgr2=0x7ff2ac153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 -- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2ac14fe20 msgr2=0x7ff2ac158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2ac14fe20 0x7ff2ac158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 -- 172.21.15.71:0/2178068790 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff294009400 con 0x7ff2ac0709f0 2023-08-25T09:49:57.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2abfff640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2ac14fe20 0x7ff2ac158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:57.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.575+0000 7ff2ab7fe640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac14e7e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00c430 tx=0x7ff29c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:57.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.576+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c0074e0 con 0x7ff2ac0709f0 2023-08-25T09:49:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.576+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c007660 con 0x7ff2ac0709f0 2023-08-25T09:49:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.576+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac157780 con 0x7ff2ac0709f0 2023-08-25T09:49:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.576+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c004a20 con 0x7ff2ac0709f0 2023-08-25T09:49:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.576+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac157d40 con 0x7ff2ac0709f0 2023-08-25T09:49:57.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.577+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac04e420 con 0x7ff2ac0709f0 2023-08-25T09:49:57.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.578+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff29c004b80 con 0x7ff2ac0709f0 2023-08-25T09:49:57.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.579+0000 7ff2a8ff9640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff28c06a020 0x7ff28c06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:57.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.579+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff29c0885c0 con 0x7ff2ac0709f0 2023-08-25T09:49:57.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.579+0000 7ff2a3fff640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff28c06a020 0x7ff28c06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:57.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.580+0000 7ff2a3fff640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff28c06a020 0x7ff28c06c4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff294036000 tx=0x7ff294030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:57.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.583+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff29c053920 con 0x7ff2ac0709f0 2023-08-25T09:49:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.789+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff2ac159e70 con 0x7ff2ac0709f0 2023-08-25T09:49:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.790+0000 7ff2a8ff9640 1 -- 172.21.15.71:0/2178068790 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v29) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff2ac159e70 con 0x7ff2ac0709f0 2023-08-25T09:49:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:listed 0 entries 2023-08-25T09:49:57.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff28c06a020 msgr2=0x7ff28c06c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff28c06a020 0x7ff28c06c4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff294036000 tx=0x7ff294030000 comp rx=0 tx=0).stop 2023-08-25T09:49:57.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 msgr2=0x7ff2ac14e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:57.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac14e7e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00c430 tx=0x7ff29c00c8f0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 shutdown_connections 2023-08-25T09:49:57.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2ac14fe20 0x7ff2ac158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff28c06a020 0x7ff28c06c4e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2ac0c6e50 0x7ff2ac153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 --2- 172.21.15.71:0/2178068790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2ac0709f0 0x7ff2ac14e7e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 >> 172.21.15.71:0/2178068790 conn(0x7ff2ac06d000 msgr2=0x7ff2ac0c9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 shutdown_connections 2023-08-25T09:49:57.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:57.792+0000 7ff2b097f640 1 -- 172.21.15.71:0/2178068790 wait complete. 2023-08-25T09:49:57.812 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi071.front.sepia.ceph.com... 2023-08-25T09:49:57.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:57.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-08-25T09:49:57.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi071.stderr:ceph-fuse[94699]: fuse finished with error 0 and tester_r 0 2023-08-25T09:49:57.914 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T09:49:58.009 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T09:49:58.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:49:58.010 DEBUG:teuthology.orchestra.run.smithi071:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-08-25T09:49:58.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:58.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.293+0000 7f3b0ca0d640 1 Processor -- start 2023-08-25T09:49:58.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.293+0000 7f3b0ca0d640 1 -- start start 2023-08-25T09:49:58.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0ca0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b080c3a40 0x7f3b080c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0ca0d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b081509d0 0x7f3b08150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0ca0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b080c1dc0 con 0x7f3b080c3660 2023-08-25T09:49:58.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0ca0d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b08157a80 con 0x7f3b081509d0 2023-08-25T09:49:58.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0ca0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b081513a0 con 0x7f3b080c3a40 2023-08-25T09:49:58.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3b080c3660 0x7f3b0806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53846/0 (socket says 172.21.15.71:53846) 2023-08-25T09:49:58.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b0659c640 1 -- 172.21.15.71:0/504711772 learned_addr learned my addr 172.21.15.71:0/504711772 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:49:58.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.295+0000 7f3b06d9d640 1 --2- 172.21.15.71:0/504711772 >> v2:172.21.15.71:3300/0 conn(0x7f3b081509d0 0x7f3b08150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.296+0000 7f3b06d9d640 1 -- 172.21.15.71:0/504711772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b080c3a40 msgr2=0x7f3b080c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:58.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.296+0000 7f3b06d9d640 1 --2- 172.21.15.71:0/504711772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b080c3a40 0x7f3b080c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.296+0000 7f3b06d9d640 1 -- 172.21.15.71:0/504711772 >> v1:172.21.15.71:6789/0 conn(0x7f3b080c3660 legacy=0x7f3b0806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.296+0000 7f3b06d9d640 1 -- 172.21.15.71:0/504711772 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b08156e00 con 0x7f3b081509d0 2023-08-25T09:49:58.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.296+0000 7f3b05d9b640 1 -- 172.21.15.71:0/504711772 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587426710 0 0) 0x7f3b080c1dc0 con 0x7f3b080c3660 2023-08-25T09:49:58.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.297+0000 7f3b06d9d640 1 --2- 172.21.15.71:0/504711772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b081509d0 0x7f3b08150e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009d90 tx=0x7f3b0002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258b1b7109447f5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:58.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.297+0000 7f3b05d9b640 1 -- 172.21.15.71:0/504711772 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0002cce0 con 0x7f3b081509d0 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.297+0000 7f3b05d9b640 1 -- 172.21.15.71:0/504711772 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0002ce40 con 0x7f3b081509d0 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.297+0000 7f3b05d9b640 1 -- 172.21.15.71:0/504711772 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00034660 con 0x7f3b081509d0 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/504711772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b081509d0 msgr2=0x7f3b08150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/504711772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b081509d0 0x7f3b08150e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009d90 tx=0x7f3b0002bf30 comp rx=0 tx=0).stop 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/504711772 shutdown_connections 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/504711772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b081509d0 0x7f3b08150e60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/504711772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b080c3a40 0x7f3b080c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/504711772 >> 172.21.15.71:0/504711772 conn(0x7f3b080654f0 msgr2=0x7f3b08065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:58.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/504711772 shutdown_connections 2023-08-25T09:49:58.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.298+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/504711772 wait complete. 2023-08-25T09:49:58.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.299+0000 7f3b0ca0d640 1 Processor -- start 2023-08-25T09:49:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.300+0000 7f3b0ca0d640 1 -- start start 2023-08-25T09:49:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.300+0000 7f3b0ca0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b080c3660 0x7f3b0806a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.300+0000 7f3b0ca0d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 0x7f3b081431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3b0ca0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b081509d0 0x7f3b08148290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3b0ca0d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08152540 con 0x7f3b080c3a40 2023-08-25T09:49:58.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3b0ca0d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08146fa0 con 0x7f3b080c3660 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3b0ca0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08147e00 con 0x7f3b081509d0 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 0x7f3b081431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 0x7f3b081431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43732/0 (socket says 172.21.15.71:43732) 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 -- 172.21.15.71:0/734612997 learned_addr learned my addr 172.21.15.71:0/734612997 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3b0659c640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b080c3660 0x7f3b0806a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 -- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b081509d0 msgr2=0x7f3b08148290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b081509d0 0x7f3b08148290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 -- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b080c3660 msgr2=0x7f3b0806a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b080c3660 0x7f3b0806a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3afffff640 1 -- 172.21.15.71:0/734612997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00009a70 con 0x7f3b080c3a40 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.301+0000 7f3b0659c640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b080c3660 0x7f3b0806a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.302+0000 7f3afffff640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 0x7f3b081431f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3af000b410 tx=0x7f3af000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.302+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af0011830 con 0x7f3b080c3a40 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.302+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af00119b0 con 0x7f3b080c3a40 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.302+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0806ade0 con 0x7f3b080c3a40 2023-08-25T09:49:58.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.302+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af0011d60 con 0x7f3b080c3a40 2023-08-25T09:49:58.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.302+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b081471f0 con 0x7f3b080c3a40 2023-08-25T09:49:58.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.303+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b08147fb0 con 0x7f3b080c3a40 2023-08-25T09:49:58.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.304+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3af001a400 con 0x7f3b080c3a40 2023-08-25T09:49:58.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.305+0000 7f3afeffd640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3ae806a050 0x7f3ae806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.305+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f3af0088a70 con 0x7f3b080c3a40 2023-08-25T09:49:58.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.305+0000 7f3b0659c640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3ae806a050 0x7f3ae806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.306+0000 7f3b0659c640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3ae806a050 0x7f3ae806c510 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3af8000f60 tx=0x7f3af8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:58.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.309+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3af0053dd0 con 0x7f3b080c3a40 2023-08-25T09:49:58.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.500+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b08069160 con 0x7f3b080c3a40 2023-08-25T09:49:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.502+0000 7f3afeffd640 1 -- 172.21.15.71:0/734612997 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7f3af00573f0 con 0x7f3b080c3a40 2023-08-25T09:49:58.505 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:58.505 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3ae806a050 msgr2=0x7f3ae806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3ae806a050 0x7f3ae806c510 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3af8000f60 tx=0x7f3af8009040 comp rx=0 tx=0).stop 2023-08-25T09:49:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 msgr2=0x7f3b081431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 0x7f3b081431f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3af000b410 tx=0x7f3af000b8d0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 shutdown_connections 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3ae806a050 0x7f3ae806c510 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b081509d0 0x7f3b08148290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b080c3a40 0x7f3b081431f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 --2- 172.21.15.71:0/734612997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b080c3660 0x7f3b0806a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 >> 172.21.15.71:0/734612997 conn(0x7f3b080654f0 msgr2=0x7f3b080c8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.505+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 shutdown_connections 2023-08-25T09:49:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.506+0000 7f3b0ca0d640 1 -- 172.21.15.71:0/734612997 wait complete. 2023-08-25T09:49:58.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:49:58.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.742+0000 7f4130ccc640 1 Processor -- start 2023-08-25T09:49:58.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.743+0000 7f4130ccc640 1 -- start start 2023-08-25T09:49:58.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.744+0000 7f4130ccc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.744+0000 7f4130ccc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f412c0c3a40 0x7f412c06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.744+0000 7f4130ccc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f412c1509d0 0x7f412c150e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.744+0000 7f4130ccc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c157a80 con 0x7f412c1509d0 2023-08-25T09:49:58.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.744+0000 7f4130ccc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412c0c1dc0 con 0x7f412c0c3a40 2023-08-25T09:49:58.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.744+0000 7f4130ccc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c151380 con 0x7f412c0c3660 2023-08-25T09:49:58.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f412c1509d0 0x7f412c150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f412c1509d0 0x7f412c150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43748/0 (socket says 172.21.15.71:43748) 2023-08-25T09:49:58.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 -- 172.21.15.71:0/2326516558 learned_addr learned my addr 172.21.15.71:0/2326516558 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:58.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412a59c640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f4129d9b640 1 --2- 172.21.15.71:0/2326516558 >> v2:172.21.15.164:3300/0 conn(0x7f412c0c3a40 0x7f412c06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 -- 172.21.15.71:0/2326516558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 msgr2=0x7f412c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:58.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 -- 172.21.15.71:0/2326516558 >> v2:172.21.15.164:3300/0 conn(0x7f412c0c3a40 msgr2=0x7f412c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:58.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 --2- 172.21.15.71:0/2326516558 >> v2:172.21.15.164:3300/0 conn(0x7f412c0c3a40 0x7f412c06ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412ad9d640 1 -- 172.21.15.71:0/2326516558 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412c06cf70 con 0x7f412c1509d0 2023-08-25T09:49:58.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f412a59c640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:58.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.745+0000 7f4129d9b640 1 --2- 172.21.15.71:0/2326516558 >> v2:172.21.15.164:3300/0 conn(0x7f412c0c3a40 0x7f412c06ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:58.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.746+0000 7f412ad9d640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1509d0 0x7f412c150e40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f412c071d70 tx=0x7f412402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa24c5b3fff4f1c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:58.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.746+0000 7f412959a640 1 -- 172.21.15.71:0/2326516558 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124030850 con 0x7f412c1509d0 2023-08-25T09:49:58.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f412959a640 1 -- 172.21.15.71:0/2326516558 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41240309b0 con 0x7f412c1509d0 2023-08-25T09:49:58.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f412959a640 1 -- 172.21.15.71:0/2326516558 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124030d40 con 0x7f412c1509d0 2023-08-25T09:49:58.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 -- 172.21.15.71:0/2326516558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1509d0 msgr2=0x7f412c150e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1509d0 0x7f412c150e40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f412c071d70 tx=0x7f412402b990 comp rx=0 tx=0).stop 2023-08-25T09:49:58.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 -- 172.21.15.71:0/2326516558 shutdown_connections 2023-08-25T09:49:58.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c1509d0 0x7f412c150e40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 --2- 172.21.15.71:0/2326516558 >> v2:172.21.15.164:3300/0 conn(0x7f412c0c3a40 0x7f412c06ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 --2- 172.21.15.71:0/2326516558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 -- 172.21.15.71:0/2326516558 >> 172.21.15.71:0/2326516558 conn(0x7f412c0654f0 msgr2=0x7f412c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 -- 172.21.15.71:0/2326516558 shutdown_connections 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.747+0000 7f4130ccc640 1 -- 172.21.15.71:0/2326516558 wait complete. 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.749+0000 7f4130ccc640 1 Processor -- start 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.749+0000 7f4130ccc640 1 -- start start 2023-08-25T09:49:58.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4130ccc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c1f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4130ccc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 0x7f412c1ecf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4130ccc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412c1509d0 0x7f412c14ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4130ccc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c94c0 con 0x7f412c0c3a40 2023-08-25T09:49:58.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4130ccc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412c0c11d0 con 0x7f412c1509d0 2023-08-25T09:49:58.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4130ccc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412c14fd60 con 0x7f412c0c3660 2023-08-25T09:49:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4129d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 0x7f412c1ecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4129d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 0x7f412c1ecf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43764/0 (socket says 172.21.15.71:43764) 2023-08-25T09:49:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.750+0000 7f4129d9b640 1 -- 172.21.15.71:0/237151893 learned_addr learned my addr 172.21.15.71:0/237151893 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f412a59c640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c1f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4129d9b640 1 -- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 msgr2=0x7f412c1f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4129d9b640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c1f1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f412ad9d640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412c1509d0 0x7f412c14ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4129d9b640 1 -- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412c1509d0 msgr2=0x7f412c14ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4129d9b640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412c1509d0 0x7f412c14ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4129d9b640 1 -- 172.21.15.71:0/237151893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240094d0 con 0x7f412c0c3a40 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4129d9b640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 0x7f412c1ecf50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f411400c7f0 tx=0x7f411400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114007820 con 0x7f412c0c3a40 2023-08-25T09:49:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412c14dab0 con 0x7f412c0c3a40 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4114007980 con 0x7f412c0c3a40 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.751+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114007cf0 con 0x7f412c0c3a40 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.752+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412c14dd50 con 0x7f412c0c3a40 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.753+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412c0c29c0 con 0x7f412c0c3a40 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.754+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f41140113f0 con 0x7f412c0c3a40 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.754+0000 7f4122ffd640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410c06a050 0x7f410c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.755+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4114088b50 con 0x7f412c0c3a40 2023-08-25T09:49:58.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.755+0000 7f412a59c640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410c06a050 0x7f410c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:58.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.755+0000 7f412a59c640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410c06a050 0x7f410c06c510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f411c008ee0 tx=0x7f411c008430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:58.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.759+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4114053ed0 con 0x7f412c0c3a40 2023-08-25T09:49:58.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.952+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f412c069160 con 0x7f412c0c3a40 2023-08-25T09:49:58.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.954+0000 7f4122ffd640 1 -- 172.21.15.71:0/237151893 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7f41140574f0 con 0x7f412c0c3a40 2023-08-25T09:49:58.956 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:49:58.957 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:49:58.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410c06a050 msgr2=0x7f410c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410c06a050 0x7f410c06c510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f411c008ee0 tx=0x7f411c008430 comp rx=0 tx=0).stop 2023-08-25T09:49:58.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 msgr2=0x7f412c1ecf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:58.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 0x7f412c1ecf50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f411400c7f0 tx=0x7f411400ccb0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 shutdown_connections 2023-08-25T09:49:58.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412c1509d0 0x7f412c14ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410c06a050 0x7f410c06c510 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f412c0c3a40 0x7f412c1ecf50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 --2- 172.21.15.71:0/237151893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f412c0c3660 0x7f412c1f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 >> 172.21.15.71:0/237151893 conn(0x7f412c0654f0 msgr2=0x7f412c069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:58.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 shutdown_connections 2023-08-25T09:49:58.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:58.957+0000 7f4130ccc640 1 -- 172.21.15.71:0/237151893 wait complete. 2023-08-25T09:49:58.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-08-25T09:49:59.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.193+0000 7f037bd5d640 1 Processor -- start 2023-08-25T09:49:59.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.194+0000 7f037bd5d640 1 -- start start 2023-08-25T09:49:59.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.195+0000 7f037bd5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03740be9f0 0x7f03740c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.195+0000 7f037bd5d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03740c7a10 0x7f037415cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.195+0000 7f037bd5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037406f650 con 0x7f03740bedd0 2023-08-25T09:49:59.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.195+0000 7f037bd5d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037406a8c0 con 0x7f03740c7a10 2023-08-25T09:49:59.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.195+0000 7f037bd5d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037406a300 con 0x7f03740be9f0 2023-08-25T09:49:59.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f037a2fa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03740c7a10 0x7f037415cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f037a2fa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03740c7a10 0x7f037415cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43766/0 (socket says 172.21.15.71:43766) 2023-08-25T09:49:59.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f037a2fa640 1 -- 172.21.15.71:0/2896255762 learned_addr learned my addr 172.21.15.71:0/2896255762 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:59.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f0379af9640 1 --2- 172.21.15.71:0/2896255762 >> v2:172.21.15.164:3300/0 conn(0x7f03740be9f0 0x7f03740c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f037a2fa640 1 -- 172.21.15.71:0/2896255762 >> v2:172.21.15.164:3300/0 conn(0x7f03740be9f0 msgr2=0x7f03740c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:59.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f037a2fa640 1 --2- 172.21.15.71:0/2896255762 >> v2:172.21.15.164:3300/0 conn(0x7f03740be9f0 0x7f03740c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.196+0000 7f037a2fa640 1 -- 172.21.15.71:0/2896255762 >> v1:172.21.15.164:6789/0 conn(0x7f03740bedd0 legacy=0x7f037415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.197+0000 7f037a2fa640 1 -- 172.21.15.71:0/2896255762 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037406a5e0 con 0x7f03740c7a10 2023-08-25T09:49:59.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.198+0000 7f037a2fa640 1 --2- 172.21.15.71:0/2896255762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740c7a10 0x7f037415cc70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f037406c270 tx=0x7f036c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e94e10e463543c7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:59.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.198+0000 7f0378af7640 1 -- 172.21.15.71:0/2896255762 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c02f850 con 0x7f03740c7a10 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.198+0000 7f0378af7640 1 -- 172.21.15.71:0/2896255762 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036c02f9b0 con 0x7f03740c7a10 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.198+0000 7f0378af7640 1 -- 172.21.15.71:0/2896255762 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c02fd40 con 0x7f03740c7a10 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.198+0000 7f037bd5d640 1 -- 172.21.15.71:0/2896255762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740c7a10 msgr2=0x7f037415cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.198+0000 7f037bd5d640 1 --2- 172.21.15.71:0/2896255762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740c7a10 0x7f037415cc70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f037406c270 tx=0x7f036c02ba30 comp rx=0 tx=0).stop 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.199+0000 7f037bd5d640 1 -- 172.21.15.71:0/2896255762 shutdown_connections 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.199+0000 7f037bd5d640 1 --2- 172.21.15.71:0/2896255762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740c7a10 0x7f037415cc70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.199+0000 7f037bd5d640 1 --2- 172.21.15.71:0/2896255762 >> v2:172.21.15.164:3300/0 conn(0x7f03740be9f0 0x7f03740c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.199+0000 7f037bd5d640 1 -- 172.21.15.71:0/2896255762 >> 172.21.15.71:0/2896255762 conn(0x7f03740c7ed0 msgr2=0x7f03740c82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:59.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.199+0000 7f037bd5d640 1 -- 172.21.15.71:0/2896255762 shutdown_connections 2023-08-25T09:49:59.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.199+0000 7f037bd5d640 1 -- 172.21.15.71:0/2896255762 wait complete. 2023-08-25T09:49:59.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.200+0000 7f037bd5d640 1 Processor -- start 2023-08-25T09:49:59.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.201+0000 7f037bd5d640 1 -- start start 2023-08-25T09:49:59.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.201+0000 7f037bd5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03740be9f0 0x7f03740bf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037bd5d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 0x7f03741eeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037bd5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03740c7a10 0x7f03740c3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037bd5d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0374069930 con 0x7f03740bedd0 2023-08-25T09:49:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037bd5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03741ef000 con 0x7f03740c7a10 2023-08-25T09:49:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037bd5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03740c2750 con 0x7f03740be9f0 2023-08-25T09:49:59.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 0x7f03741eeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 0x7f03741eeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43772/0 (socket says 172.21.15.71:43772) 2023-08-25T09:49:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 -- 172.21.15.71:0/4066416908 learned_addr learned my addr 172.21.15.71:0/4066416908 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f0379af9640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03740be9f0 0x7f03740bf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037a2fa640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03740c7a10 0x7f03740c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 -- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03740be9f0 msgr2=0x7f03740bf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03740be9f0 0x7f03740bf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 -- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03740c7a10 msgr2=0x7f03740c3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03740c7a10 0x7f03740c3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f03792f8640 1 -- 172.21.15.71:0/4066416908 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c0094d0 con 0x7f03740bedd0 2023-08-25T09:49:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.202+0000 7f037a2fa640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03740c7a10 0x7f03740c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.203+0000 7f03792f8640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 0x7f03741eeac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f036800c8a0 tx=0x7f036800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.203+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368003b70 con 0x7f03740bedd0 2023-08-25T09:49:59.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.203+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0368003cf0 con 0x7f03740bedd0 2023-08-25T09:49:59.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.203+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036800aa30 con 0x7f03740bedd0 2023-08-25T09:49:59.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.203+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740c2ff0 con 0x7f03740bedd0 2023-08-25T09:49:59.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.203+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03740c39c0 con 0x7f03740bedd0 2023-08-25T09:49:59.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.204+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037406ef00 con 0x7f03740bedd0 2023-08-25T09:49:59.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.205+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f036800ab90 con 0x7f03740bedd0 2023-08-25T09:49:59.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.206+0000 7f03727fc640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f036006a050 0x7f036006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.206+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f03680887c0 con 0x7f03740bedd0 2023-08-25T09:49:59.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.206+0000 7f0379af9640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f036006a050 0x7f036006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.207+0000 7f0379af9640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f036006a050 0x7f036006c510 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0364009a30 tx=0x7f0364008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:59.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.211+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0368053b20 con 0x7f03740bedd0 2023-08-25T09:49:59.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.439+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f03740c3180 con 0x7f03740bedd0 2023-08-25T09:49:59.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.572+0000 7f03727fc640 1 -- 172.21.15.71:0/4066416908 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0368003e60 con 0x7f03740bedd0 2023-08-25T09:49:59.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f036006a050 msgr2=0x7f036006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f036006a050 0x7f036006c510 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0364009a30 tx=0x7f0364008040 comp rx=0 tx=0).stop 2023-08-25T09:49:59.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 msgr2=0x7f03741eeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 0x7f03741eeac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f036800c8a0 tx=0x7f036800cd60 comp rx=0 tx=0).stop 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 shutdown_connections 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03740c7a10 0x7f03740c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f036006a050 0x7f036006c510 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.575+0000 7f037bd5d640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03740bedd0 0x7f03741eeac0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.576+0000 7f037bd5d640 1 --2- 172.21.15.71:0/4066416908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03740be9f0 0x7f03740bf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.576+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 >> 172.21.15.71:0/4066416908 conn(0x7f03740c7ed0 msgr2=0x7f037406f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:59.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.576+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 shutdown_connections 2023-08-25T09:49:59.579 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.576+0000 7f037bd5d640 1 -- 172.21.15.71:0/4066416908 wait complete. 2023-08-25T09:49:59.598 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2023-08-25T09:49:59.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-08-25T09:49:59.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.813+0000 7f9bfa908640 1 Processor -- start 2023-08-25T09:49:59.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.813+0000 7f9bfa908640 1 -- start start 2023-08-25T09:49:59.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.814+0000 7f9bfa908640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9bf41542d0 0x7f9bf41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bfa908640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9bf4069850 0x7f9bf40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bfa908640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4157bf0 con 0x7f9bf40c75c0 2023-08-25T09:49:59.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bfa908640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf40c4b60 con 0x7f9bf41542d0 2023-08-25T09:49:59.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bfa908640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf406a130 con 0x7f9bf4069850 2023-08-25T09:49:59.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9bf41542d0 0x7f9bf41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9bf41542d0 0x7f9bf41576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43782/0 (socket says 172.21.15.71:43782) 2023-08-25T09:49:59.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 -- 172.21.15.71:0/2619277763 learned_addr learned my addr 172.21.15.71:0/2619277763 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:59.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 -- 172.21.15.71:0/2619277763 >> v2:172.21.15.164:3300/0 conn(0x7f9bf4069850 msgr2=0x7f9bf40c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:59.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 --2- 172.21.15.71:0/2619277763 >> v2:172.21.15.164:3300/0 conn(0x7f9bf4069850 0x7f9bf40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 -- 172.21.15.71:0/2619277763 >> v1:172.21.15.164:6789/0 conn(0x7f9bf40c75c0 legacy=0x7f9bf40c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:49:59.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.815+0000 7f9bf9906640 1 -- 172.21.15.71:0/2619277763 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf41546b0 con 0x7f9bf41542d0 2023-08-25T09:49:59.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.816+0000 7f9bf9906640 1 --2- 172.21.15.71:0/2619277763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41576b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40c13c0 tx=0x7f9bf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f619baff4e249aee server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:59.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bf8904640 1 -- 172.21.15.71:0/2619277763 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0031870 con 0x7f9bf41542d0 2023-08-25T09:49:59.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bf8904640 1 -- 172.21.15.71:0/2619277763 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf00319d0 con 0x7f9bf41542d0 2023-08-25T09:49:59.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bf8904640 1 -- 172.21.15.71:0/2619277763 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0031d60 con 0x7f9bf41542d0 2023-08-25T09:49:59.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bfa908640 1 -- 172.21.15.71:0/2619277763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 msgr2=0x7f9bf41576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2619277763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41576b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40c13c0 tx=0x7f9bf002b910 comp rx=0 tx=0).stop 2023-08-25T09:49:59.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bfa908640 1 -- 172.21.15.71:0/2619277763 shutdown_connections 2023-08-25T09:49:59.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2619277763 >> v2:172.21.15.164:3300/0 conn(0x7f9bf4069850 0x7f9bf40c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2619277763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41576b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.817+0000 7f9bfa908640 1 -- 172.21.15.71:0/2619277763 >> 172.21.15.71:0/2619277763 conn(0x7f9bf406c280 msgr2=0x7f9bf406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:49:59.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.818+0000 7f9bfa908640 1 -- 172.21.15.71:0/2619277763 shutdown_connections 2023-08-25T09:49:59.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.818+0000 7f9bfa908640 1 -- 172.21.15.71:0/2619277763 wait complete. 2023-08-25T09:49:59.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.819+0000 7f9bfa908640 1 Processor -- start 2023-08-25T09:49:59.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.819+0000 7f9bfa908640 1 -- start start 2023-08-25T09:49:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.820+0000 7f9bfa908640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4069850 0x7f9bf4151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.820+0000 7f9bfa908640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40c75c0 0x7f9bf41ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.820+0000 7f9bfa908640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41ed710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.820+0000 7f9bfa908640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4157560 con 0x7f9bf41542d0 2023-08-25T09:49:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.820+0000 7f9bfa908640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf41554b0 con 0x7f9bf40c75c0 2023-08-25T09:49:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.820+0000 7f9bfa908640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4156310 con 0x7f9bf4069850 2023-08-25T09:49:59.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41ed710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43784/0 (socket says 172.21.15.71:43784) 2023-08-25T09:49:59.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 -- 172.21.15.71:0/2019097911 learned_addr learned my addr 172.21.15.71:0/2019097911 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:49:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bf9906640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4069850 0x7f9bf4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bf9105640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40c75c0 0x7f9bf41ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 -- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4069850 msgr2=0x7f9bf4151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4069850 0x7f9bf4151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 -- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40c75c0 msgr2=0x7f9bf41ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:49:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40c75c0 0x7f9bf41ed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:49:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 -- 172.21.15.71:0/2019097911 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00094d0 con 0x7f9bf41542d0 2023-08-25T09:49:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bf9105640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40c75c0 0x7f9bf41ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:49:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.821+0000 7f9bfa107640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41ed710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00c7b0 tx=0x7f9bec00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.822+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec003d80 con 0x7f9bf41542d0 2023-08-25T09:49:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.822+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec005070 con 0x7f9bf41542d0 2023-08-25T09:49:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.822+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf4153ac0 con 0x7f9bf41542d0 2023-08-25T09:49:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.822+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec005420 con 0x7f9bf41542d0 2023-08-25T09:49:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.822+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf41559e0 con 0x7f9bf41542d0 2023-08-25T09:49:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.823+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf404e420 con 0x7f9bf41542d0 2023-08-25T09:49:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.824+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9bec005580 con 0x7f9bf41542d0 2023-08-25T09:49:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.825+0000 7f9bea7fc640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd806a080 0x7f9bd806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:49:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.825+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f9bec0880c0 con 0x7f9bf41542d0 2023-08-25T09:49:59.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.825+0000 7f9bf9906640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd806a080 0x7f9bd806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:49:59.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.826+0000 7f9bf9906640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd806a080 0x7f9bd806c540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0036000 tx=0x7f9bf003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:49:59.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:49:59.829+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9bec0534b0 con 0x7f9bf41542d0 2023-08-25T09:50:00.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.058+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f9bf4069f10 con 0x7f9bf41542d0 2023-08-25T09:50:00.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.515+0000 7f9bea7fc640 1 -- 172.21.15.71:0/2019097911 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9bec017020 con 0x7f9bf41542d0 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd806a080 msgr2=0x7f9bd806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd806a080 0x7f9bd806c540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0036000 tx=0x7f9bf003c040 comp rx=0 tx=0).stop 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 msgr2=0x7f9bf41ed710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41ed710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9bec00c7b0 tx=0x7f9bec00cc70 comp rx=0 tx=0).stop 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 shutdown_connections 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bf40c75c0 0x7f9bf41ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd806a080 0x7f9bd806c540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf41542d0 0x7f9bf41ed710 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 --2- 172.21.15.71:0/2019097911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4069850 0x7f9bf4151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.518+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 >> 172.21.15.71:0/2019097911 conn(0x7f9bf406c280 msgr2=0x7f9bf40c9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:00.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.519+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 shutdown_connections 2023-08-25T09:50:00.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.519+0000 7f9bfa908640 1 -- 172.21.15.71:0/2019097911 wait complete. 2023-08-25T09:50:00.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-08-25T09:50:00.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.772+0000 7f36143e1640 1 Processor -- start 2023-08-25T09:50:00.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.773+0000 7f36143e1640 1 -- start start 2023-08-25T09:50:00.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f36143e1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f360c0c8600 0x7f360c069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:00.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f36143e1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360c0b2390 con 0x7f360c15c4f0 2023-08-25T09:50:00.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f36143e1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360c1536c0 con 0x7f360c0b2820 2023-08-25T09:50:00.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f36143e1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f360c0b4430 con 0x7f360c0c8600 2023-08-25T09:50:00.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f361217d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f360c0c8600 0x7f360c069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f361197c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f360c15c4f0 0x7f360c0b1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46336/0 (socket says 172.21.15.71:46336) 2023-08-25T09:50:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.774+0000 7f361197c640 1 -- 172.21.15.71:0/1313771046 learned_addr learned my addr 172.21.15.71:0/1313771046 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.775+0000 7f361217d640 1 -- 172.21.15.71:0/1313771046 >> v1:172.21.15.164:6789/0 conn(0x7f360c0b2820 legacy=0x7f360c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.775+0000 7f361217d640 1 -- 172.21.15.71:0/1313771046 >> v1:172.21.15.71:6789/0 conn(0x7f360c15c4f0 legacy=0x7f360c0b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.775+0000 7f361217d640 1 -- 172.21.15.71:0/1313771046 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0c89e0 con 0x7f360c0c8600 2023-08-25T09:50:00.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.775+0000 7f361117b640 1 -- 172.21.15.71:0/1313771046 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320935905 0 0) 0x7f360c0b2390 con 0x7f360c15c4f0 2023-08-25T09:50:00.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.776+0000 7f361217d640 1 --2- 172.21.15.71:0/1313771046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0c8600 0x7f360c069850 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009d90 tx=0x7f35fc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1647622277da5dfa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:00.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.776+0000 7f361117b640 1 -- 172.21.15.71:0/1313771046 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc02cd00 con 0x7f360c0c8600 2023-08-25T09:50:00.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.776+0000 7f361117b640 1 -- 172.21.15.71:0/1313771046 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35fc02ce60 con 0x7f360c0c8600 2023-08-25T09:50:00.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.776+0000 7f361117b640 1 -- 172.21.15.71:0/1313771046 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc034660 con 0x7f360c0c8600 2023-08-25T09:50:00.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 -- 172.21.15.71:0/1313771046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0c8600 msgr2=0x7f360c069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 --2- 172.21.15.71:0/1313771046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0c8600 0x7f360c069850 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009d90 tx=0x7f35fc02bf50 comp rx=0 tx=0).stop 2023-08-25T09:50:00.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 -- 172.21.15.71:0/1313771046 shutdown_connections 2023-08-25T09:50:00.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 --2- 172.21.15.71:0/1313771046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0c8600 0x7f360c069850 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 -- 172.21.15.71:0/1313771046 >> 172.21.15.71:0/1313771046 conn(0x7f360c0ce720 msgr2=0x7f360c0ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:00.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 -- 172.21.15.71:0/1313771046 shutdown_connections 2023-08-25T09:50:00.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.777+0000 7f36143e1640 1 -- 172.21.15.71:0/1313771046 wait complete. 2023-08-25T09:50:00.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.779+0000 7f36143e1640 1 Processor -- start 2023-08-25T09:50:00.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.779+0000 7f36143e1640 1 -- start start 2023-08-25T09:50:00.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f36143e1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 0x7f360c148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:00.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f36143e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360c0c8600 0x7f360c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:00.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f36143e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f360c15c4f0 0x7f360c14a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:00.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f36143e1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c14c450 con 0x7f360c0b2820 2023-08-25T09:50:00.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f36143e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1eed30 con 0x7f360c15c4f0 2023-08-25T09:50:00.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f36143e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360c1ede40 con 0x7f360c0c8600 2023-08-25T09:50:00.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f361217d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 0x7f360c148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:00.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f361217d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 0x7f360c148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59862/0 (socket says 172.21.15.71:59862) 2023-08-25T09:50:00.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f361217d640 1 -- 172.21.15.71:0/4278056251 learned_addr learned my addr 172.21.15.71:0/4278056251 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:00.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f361197c640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360c0c8600 0x7f360c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:00.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f361217d640 1 -- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360c0c8600 msgr2=0x7f360c0cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.780+0000 7f361297e640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f360c15c4f0 0x7f360c14a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:00.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361217d640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360c0c8600 0x7f360c0cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361217d640 1 -- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f360c15c4f0 msgr2=0x7f360c14a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:00.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361217d640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f360c15c4f0 0x7f360c14a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:00.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361217d640 1 -- 172.21.15.71:0/4278056251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc009a70 con 0x7f360c0b2820 2023-08-25T09:50:00.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361197c640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360c0c8600 0x7f360c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:00.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361297e640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f360c15c4f0 0x7f360c14a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:00.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f361217d640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 0x7f360c148970 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009d90 tx=0x7f35fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc03d600 con 0x7f360c0b2820 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35fc03d760 con 0x7f360c0b2820 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc03db10 con 0x7f360c0b2820 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360c1ee120 con 0x7f360c0b2820 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.781+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360c1ee6e0 con 0x7f360c0b2820 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.782+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f360c069160 con 0x7f360c0b2820 2023-08-25T09:50:00.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.783+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f35fc03b040 con 0x7f360c0b2820 2023-08-25T09:50:00.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.784+0000 7f360affd640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f35f806a050 0x7f35f806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:00.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.784+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f35fc0aaa40 con 0x7f360c0b2820 2023-08-25T09:50:00.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.784+0000 7f361197c640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f35f806a050 0x7f35f806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:00.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.785+0000 7f361197c640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f35f806a050 0x7f35f806c510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f360c0b0fa0 tx=0x7f3600009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:00.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:00.789+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f35fc075d10 con 0x7f360c0b2820 2023-08-25T09:50:01.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.017+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f360c153100 con 0x7f360c0b2820 2023-08-25T09:50:01.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.587+0000 7f360affd640 1 -- 172.21.15.71:0/4278056251 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f35fc04d0c0 con 0x7f360c0b2820 2023-08-25T09:50:01.592 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f35f806a050 msgr2=0x7f35f806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:01.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f35f806a050 0x7f35f806c510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f360c0b0fa0 tx=0x7f3600009040 comp rx=0 tx=0).stop 2023-08-25T09:50:01.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 msgr2=0x7f360c148970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:01.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 0x7f360c148970 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009d90 tx=0x7f35fc004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:01.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 shutdown_connections 2023-08-25T09:50:01.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f360c15c4f0 0x7f360c14a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f35f806a050 0x7f35f806c510 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360c0c8600 0x7f360c0cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 --2- 172.21.15.71:0/4278056251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f360c0b2820 0x7f360c148970 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 >> 172.21.15.71:0/4278056251 conn(0x7f360c0ce720 msgr2=0x7f360c15c910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:01.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 shutdown_connections 2023-08-25T09:50:01.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.591+0000 7f36143e1640 1 -- 172.21.15.71:0/4278056251 wait complete. 2023-08-25T09:50:01.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-08-25T09:50:01.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.827+0000 7f4570d42640 1 Processor -- start 2023-08-25T09:50:01.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.827+0000 7f4570d42640 1 -- start start 2023-08-25T09:50:01.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.828+0000 7f4570d42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c150ce0 0x7f456c0c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f4570d42640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f456c0c58c0 0x7f456c0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f4570d42640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c0c2ba0 con 0x7f456c0c76b0 2023-08-25T09:50:01.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f4570d42640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456c0c1770 con 0x7f456c0c58c0 2023-08-25T09:50:01.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f4570d42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0c1f30 con 0x7f456c150ce0 2023-08-25T09:50:01.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f456ad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f456c0c76b0 0x7f456c15a060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46340/0 (socket says 172.21.15.71:46340) 2023-08-25T09:50:01.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f456ad9d640 1 -- 172.21.15.71:0/2795848159 learned_addr learned my addr 172.21.15.71:0/2795848159 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:01.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f456a59c640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c150ce0 0x7f456c0c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f4569d9b640 1 --2- 172.21.15.71:0/2795848159 >> v2:172.21.15.164:3300/0 conn(0x7f456c0c58c0 0x7f456c0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722813100 0 0) 0x7f4564001490 con 0x7f456c0c76b0 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.829+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554003520 con 0x7f456c0c76b0 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915795609 0 0) 0x7f4554003520 con 0x7f456c0c76b0 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c150ce0 msgr2=0x7f456c0c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c150ce0 0x7f456c0c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 >> v2:172.21.15.164:3300/0 conn(0x7f456c0c58c0 msgr2=0x7f456c0c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 --2- 172.21.15.71:0/2795848159 >> v2:172.21.15.164:3300/0 conn(0x7f456c0c58c0 0x7f456c0c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c0c6400 con 0x7f456c0c76b0 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4564001b80 con 0x7f456c0c76b0 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 >> v1:172.21.15.71:6789/0 conn(0x7f456c0c76b0 legacy=0x7f456c15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f4554003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540047c0 0x7f4554004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4554005180 0x7f455400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0c1770 con 0x7f4554003970 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c0c1f30 con 0x7f45540047c0 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455400bec0 con 0x7f4554005180 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456a59c640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c150ce0 0x7f456c0c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.830+0000 7f456a59c640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f4554003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f456ad9d640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4554005180 0x7f455400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f4569d9b640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540047c0 0x7f4554004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f456a59c640 1 -- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4554005180 msgr2=0x7f455400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:01.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f456a59c640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4554005180 0x7f455400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f456a59c640 1 -- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540047c0 msgr2=0x7f4554004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:01.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f456a59c640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45540047c0 0x7f4554004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.831+0000 7f456a59c640 1 -- 172.21.15.71:0/2795848159 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554003520 con 0x7f4554003970 2023-08-25T09:50:01.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.832+0000 7f456a59c640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f4554003d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f456c0740c0 tx=0x7f45540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8417d041de760a78 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.832+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c004030 con 0x7f4554003970 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.832+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c004190 con 0x7f4554003970 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.832+0000 7f456959a640 1 -- 172.21.15.71:0/2795848159 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c004520 con 0x7f4554003970 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f4570d42640 1 -- 172.21.15.71:0/2795848159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 msgr2=0x7f4554003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f4570d42640 1 --2- 172.21.15.71:0/2795848159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f4554003d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f456c0740c0 tx=0x7f45540057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f456ad9d640 1 -- 172.21.15.71:0/2795848159 reap_dead start 2023-08-25T09:50:01.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f4570d42640 1 -- 172.21.15.71:0/2795848159 shutdown_connections 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f4570d42640 1 -- 172.21.15.71:0/2795848159 >> 172.21.15.71:0/2795848159 conn(0x7f456c06bff0 msgr2=0x7f456c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f4570d42640 1 -- 172.21.15.71:0/2795848159 shutdown_connections 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.833+0000 7f4570d42640 1 -- 172.21.15.71:0/2795848159 wait complete. 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.835+0000 7f4570d42640 1 Processor -- start 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.835+0000 7f4570d42640 1 -- start start 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4570d42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f456c155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4570d42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c1ebba0 0x7f456c1ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4570d42640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456c1ed450 0x7f456c1ed8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4570d42640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c1eb150 con 0x7f4554003970 2023-08-25T09:50:01.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4570d42640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456c1eb710 con 0x7f456c1ed450 2023-08-25T09:50:01.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4570d42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456c142990 con 0x7f456c1ebba0 2023-08-25T09:50:01.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f456c155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f456c155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59880/0 (socket says 172.21.15.71:59880) 2023-08-25T09:50:01.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 -- 172.21.15.71:0/1159734274 learned_addr learned my addr 172.21.15.71:0/1159734274 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:01.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f4569d9b640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c1ebba0 0x7f456c1ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456ad9d640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456c1ed450 0x7f456c1ed8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 -- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c1ebba0 msgr2=0x7f456c1ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:01.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c1ebba0 0x7f456c1ebfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 -- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456c1ed450 msgr2=0x7f456c1ed8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:01.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456c1ed450 0x7f456c1ed8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:01.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.836+0000 7f456a59c640 1 -- 172.21.15.71:0/1159734274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c1556b0 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.837+0000 7f456a59c640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f456c155170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f455c009be0 tx=0x7f455c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.837+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c02cb20 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.837+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c005e90 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.837+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c1ec520 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.837+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c02ce80 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.837+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c142ef0 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.838+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c069160 con 0x7f4554003970 2023-08-25T09:50:01.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.839+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f455c006040 con 0x7f4554003970 2023-08-25T09:50:01.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.840+0000 7f455affd640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f454c06a080 0x7f454c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:01.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.840+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f455c0aaf30 con 0x7f4554003970 2023-08-25T09:50:01.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.840+0000 7f4569d9b640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f454c06a080 0x7f454c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:01.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.841+0000 7f4569d9b640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f454c06a080 0x7f454c06c540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f456c14c3c0 tx=0x7f4560008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:01.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:01.845+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f455c0763b0 con 0x7f4554003970 2023-08-25T09:50:02.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.072+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f456c0c5ed0 con 0x7f4554003970 2023-08-25T09:50:02.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.592+0000 7f455affd640 1 -- 172.21.15.71:0/1159734274 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v10) v1 ==== 115+0+0 (secure 0 0 0) 0x7f455c04c020 con 0x7f4554003970 2023-08-25T09:50:02.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.595+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f454c06a080 msgr2=0x7f454c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:02.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.595+0000 7f4570d42640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f454c06a080 0x7f454c06c540 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f456c14c3c0 tx=0x7f4560008040 comp rx=0 tx=0).stop 2023-08-25T09:50:02.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 msgr2=0x7f456c155170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:02.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f456c155170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f455c009be0 tx=0x7f455c004730 comp rx=0 tx=0).stop 2023-08-25T09:50:02.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 shutdown_connections 2023-08-25T09:50:02.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f456c1ed450 0x7f456c1ed8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f454c06a080 0x7f454c06c540 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f456c1ebba0 0x7f456c1ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 --2- 172.21.15.71:0/1159734274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4554003970 0x7f456c155170 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 >> 172.21.15.71:0/1159734274 conn(0x7f456c06bff0 msgr2=0x7f456c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:02.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 shutdown_connections 2023-08-25T09:50:02.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.596+0000 7f4570d42640 1 -- 172.21.15.71:0/1159734274 wait complete. 2023-08-25T09:50:02.619 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-08-25T09:50:02.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:50:02.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.843+0000 7f97d475d640 1 Processor -- start 2023-08-25T09:50:02.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.843+0000 7f97d475d640 1 -- start start 2023-08-25T09:50:02.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d475d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc152d80 0x7f97cc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:02.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d475d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97cc0c4560 0x7f97cc0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:02.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d475d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc06eb60 con 0x7f97cc070cf0 2023-08-25T09:50:02.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d475d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97cc06ca30 con 0x7f97cc0c4560 2023-08-25T09:50:02.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d475d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc06cf90 con 0x7f97cc152d80 2023-08-25T09:50:02.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d1cf8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f97cc0c4560 0x7f97cc0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:02.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d2cfa640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f97cc070cf0 0x7f97cc15c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46356/0 (socket says 172.21.15.71:46356) 2023-08-25T09:50:02.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d2cfa640 1 -- 172.21.15.71:0/165130391 learned_addr learned my addr 172.21.15.71:0/165130391 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:02.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d1cf8640 1 --2- 172.21.15.71:0/165130391 >> v2:172.21.15.71:3300/0 conn(0x7f97cc0c4560 0x7f97cc0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59894/0 (socket says 172.21.15.71:59894) 2023-08-25T09:50:02.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d24f9640 1 --2- 172.21.15.71:0/165130391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc152d80 0x7f97cc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:02.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d1cf8640 1 -- 172.21.15.71:0/165130391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc152d80 msgr2=0x7f97cc06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:02.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.845+0000 7f97d1cf8640 1 --2- 172.21.15.71:0/165130391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc152d80 0x7f97cc06e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.846+0000 7f97d1cf8640 1 -- 172.21.15.71:0/165130391 >> v1:172.21.15.71:6789/0 conn(0x7f97cc070cf0 legacy=0x7f97cc15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.846+0000 7f97d14f7640 1 -- 172.21.15.71:0/165130391 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064866107 0 0) 0x7f97cc06eb60 con 0x7f97cc070cf0 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.846+0000 7f97d1cf8640 1 -- 172.21.15.71:0/165130391 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc158fd0 con 0x7f97cc0c4560 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.847+0000 7f97d1cf8640 1 --2- 172.21.15.71:0/165130391 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc0c6080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e31e98804eedc68 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.847+0000 7f97d14f7640 1 -- 172.21.15.71:0/165130391 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc02e070 con 0x7f97cc0c4560 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.847+0000 7f97d14f7640 1 -- 172.21.15.71:0/165130391 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc0315a0 con 0x7f97cc0c4560 2023-08-25T09:50:02.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.847+0000 7f97d14f7640 1 -- 172.21.15.71:0/165130391 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc031930 con 0x7f97cc0c4560 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 -- 172.21.15.71:0/165130391 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 msgr2=0x7f97cc0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 --2- 172.21.15.71:0/165130391 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc0c6080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0097a0 tx=0x7f97bc02b840 comp rx=0 tx=0).stop 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 -- 172.21.15.71:0/165130391 shutdown_connections 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 --2- 172.21.15.71:0/165130391 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc0c6080 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 --2- 172.21.15.71:0/165130391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc152d80 0x7f97cc06e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 -- 172.21.15.71:0/165130391 >> 172.21.15.71:0/165130391 conn(0x7f97cc0654f0 msgr2=0x7f97cc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 -- 172.21.15.71:0/165130391 shutdown_connections 2023-08-25T09:50:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.848+0000 7f97d475d640 1 -- 172.21.15.71:0/165130391 wait complete. 2023-08-25T09:50:02.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.850+0000 7f97d475d640 1 Processor -- start 2023-08-25T09:50:02.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.850+0000 7f97d475d640 1 -- start start 2023-08-25T09:50:02.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d475d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc070cf0 0x7f97cc1f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:02.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d475d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc1ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:02.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d475d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97cc152d80 0x7f97cc14be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d475d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc1ee840 con 0x7f97cc0c4560 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d475d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc15a270 con 0x7f97cc152d80 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d475d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc1f2070 con 0x7f97cc070cf0 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d1cf8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc1ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d1cf8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc1ee300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59910/0 (socket says 172.21.15.71:59910) 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d1cf8640 1 -- 172.21.15.71:0/1015496971 learned_addr learned my addr 172.21.15.71:0/1015496971 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:02.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d2cfa640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97cc152d80 0x7f97cc14be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.851+0000 7f97d24f9640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc070cf0 0x7f97cc1f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d1cf8640 1 -- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc070cf0 msgr2=0x7f97cc1f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d1cf8640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc070cf0 0x7f97cc1f2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d1cf8640 1 -- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97cc152d80 msgr2=0x7f97cc14be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d1cf8640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97cc152d80 0x7f97cc14be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d1cf8640 1 -- 172.21.15.71:0/1015496971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc009400 con 0x7f97cc0c4560 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d2cfa640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97cc152d80 0x7f97cc14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:02.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d1cf8640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc1ee300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97bc034040 tx=0x7f97bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:02.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc032840 con 0x7f97cc0c4560 2023-08-25T09:50:02.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc0329a0 con 0x7f97cc0c4560 2023-08-25T09:50:02.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc032d50 con 0x7f97cc0c4560 2023-08-25T09:50:02.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc14b760 con 0x7f97cc0c4560 2023-08-25T09:50:02.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.852+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc14c390 con 0x7f97cc0c4560 2023-08-25T09:50:02.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.853+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc069160 con 0x7f97cc0c4560 2023-08-25T09:50:02.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.854+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f97bc04e430 con 0x7f97cc0c4560 2023-08-25T09:50:02.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.855+0000 7f97c2ffd640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f97b006a080 0x7f97b006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:02.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.855+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f97bc0aa9d0 con 0x7f97cc0c4560 2023-08-25T09:50:02.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.856+0000 7f97d24f9640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f97b006a080 0x7f97b006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:02.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.856+0000 7f97d24f9640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f97b006a080 0x7f97b006c540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f97c8006fd0 tx=0x7f97c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:02.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:02.859+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f97bc075d30 con 0x7f97cc0c4560 2023-08-25T09:50:03.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.085+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97cc0c8210 con 0x7f97cc0c4560 2023-08-25T09:50:03.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.086+0000 7f97c2ffd640 1 -- 172.21.15.71:0/1015496971 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6476 (secure 0 0 0) 0x7f97bc079350 con 0x7f97cc0c4560 2023-08-25T09:50:03.088 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:03.088 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:50:02.584545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4306,"mds_1":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:creating","state_seq":1,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:50:03.089 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 10 2023-08-25T09:50:03.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f97b006a080 msgr2=0x7f97b006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f97b006a080 0x7f97b006c540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f97c8006fd0 tx=0x7f97c8008040 comp rx=0 tx=0).stop 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 msgr2=0x7f97cc1ee300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc1ee300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97bc034040 tx=0x7f97bc0046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 shutdown_connections 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97cc152d80 0x7f97cc14be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f97b006a080 0x7f97b006c540 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97cc0c4560 0x7f97cc1ee300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 --2- 172.21.15.71:0/1015496971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97cc070cf0 0x7f97cc1f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 >> 172.21.15.71:0/1015496971 conn(0x7f97cc0654f0 msgr2=0x7f97cc0c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 shutdown_connections 2023-08-25T09:50:03.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.089+0000 7f97d475d640 1 -- 172.21.15.71:0/1015496971 wait complete. 2023-08-25T09:50:03.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.307+0000 7f7b0cc71640 1 Processor -- start 2023-08-25T09:50:03.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.307+0000 7f7b0cc71640 1 -- start start 2023-08-25T09:50:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b0cc71640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7b081542d0 0x7f7b081576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b0cc71640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b080c4fb0 0x7f7b080c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b0cc71640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b08157bf0 con 0x7f7b08069850 2023-08-25T09:50:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b0cc71640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b080c4b60 con 0x7f7b081542d0 2023-08-25T09:50:03.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b0cc71640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0806a130 con 0x7f7b080c4fb0 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b077fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7b081542d0 0x7f7b081576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b077fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7b081542d0 0x7f7b081576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59912/0 (socket says 172.21.15.71:59912) 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b077fe640 1 -- 172.21.15.71:0/299310681 learned_addr learned my addr 172.21.15.71:0/299310681 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b07fff640 1 --2- 172.21.15.71:0/299310681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b080c4fb0 0x7f7b080c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b077fe640 1 -- 172.21.15.71:0/299310681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b080c4fb0 msgr2=0x7f7b080c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b077fe640 1 --2- 172.21.15.71:0/299310681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b080c4fb0 0x7f7b080c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.309+0000 7f7b077fe640 1 -- 172.21.15.71:0/299310681 >> v1:172.21.15.164:6789/0 conn(0x7f7b08069850 legacy=0x7f7b080c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:03.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.310+0000 7f7b077fe640 1 -- 172.21.15.71:0/299310681 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080c5980 con 0x7f7b081542d0 2023-08-25T09:50:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.311+0000 7f7b077fe640 1 --2- 172.21.15.71:0/299310681 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b081542d0 0x7f7b081576b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009870 tx=0x7f7af802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b5e0e9e2ac83a7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.311+0000 7f7b067fc640 1 -- 172.21.15.71:0/299310681 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8031640 con 0x7f7b081542d0 2023-08-25T09:50:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.311+0000 7f7b067fc640 1 -- 172.21.15.71:0/299310681 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af80317a0 con 0x7f7b081542d0 2023-08-25T09:50:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.311+0000 7f7b067fc640 1 -- 172.21.15.71:0/299310681 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af8031b30 con 0x7f7b081542d0 2023-08-25T09:50:03.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 -- 172.21.15.71:0/299310681 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b081542d0 msgr2=0x7f7b081576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/299310681 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b081542d0 0x7f7b081576b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009870 tx=0x7f7af802b8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 -- 172.21.15.71:0/299310681 shutdown_connections 2023-08-25T09:50:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/299310681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b080c4fb0 0x7f7b080c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/299310681 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b081542d0 0x7f7b081576b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 -- 172.21.15.71:0/299310681 >> 172.21.15.71:0/299310681 conn(0x7f7b0806c280 msgr2=0x7f7b0806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 -- 172.21.15.71:0/299310681 shutdown_connections 2023-08-25T09:50:03.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.312+0000 7f7b0cc71640 1 -- 172.21.15.71:0/299310681 wait complete. 2023-08-25T09:50:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.314+0000 7f7b0cc71640 1 Processor -- start 2023-08-25T09:50:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.314+0000 7f7b0cc71640 1 -- start start 2023-08-25T09:50:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b0cc71640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b08069850 0x7f7b08155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b0cc71640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 0x7f7b080ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b0cc71640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b081542d0 0x7f7b080adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b0cc71640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b081f2200 con 0x7f7b080c4fb0 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b0cc71640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0815b080 con 0x7f7b081542d0 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b0cc71640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b081555d0 con 0x7f7b08069850 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.315+0000 7f7b06ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 0x7f7b080ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 0x7f7b080ad8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59928/0 (socket says 172.21.15.71:59928) 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 -- 172.21.15.71:0/1568845527 learned_addr learned my addr 172.21.15.71:0/1568845527 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:03.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b077fe640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b08069850 0x7f7b08155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b07fff640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b081542d0 0x7f7b080adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 -- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b08069850 msgr2=0x7f7b08155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b08069850 0x7f7b08155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 -- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b081542d0 msgr2=0x7f7b080adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b081542d0 0x7f7b080adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 -- 172.21.15.71:0/1568845527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80094d0 con 0x7f7b080c4fb0 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b07fff640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b081542d0 0x7f7b080adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.316+0000 7f7b06ffd640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 0x7f7b080ad8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00c460 tx=0x7f7afc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.317+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc011790 con 0x7f7b080c4fb0 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.317+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc011930 con 0x7f7b080c4fb0 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.317+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc011ce0 con 0x7f7b080c4fb0 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.317+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0815ab00 con 0x7f7b080c4fb0 2023-08-25T09:50:03.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.317+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b081edea0 con 0x7f7b080c4fb0 2023-08-25T09:50:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.318+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b081edd00 con 0x7f7b080c4fb0 2023-08-25T09:50:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.319+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7afc01c460 con 0x7f7b080c4fb0 2023-08-25T09:50:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.320+0000 7f7aeffff640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7af406a050 0x7f7af406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.320+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f7afc088910 con 0x7f7b080c4fb0 2023-08-25T09:50:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.320+0000 7f7b077fe640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7af406a050 0x7f7af406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.321+0000 7f7b077fe640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7af406a050 0x7f7af406c510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7af8036000 tx=0x7f7af803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:03.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.324+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7b081edd00 con 0x7f7b080c4fb0 2023-08-25T09:50:03.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.515+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b081edd00 con 0x7f7b080c4fb0 2023-08-25T09:50:03.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.518+0000 7f7aeffff640 1 -- 172.21.15.71:0/1568845527 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7f7b081edd00 con 0x7f7b080c4fb0 2023-08-25T09:50:03.521 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:03.522 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:03.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7af406a050 msgr2=0x7f7af406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7af406a050 0x7f7af406c510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7af8036000 tx=0x7f7af803c040 comp rx=0 tx=0).stop 2023-08-25T09:50:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 msgr2=0x7f7b080ad8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 0x7f7b080ad8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00c460 tx=0x7f7afc00c920 comp rx=0 tx=0).stop 2023-08-25T09:50:03.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 shutdown_connections 2023-08-25T09:50:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b081542d0 0x7f7b080adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7af406a050 0x7f7af406c510 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b080c4fb0 0x7f7b080ad8a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 --2- 172.21.15.71:0/1568845527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b08069850 0x7f7b08155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.521+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 >> 172.21.15.71:0/1568845527 conn(0x7f7b0806c280 msgr2=0x7f7b08152190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.522+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 shutdown_connections 2023-08-25T09:50:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.522+0000 7f7b0cc71640 1 -- 172.21.15.71:0/1568845527 wait complete. 2023-08-25T09:50:03.543 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-08-25T09:50:03.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:03.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.735+0000 7f8e107a6640 1 Processor -- start 2023-08-25T09:50:03.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.736+0000 7f8e107a6640 1 -- start start 2023-08-25T09:50:03.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.737+0000 7f8e107a6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8e080c3a40 0x7f8e080c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.737+0000 7f8e107a6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8e081509d0 0x7f8e08150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.737+0000 7f8e107a6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080c1dc0 con 0x7f8e080c3660 2023-08-25T09:50:03.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.737+0000 7f8e107a6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e08157a80 con 0x7f8e080c3a40 2023-08-25T09:50:03.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.737+0000 7f8e107a6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e08151320 con 0x7f8e081509d0 2023-08-25T09:50:03.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.737+0000 7f8e0dd41640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8e080c3a40 0x7f8e080c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0dd41640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8e080c3a40 0x7f8e080c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59932/0 (socket says 172.21.15.71:59932) 2023-08-25T09:50:03.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0dd41640 1 -- 172.21.15.71:0/3321356538 learned_addr learned my addr 172.21.15.71:0/3321356538 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:03.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0e542640 1 --1- 172.21.15.71:0/3321356538 >> v1:172.21.15.71:6789/0 conn(0x7f8e080c3660 0x7f8e0806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46362/0 (socket says 172.21.15.71:46362) 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0ed43640 1 --2- 172.21.15.71:0/3321356538 >> v2:172.21.15.164:3300/0 conn(0x7f8e081509d0 0x7f8e08150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0dd41640 1 -- 172.21.15.71:0/3321356538 >> v2:172.21.15.164:3300/0 conn(0x7f8e081509d0 msgr2=0x7f8e08150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0dd41640 1 --2- 172.21.15.71:0/3321356538 >> v2:172.21.15.164:3300/0 conn(0x7f8e081509d0 0x7f8e08150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0dd41640 1 -- 172.21.15.71:0/3321356538 >> v1:172.21.15.71:6789/0 conn(0x7f8e080c3660 legacy=0x7f8e0806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0dd41640 1 -- 172.21.15.71:0/3321356538 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e08156e00 con 0x7f8e080c3a40 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0ed43640 1 --2- 172.21.15.71:0/3321356538 >> v2:172.21.15.164:3300/0 conn(0x7f8e081509d0 0x7f8e08150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:03.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.738+0000 7f8e0d540640 1 -- 172.21.15.71:0/3321356538 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163446402 0 0) 0x7f8e080c1dc0 con 0x7f8e080c3660 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.739+0000 7f8e0dd41640 1 --2- 172.21.15.71:0/3321356538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e080c3a40 0x7f8e080c1710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e080663f0 tx=0x7f8df802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d90546166e8b5d9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.739+0000 7f8e0d540640 1 -- 172.21.15.71:0/3321356538 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df802f850 con 0x7f8e080c3a40 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.739+0000 7f8e0d540640 1 -- 172.21.15.71:0/3321356538 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df802f9b0 con 0x7f8e080c3a40 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.739+0000 7f8e0d540640 1 -- 172.21.15.71:0/3321356538 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df802fd40 con 0x7f8e080c3a40 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 -- 172.21.15.71:0/3321356538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e080c3a40 msgr2=0x7f8e080c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 --2- 172.21.15.71:0/3321356538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e080c3a40 0x7f8e080c1710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e080663f0 tx=0x7f8df802ba30 comp rx=0 tx=0).stop 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 -- 172.21.15.71:0/3321356538 shutdown_connections 2023-08-25T09:50:03.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 --2- 172.21.15.71:0/3321356538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e080c3a40 0x7f8e080c1710 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 --2- 172.21.15.71:0/3321356538 >> v2:172.21.15.164:3300/0 conn(0x7f8e081509d0 0x7f8e08150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 -- 172.21.15.71:0/3321356538 >> 172.21.15.71:0/3321356538 conn(0x7f8e080654f0 msgr2=0x7f8e08065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:03.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 -- 172.21.15.71:0/3321356538 shutdown_connections 2023-08-25T09:50:03.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.740+0000 7f8e107a6640 1 -- 172.21.15.71:0/3321356538 wait complete. 2023-08-25T09:50:03.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.742+0000 7f8e107a6640 1 Processor -- start 2023-08-25T09:50:03.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.742+0000 7f8e107a6640 1 -- start start 2023-08-25T09:50:03.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e107a6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e080c3660 0x7f8e0814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e107a6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e080c3a40 0x7f8e08158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e107a6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 0x7f8e081ec320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e107a6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0814a640 con 0x7f8e081509d0 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e107a6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0814aee0 con 0x7f8e080c3660 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e107a6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0806aa20 con 0x7f8e080c3a40 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e0ed43640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 0x7f8e081ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e0ed43640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 0x7f8e081ec320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59948/0 (socket says 172.21.15.71:59948) 2023-08-25T09:50:03.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e0ed43640 1 -- 172.21.15.71:0/4289918210 learned_addr learned my addr 172.21.15.71:0/4289918210 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:03.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.743+0000 7f8e0e542640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e080c3660 0x7f8e0814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e0ed43640 1 -- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e080c3a40 msgr2=0x7f8e08158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e0ed43640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e080c3a40 0x7f8e08158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e0ed43640 1 -- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e080c3660 msgr2=0x7f8e0814b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e0ed43640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e080c3660 0x7f8e0814b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e0ed43640 1 -- 172.21.15.71:0/4289918210 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80094d0 con 0x7f8e081509d0 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e0ed43640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 0x7f8e081ec320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000cd80 tx=0x7f8e0000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000f680 con 0x7f8e081509d0 2023-08-25T09:50:03.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0000f800 con 0x7f8e081509d0 2023-08-25T09:50:03.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000fbb0 con 0x7f8e081509d0 2023-08-25T09:50:03.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0806ace0 con 0x7f8e081509d0 2023-08-25T09:50:03.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.744+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0814ab70 con 0x7f8e081509d0 2023-08-25T09:50:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.746+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e080c29c0 con 0x7f8e081509d0 2023-08-25T09:50:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.746+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8e0000b040 con 0x7f8e081509d0 2023-08-25T09:50:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.747+0000 7f8dfeffd640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8dec06a080 0x7f8dec06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.747+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8e000893a0 con 0x7f8e081509d0 2023-08-25T09:50:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.748+0000 7f8e0e542640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8dec06a080 0x7f8dec06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:03.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.748+0000 7f8e0e542640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8dec06a080 0x7f8dec06c540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8e04000f60 tx=0x7f8e04007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:03.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.752+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8e00054670 con 0x7f8e081509d0 2023-08-25T09:50:03.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.932+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e08154100 con 0x7f8dec06a080 2023-08-25T09:50:03.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.945+0000 7f8dfeffd640 1 -- 172.21.15.71:0/4289918210 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367292 (secure 0 0 0) 0x7f8e08154100 con 0x7f8dec06a080 2023-08-25T09:50:03.954 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:03.956 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:03.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8dec06a080 msgr2=0x7f8dec06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8dec06a080 0x7f8dec06c540 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8e04000f60 tx=0x7f8e04007b50 comp rx=0 tx=0).stop 2023-08-25T09:50:03.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 msgr2=0x7f8e081ec320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:03.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 0x7f8e081ec320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000cd80 tx=0x7f8e0000a650 comp rx=0 tx=0).stop 2023-08-25T09:50:03.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 shutdown_connections 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8dec06a080 0x7f8dec06c540 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e081509d0 0x7f8e081ec320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e080c3a40 0x7f8e08158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 --2- 172.21.15.71:0/4289918210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e080c3660 0x7f8e0814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 >> 172.21.15.71:0/4289918210 conn(0x7f8e080654f0 msgr2=0x7f8e080c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.951+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 shutdown_connections 2023-08-25T09:50:03.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:03.952+0000 7f8e107a6640 1 -- 172.21.15.71:0/4289918210 wait complete. 2023-08-25T09:50:03.969 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2023-08-25T09:50:02.643913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227080,"kb_used_data":2592,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747640,"statfs":{"total":773094113280,"available":772861583360,"internally_reserved":0,"allocated":2654208,"data_stored":2199862,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004678"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532259+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:49:40.532259+0000","last_peered":"2023-08-25T09:49:40.532259+0000","last_clean":"2023-08-25T09:49:40.532259+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:49:40.532259+0000","last_undegraded":"2023-08-25T09:49:40.532259+0000","last_fullsized":"2023-08-25T09:49:40.532259+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:08:44.154380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670137+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:49:39.670137+0000","last_peered":"2023-08-25T09:49:39.670137+0000","last_clean":"2023-08-25T09:49:39.670137+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:49:39.670137+0000","last_undegraded":"2023-08-25T09:49:39.670137+0000","last_fullsized":"2023-08-25T09:49:39.670137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:19:21.160750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672075+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:49:39.672075+0000","last_peered":"2023-08-25T09:49:39.672075+0000","last_clean":"2023-08-25T09:49:39.672075+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:49:39.672075+0000","last_undegraded":"2023-08-25T09:49:39.672075+0000","last_fullsized":"2023-08-25T09:49:39.672075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:39:44.896839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.667983+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:49:39.667983+0000","last_peered":"2023-08-25T09:49:39.667983+0000","last_clean":"2023-08-25T09:49:39.667983+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:49:39.667983+0000","last_undegraded":"2023-08-25T09:49:39.667983+0000","last_fullsized":"2023-08-25T09:49:39.667983+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:52:21.149049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610317+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:49:40.610317+0000","last_peered":"2023-08-25T09:49:40.610317+0000","last_clean":"2023-08-25T09:49:40.610317+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:49:40.610317+0000","last_undegraded":"2023-08-25T09:49:40.610317+0000","last_fullsized":"2023-08-25T09:49:40.610317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.067887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609551+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:49:40.609551+0000","last_peered":"2023-08-25T09:49:40.609551+0000","last_clean":"2023-08-25T09:49:40.609551+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:49:40.609551+0000","last_undegraded":"2023-08-25T09:49:40.609551+0000","last_fullsized":"2023-08-25T09:49:40.609551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:31:30.676338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669356+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:49:39.669356+0000","last_peered":"2023-08-25T09:49:39.669356+0000","last_clean":"2023-08-25T09:49:39.669356+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:49:39.669356+0000","last_undegraded":"2023-08-25T09:49:39.669356+0000","last_fullsized":"2023-08-25T09:49:39.669356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:52:41.076094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672443+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:49:39.672443+0000","last_peered":"2023-08-25T09:49:39.672443+0000","last_clean":"2023-08-25T09:49:39.672443+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:49:39.672443+0000","last_undegraded":"2023-08-25T09:49:39.672443+0000","last_fullsized":"2023-08-25T09:49:39.672443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:41:06.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670837+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:49:39.670837+0000","last_peered":"2023-08-25T09:49:39.670837+0000","last_clean":"2023-08-25T09:49:39.670837+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:49:39.670837+0000","last_undegraded":"2023-08-25T09:49:39.670837+0000","last_fullsized":"2023-08-25T09:49:39.670837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:43.561619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669087+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:49:39.669087+0000","last_peered":"2023-08-25T09:49:39.669087+0000","last_clean":"2023-08-25T09:49:39.669087+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:49:39.669087+0000","last_undegraded":"2023-08-25T09:49:39.669087+0000","last_fullsized":"2023-08-25T09:49:39.669087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:29:31.719174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668364+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:49:39.668364+0000","last_peered":"2023-08-25T09:49:39.668364+0000","last_clean":"2023-08-25T09:49:39.668364+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:49:39.668364+0000","last_undegraded":"2023-08-25T09:49:39.668364+0000","last_fullsized":"2023-08-25T09:49:39.668364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:54:21.024745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669503+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:49:39.669503+0000","last_peered":"2023-08-25T09:49:39.669503+0000","last_clean":"2023-08-25T09:49:39.669503+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:49:39.669503+0000","last_undegraded":"2023-08-25T09:49:39.669503+0000","last_fullsized":"2023-08-25T09:49:39.669503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:38.132579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533392+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:49:40.533392+0000","last_peered":"2023-08-25T09:49:40.533392+0000","last_clean":"2023-08-25T09:49:40.533392+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:49:40.533392+0000","last_undegraded":"2023-08-25T09:49:40.533392+0000","last_fullsized":"2023-08-25T09:49:40.533392+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:44:04.188030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669599+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:49:39.669599+0000","last_peered":"2023-08-25T09:49:39.669599+0000","last_clean":"2023-08-25T09:49:39.669599+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:49:39.669599+0000","last_undegraded":"2023-08-25T09:49:39.669599+0000","last_fullsized":"2023-08-25T09:49:39.669599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:36:03.677003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314444+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:49:40.314444+0000","last_peered":"2023-08-25T09:49:40.314444+0000","last_clean":"2023-08-25T09:49:40.314444+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:49:40.314444+0000","last_undegraded":"2023-08-25T09:49:40.314444+0000","last_fullsized":"2023-08-25T09:49:40.314444+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:24:33.940678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670152+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:49:39.670152+0000","last_peered":"2023-08-25T09:49:39.670152+0000","last_clean":"2023-08-25T09:49:39.670152+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:49:39.670152+0000","last_undegraded":"2023-08-25T09:49:39.670152+0000","last_fullsized":"2023-08-25T09:49:39.670152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:04.589805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671210+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:49:39.671210+0000","last_peered":"2023-08-25T09:49:39.671210+0000","last_clean":"2023-08-25T09:49:39.671210+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:49:39.671210+0000","last_undegraded":"2023-08-25T09:49:39.671210+0000","last_fullsized":"2023-08-25T09:49:39.671210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:21.872702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610243+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:49:40.610243+0000","last_peered":"2023-08-25T09:49:40.610243+0000","last_clean":"2023-08-25T09:49:40.610243+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:49:40.610243+0000","last_undegraded":"2023-08-25T09:49:40.610243+0000","last_fullsized":"2023-08-25T09:49:40.610243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:40:11.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610311+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:49:40.610311+0000","last_peered":"2023-08-25T09:49:40.610311+0000","last_clean":"2023-08-25T09:49:40.610311+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:49:40.610311+0000","last_undegraded":"2023-08-25T09:49:40.610311+0000","last_fullsized":"2023-08-25T09:49:40.610311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:24:36.237719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532845+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:49:40.532845+0000","last_peered":"2023-08-25T09:49:40.532845+0000","last_clean":"2023-08-25T09:49:40.532845+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:49:40.532845+0000","last_undegraded":"2023-08-25T09:49:40.532845+0000","last_fullsized":"2023-08-25T09:49:40.532845+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:17.775607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669479+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:49:39.669479+0000","last_peered":"2023-08-25T09:49:39.669479+0000","last_clean":"2023-08-25T09:49:39.669479+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:49:39.669479+0000","last_undegraded":"2023-08-25T09:49:39.669479+0000","last_fullsized":"2023-08-25T09:49:39.669479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:24:10.790283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669214+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:49:39.669214+0000","last_peered":"2023-08-25T09:49:39.669214+0000","last_clean":"2023-08-25T09:49:39.669214+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:49:39.669214+0000","last_undegraded":"2023-08-25T09:49:39.669214+0000","last_fullsized":"2023-08-25T09:49:39.669214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:45:44.040670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609056+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:49:40.609056+0000","last_peered":"2023-08-25T09:49:40.609056+0000","last_clean":"2023-08-25T09:49:40.609056+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:49:40.609056+0000","last_undegraded":"2023-08-25T09:49:40.609056+0000","last_fullsized":"2023-08-25T09:49:40.609056+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:30:29.949537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610535+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:49:40.610535+0000","last_peered":"2023-08-25T09:49:40.610535+0000","last_clean":"2023-08-25T09:49:40.610535+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:49:40.610535+0000","last_undegraded":"2023-08-25T09:49:40.610535+0000","last_fullsized":"2023-08-25T09:49:40.610535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:33:47.145180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608671+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.608671+0000","last_peered":"2023-08-25T09:49:40.608671+0000","last_clean":"2023-08-25T09:49:40.608671+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:49:40.608671+0000","last_undegraded":"2023-08-25T09:49:40.608671+0000","last_fullsized":"2023-08-25T09:49:40.608671+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:49:14.644166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670296+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:49:39.670296+0000","last_peered":"2023-08-25T09:49:39.670296+0000","last_clean":"2023-08-25T09:49:39.670296+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:49:39.670296+0000","last_undegraded":"2023-08-25T09:49:39.670296+0000","last_fullsized":"2023-08-25T09:49:39.670296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:33:42.072752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609238+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:49:40.609238+0000","last_peered":"2023-08-25T09:49:40.609238+0000","last_clean":"2023-08-25T09:49:40.609238+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:49:40.609238+0000","last_undegraded":"2023-08-25T09:49:40.609238+0000","last_fullsized":"2023-08-25T09:49:40.609238+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:01:49.724298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609229+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:49:40.609229+0000","last_peered":"2023-08-25T09:49:40.609229+0000","last_clean":"2023-08-25T09:49:40.609229+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:49:40.609229+0000","last_undegraded":"2023-08-25T09:49:40.609229+0000","last_fullsized":"2023-08-25T09:49:40.609229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:35:14.991310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669355+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:49:39.669355+0000","last_peered":"2023-08-25T09:49:39.669355+0000","last_clean":"2023-08-25T09:49:39.669355+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:49:39.669355+0000","last_undegraded":"2023-08-25T09:49:39.669355+0000","last_fullsized":"2023-08-25T09:49:39.669355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:36:04.594929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669714+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:49:39.669714+0000","last_peered":"2023-08-25T09:49:39.669714+0000","last_clean":"2023-08-25T09:49:39.669714+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:49:39.669714+0000","last_undegraded":"2023-08-25T09:49:39.669714+0000","last_fullsized":"2023-08-25T09:49:39.669714+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:13:29.132994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"28'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671383+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:49:39.671383+0000","last_peered":"2023-08-25T09:49:39.671383+0000","last_clean":"2023-08-25T09:49:39.671383+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:49:39.671383+0000","last_undegraded":"2023-08-25T09:49:39.671383+0000","last_fullsized":"2023-08-25T09:49:39.671383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:08:04.314595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533055+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:49:40.533055+0000","last_peered":"2023-08-25T09:49:40.533055+0000","last_clean":"2023-08-25T09:49:40.533055+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:49:40.533055+0000","last_undegraded":"2023-08-25T09:49:40.533055+0000","last_fullsized":"2023-08-25T09:49:40.533055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:29.319338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609374+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:49:40.609374+0000","last_peered":"2023-08-25T09:49:40.609374+0000","last_clean":"2023-08-25T09:49:40.609374+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:49:40.609374+0000","last_undegraded":"2023-08-25T09:49:40.609374+0000","last_fullsized":"2023-08-25T09:49:40.609374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:11:38.846168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"29'10","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:57.914119+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:49:57.914119+0000","last_peered":"2023-08-25T09:49:57.914119+0000","last_clean":"2023-08-25T09:49:57.914119+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:49:57.914119+0000","last_undegraded":"2023-08-25T09:49:57.914119+0000","last_fullsized":"2023-08-25T09:49:57.914119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:19:49.118130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669680+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:49:39.669680+0000","last_peered":"2023-08-25T09:49:39.669680+0000","last_clean":"2023-08-25T09:49:39.669680+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:49:39.669680+0000","last_undegraded":"2023-08-25T09:49:39.669680+0000","last_fullsized":"2023-08-25T09:49:39.669680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:15:08.913052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532469+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:49:40.532469+0000","last_peered":"2023-08-25T09:49:40.532469+0000","last_clean":"2023-08-25T09:49:40.532469+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:49:40.532469+0000","last_undegraded":"2023-08-25T09:49:40.532469+0000","last_fullsized":"2023-08-25T09:49:40.532469+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:54:59.656364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669355+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:49:39.669355+0000","last_peered":"2023-08-25T09:49:39.669355+0000","last_clean":"2023-08-25T09:49:39.669355+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:49:39.669355+0000","last_undegraded":"2023-08-25T09:49:39.669355+0000","last_fullsized":"2023-08-25T09:49:39.669355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:48.989731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610694+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:49:40.610694+0000","last_peered":"2023-08-25T09:49:40.610694+0000","last_clean":"2023-08-25T09:49:40.610694+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:49:40.610694+0000","last_undegraded":"2023-08-25T09:49:40.610694+0000","last_fullsized":"2023-08-25T09:49:40.610694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:46.063058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670003+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:49:39.670003+0000","last_peered":"2023-08-25T09:49:39.670003+0000","last_clean":"2023-08-25T09:49:39.670003+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:49:39.670003+0000","last_undegraded":"2023-08-25T09:49:39.670003+0000","last_fullsized":"2023-08-25T09:49:39.670003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:13:11.987428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608483+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:49:40.608483+0000","last_peered":"2023-08-25T09:49:40.608483+0000","last_clean":"2023-08-25T09:49:40.608483+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:49:40.608483+0000","last_undegraded":"2023-08-25T09:49:40.608483+0000","last_fullsized":"2023-08-25T09:49:40.608483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:28:33.537079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608779+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:49:40.608779+0000","last_peered":"2023-08-25T09:49:40.608779+0000","last_clean":"2023-08-25T09:49:40.608779+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:49:40.608779+0000","last_undegraded":"2023-08-25T09:49:40.608779+0000","last_fullsized":"2023-08-25T09:49:40.608779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:01:10.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.534056+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:49:40.534056+0000","last_peered":"2023-08-25T09:49:40.534056+0000","last_clean":"2023-08-25T09:49:40.534056+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:49:40.534056+0000","last_undegraded":"2023-08-25T09:49:40.534056+0000","last_fullsized":"2023-08-25T09:49:40.534056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:30.208912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.611163+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:49:40.611163+0000","last_peered":"2023-08-25T09:49:40.611163+0000","last_clean":"2023-08-25T09:49:40.611163+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:49:40.611163+0000","last_undegraded":"2023-08-25T09:49:40.611163+0000","last_fullsized":"2023-08-25T09:49:40.611163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:46:45.992240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668225+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:49:39.668225+0000","last_peered":"2023-08-25T09:49:39.668225+0000","last_clean":"2023-08-25T09:49:39.668225+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:49:39.668225+0000","last_undegraded":"2023-08-25T09:49:39.668225+0000","last_fullsized":"2023-08-25T09:49:39.668225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:00:13.974928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669196+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:49:39.669196+0000","last_peered":"2023-08-25T09:49:39.669196+0000","last_clean":"2023-08-25T09:49:39.669196+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:49:39.669196+0000","last_undegraded":"2023-08-25T09:49:39.669196+0000","last_fullsized":"2023-08-25T09:49:39.669196+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:28:03.992015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671806+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:49:39.671806+0000","last_peered":"2023-08-25T09:49:39.671806+0000","last_clean":"2023-08-25T09:49:39.671806+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:49:39.671806+0000","last_undegraded":"2023-08-25T09:49:39.671806+0000","last_fullsized":"2023-08-25T09:49:39.671806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:14:52.844903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533580+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:49:40.533580+0000","last_peered":"2023-08-25T09:49:40.533580+0000","last_clean":"2023-08-25T09:49:40.533580+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:49:40.533580+0000","last_undegraded":"2023-08-25T09:49:40.533580+0000","last_fullsized":"2023-08-25T09:49:40.533580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:00.063674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609540+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:49:40.609540+0000","last_peered":"2023-08-25T09:49:40.609540+0000","last_clean":"2023-08-25T09:49:40.609540+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:49:40.609540+0000","last_undegraded":"2023-08-25T09:49:40.609540+0000","last_fullsized":"2023-08-25T09:49:40.609540+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:59:18.199533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532919+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:49:40.532919+0000","last_peered":"2023-08-25T09:49:40.532919+0000","last_clean":"2023-08-25T09:49:40.532919+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:49:40.532919+0000","last_undegraded":"2023-08-25T09:49:40.532919+0000","last_fullsized":"2023-08-25T09:49:40.532919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:00:24.110176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533867+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:49:40.533867+0000","last_peered":"2023-08-25T09:49:40.533867+0000","last_clean":"2023-08-25T09:49:40.533867+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:49:40.533867+0000","last_undegraded":"2023-08-25T09:49:40.533867+0000","last_fullsized":"2023-08-25T09:49:40.533867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:25:26.896240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672580+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:49:39.672580+0000","last_peered":"2023-08-25T09:49:39.672580+0000","last_clean":"2023-08-25T09:49:39.672580+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:49:39.672580+0000","last_undegraded":"2023-08-25T09:49:39.672580+0000","last_fullsized":"2023-08-25T09:49:39.672580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:23.747043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668857+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:49:39.668857+0000","last_peered":"2023-08-25T09:49:39.668857+0000","last_clean":"2023-08-25T09:49:39.668857+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:49:39.668857+0000","last_undegraded":"2023-08-25T09:49:39.668857+0000","last_fullsized":"2023-08-25T09:49:39.668857+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:12:48.827644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533435+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:49:40.533435+0000","last_peered":"2023-08-25T09:49:40.533435+0000","last_clean":"2023-08-25T09:49:40.533435+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:49:40.533435+0000","last_undegraded":"2023-08-25T09:49:40.533435+0000","last_fullsized":"2023-08-25T09:49:40.533435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:07:36.521502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669650+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:49:39.669650+0000","last_peered":"2023-08-25T09:49:39.669650+0000","last_clean":"2023-08-25T09:49:39.669650+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:49:39.669650+0000","last_undegraded":"2023-08-25T09:49:39.669650+0000","last_fullsized":"2023-08-25T09:49:39.669650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:26:07.776398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668924+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:49:39.668924+0000","last_peered":"2023-08-25T09:49:39.668924+0000","last_clean":"2023-08-25T09:49:39.668924+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:49:39.668924+0000","last_undegraded":"2023-08-25T09:49:39.668924+0000","last_fullsized":"2023-08-25T09:49:39.668924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:22:13.379657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669047+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:49:39.669047+0000","last_peered":"2023-08-25T09:49:39.669047+0000","last_clean":"2023-08-25T09:49:39.669047+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:49:39.669047+0000","last_undegraded":"2023-08-25T09:49:39.669047+0000","last_fullsized":"2023-08-25T09:49:39.669047+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:36:02.088533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533994+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:49:40.533994+0000","last_peered":"2023-08-25T09:49:40.533994+0000","last_clean":"2023-08-25T09:49:40.533994+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:49:40.533994+0000","last_undegraded":"2023-08-25T09:49:40.533994+0000","last_fullsized":"2023-08-25T09:49:40.533994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:34:02.269298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532502+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:49:40.532502+0000","last_peered":"2023-08-25T09:49:40.532502+0000","last_clean":"2023-08-25T09:49:40.532502+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:49:40.532502+0000","last_undegraded":"2023-08-25T09:49:40.532502+0000","last_fullsized":"2023-08-25T09:49:40.532502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:52:34.012911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313601+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:49:40.313601+0000","last_peered":"2023-08-25T09:49:40.313601+0000","last_clean":"2023-08-25T09:49:40.313601+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:49:40.313601+0000","last_undegraded":"2023-08-25T09:49:40.313601+0000","last_fullsized":"2023-08-25T09:49:40.313601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:28.801938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314730+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:49:40.314730+0000","last_peered":"2023-08-25T09:49:40.314730+0000","last_clean":"2023-08-25T09:49:40.314730+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:49:40.314730+0000","last_undegraded":"2023-08-25T09:49:40.314730+0000","last_fullsized":"2023-08-25T09:49:40.314730+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:17:34.425122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669544+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:49:39.669544+0000","last_peered":"2023-08-25T09:49:39.669544+0000","last_clean":"2023-08-25T09:49:39.669544+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:49:39.669544+0000","last_undegraded":"2023-08-25T09:49:39.669544+0000","last_fullsized":"2023-08-25T09:49:39.669544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:44.776136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668099+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:49:39.668099+0000","last_peered":"2023-08-25T09:49:39.668099+0000","last_clean":"2023-08-25T09:49:39.668099+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:49:39.668099+0000","last_undegraded":"2023-08-25T09:49:39.668099+0000","last_fullsized":"2023-08-25T09:49:39.668099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:08.959803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670236+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:49:39.670236+0000","last_peered":"2023-08-25T09:49:39.670236+0000","last_clean":"2023-08-25T09:49:39.670236+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:49:39.670236+0000","last_undegraded":"2023-08-25T09:49:39.670236+0000","last_fullsized":"2023-08-25T09:49:39.670236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:04.664807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668818+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:49:39.668818+0000","last_peered":"2023-08-25T09:49:39.668818+0000","last_clean":"2023-08-25T09:49:39.668818+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:49:39.668818+0000","last_undegraded":"2023-08-25T09:49:39.668818+0000","last_fullsized":"2023-08-25T09:49:39.668818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:45:13.832806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610040+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:49:40.610040+0000","last_peered":"2023-08-25T09:49:40.610040+0000","last_clean":"2023-08-25T09:49:40.610040+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:49:40.610040+0000","last_undegraded":"2023-08-25T09:49:40.610040+0000","last_fullsized":"2023-08-25T09:49:40.610040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:08.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609484+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:49:40.609484+0000","last_peered":"2023-08-25T09:49:40.609484+0000","last_clean":"2023-08-25T09:49:40.609484+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:49:40.609484+0000","last_undegraded":"2023-08-25T09:49:40.609484+0000","last_fullsized":"2023-08-25T09:49:40.609484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:02.892745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671904+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:49:39.671904+0000","last_peered":"2023-08-25T09:49:39.671904+0000","last_clean":"2023-08-25T09:49:39.671904+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:49:39.671904+0000","last_undegraded":"2023-08-25T09:49:39.671904+0000","last_fullsized":"2023-08-25T09:49:39.671904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:12:08.551735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":183,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670426+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:49:39.670426+0000","last_peered":"2023-08-25T09:49:39.670426+0000","last_clean":"2023-08-25T09:49:39.670426+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:49:39.670426+0000","last_undegraded":"2023-08-25T09:49:39.670426+0000","last_fullsized":"2023-08-25T09:49:39.670426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:14:32.439794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670825+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:49:39.670825+0000","last_peered":"2023-08-25T09:49:39.670825+0000","last_clean":"2023-08-25T09:49:39.670825+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:49:39.670825+0000","last_undegraded":"2023-08-25T09:49:39.670825+0000","last_fullsized":"2023-08-25T09:49:39.670825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:14:42.235888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533862+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:49:40.533862+0000","last_peered":"2023-08-25T09:49:40.533862+0000","last_clean":"2023-08-25T09:49:40.533862+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:49:40.533862+0000","last_undegraded":"2023-08-25T09:49:40.533862+0000","last_fullsized":"2023-08-25T09:49:40.533862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:47:24.625104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609113+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:49:40.609113+0000","last_peered":"2023-08-25T09:49:40.609113+0000","last_clean":"2023-08-25T09:49:40.609113+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:49:40.609113+0000","last_undegraded":"2023-08-25T09:49:40.609113+0000","last_fullsized":"2023-08-25T09:49:40.609113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:34:01.834958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670693+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:49:39.670693+0000","last_peered":"2023-08-25T09:49:39.670693+0000","last_clean":"2023-08-25T09:49:39.670693+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:49:39.670693+0000","last_undegraded":"2023-08-25T09:49:39.670693+0000","last_fullsized":"2023-08-25T09:49:39.670693+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:53:52.223404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670888+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:49:39.670888+0000","last_peered":"2023-08-25T09:49:39.670888+0000","last_clean":"2023-08-25T09:49:39.670888+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:49:39.670888+0000","last_undegraded":"2023-08-25T09:49:39.670888+0000","last_fullsized":"2023-08-25T09:49:39.670888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:52.646017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669643+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:49:39.669643+0000","last_peered":"2023-08-25T09:49:39.669643+0000","last_clean":"2023-08-25T09:49:39.669643+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:49:39.669643+0000","last_undegraded":"2023-08-25T09:49:39.669643+0000","last_fullsized":"2023-08-25T09:49:39.669643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:07:22.260320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668612+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:49:39.668612+0000","last_peered":"2023-08-25T09:49:39.668612+0000","last_clean":"2023-08-25T09:49:39.668612+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:49:39.668612+0000","last_undegraded":"2023-08-25T09:49:39.668612+0000","last_fullsized":"2023-08-25T09:49:39.668612+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:48:39.386372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670019+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:49:39.670019+0000","last_peered":"2023-08-25T09:49:39.670019+0000","last_clean":"2023-08-25T09:49:39.670019+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:49:39.670019+0000","last_undegraded":"2023-08-25T09:49:39.670019+0000","last_fullsized":"2023-08-25T09:49:39.670019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:29.348922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610153+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:49:40.610153+0000","last_peered":"2023-08-25T09:49:40.610153+0000","last_clean":"2023-08-25T09:49:40.610153+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:49:40.610153+0000","last_undegraded":"2023-08-25T09:49:40.610153+0000","last_fullsized":"2023-08-25T09:49:40.610153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:50:39.530928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532880+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:49:40.532880+0000","last_peered":"2023-08-25T09:49:40.532880+0000","last_clean":"2023-08-25T09:49:40.532880+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:49:40.532880+0000","last_undegraded":"2023-08-25T09:49:40.532880+0000","last_fullsized":"2023-08-25T09:49:40.532880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:08:17.159830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670546+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:49:39.670546+0000","last_peered":"2023-08-25T09:49:39.670546+0000","last_clean":"2023-08-25T09:49:39.670546+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:49:39.670546+0000","last_undegraded":"2023-08-25T09:49:39.670546+0000","last_fullsized":"2023-08-25T09:49:39.670546+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:36.871310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610203+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:49:40.610203+0000","last_peered":"2023-08-25T09:49:40.610203+0000","last_clean":"2023-08-25T09:49:40.610203+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:49:40.610203+0000","last_undegraded":"2023-08-25T09:49:40.610203+0000","last_fullsized":"2023-08-25T09:49:40.610203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:13:22.282204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314057+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:49:40.314057+0000","last_peered":"2023-08-25T09:49:40.314057+0000","last_clean":"2023-08-25T09:49:40.314057+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:49:40.314057+0000","last_undegraded":"2023-08-25T09:49:40.314057+0000","last_fullsized":"2023-08-25T09:49:40.314057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:32:23.040483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668795+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:49:39.668795+0000","last_peered":"2023-08-25T09:49:39.668795+0000","last_clean":"2023-08-25T09:49:39.668795+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:49:39.668795+0000","last_undegraded":"2023-08-25T09:49:39.668795+0000","last_fullsized":"2023-08-25T09:49:39.668795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:21:47.452907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668565+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:49:39.668565+0000","last_peered":"2023-08-25T09:49:39.668565+0000","last_clean":"2023-08-25T09:49:39.668565+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:49:39.668565+0000","last_undegraded":"2023-08-25T09:49:39.668565+0000","last_fullsized":"2023-08-25T09:49:39.668565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:12.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313625+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:49:40.313625+0000","last_peered":"2023-08-25T09:49:40.313625+0000","last_clean":"2023-08-25T09:49:40.313625+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:49:40.313625+0000","last_undegraded":"2023-08-25T09:49:40.313625+0000","last_fullsized":"2023-08-25T09:49:40.313625+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:04:41.693477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609626+0000","last_change":"2023-08-25T09:49:34.628078+0000","last_active":"2023-08-25T09:49:40.609626+0000","last_peered":"2023-08-25T09:49:40.609626+0000","last_clean":"2023-08-25T09:49:40.609626+0000","last_became_active":"2023-08-25T09:49:34.627744+0000","last_became_peered":"2023-08-25T09:49:34.627744+0000","last_unstale":"2023-08-25T09:49:40.609626+0000","last_undegraded":"2023-08-25T09:49:40.609626+0000","last_fullsized":"2023-08-25T09:49:40.609626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:15.076203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671559+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:49:39.671559+0000","last_peered":"2023-08-25T09:49:39.671559+0000","last_clean":"2023-08-25T09:49:39.671559+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:49:39.671559+0000","last_undegraded":"2023-08-25T09:49:39.671559+0000","last_fullsized":"2023-08-25T09:49:39.671559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:50:31.566722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313898+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:49:40.313898+0000","last_peered":"2023-08-25T09:49:40.313898+0000","last_clean":"2023-08-25T09:49:40.313898+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:49:40.313898+0000","last_undegraded":"2023-08-25T09:49:40.313898+0000","last_fullsized":"2023-08-25T09:49:40.313898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:29:27.667548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668920+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:49:39.668920+0000","last_peered":"2023-08-25T09:49:39.668920+0000","last_clean":"2023-08-25T09:49:39.668920+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:49:39.668920+0000","last_undegraded":"2023-08-25T09:49:39.668920+0000","last_fullsized":"2023-08-25T09:49:39.668920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:58.529027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671825+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:49:39.671825+0000","last_peered":"2023-08-25T09:49:39.671825+0000","last_clean":"2023-08-25T09:49:39.671825+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:49:39.671825+0000","last_undegraded":"2023-08-25T09:49:39.671825+0000","last_fullsized":"2023-08-25T09:49:39.671825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:53:27.311764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313371+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:49:40.313371+0000","last_peered":"2023-08-25T09:49:40.313371+0000","last_clean":"2023-08-25T09:49:40.313371+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:49:40.313371+0000","last_undegraded":"2023-08-25T09:49:40.313371+0000","last_fullsized":"2023-08-25T09:49:40.313371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:23:01.195200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668159+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:49:39.668159+0000","last_peered":"2023-08-25T09:49:39.668159+0000","last_clean":"2023-08-25T09:49:39.668159+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:49:39.668159+0000","last_undegraded":"2023-08-25T09:49:39.668159+0000","last_fullsized":"2023-08-25T09:49:39.668159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:32:15.403697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672436+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:49:39.672436+0000","last_peered":"2023-08-25T09:49:39.672436+0000","last_clean":"2023-08-25T09:49:39.672436+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:49:39.672436+0000","last_undegraded":"2023-08-25T09:49:39.672436+0000","last_fullsized":"2023-08-25T09:49:39.672436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:53.368049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671121+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:49:39.671121+0000","last_peered":"2023-08-25T09:49:39.671121+0000","last_clean":"2023-08-25T09:49:39.671121+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:49:39.671121+0000","last_undegraded":"2023-08-25T09:49:39.671121+0000","last_fullsized":"2023-08-25T09:49:39.671121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:05:58.232199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671520+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:49:39.671520+0000","last_peered":"2023-08-25T09:49:39.671520+0000","last_clean":"2023-08-25T09:49:39.671520+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:49:39.671520+0000","last_undegraded":"2023-08-25T09:49:39.671520+0000","last_fullsized":"2023-08-25T09:49:39.671520+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:08:33.021697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"28'8","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609696+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:49:40.609696+0000","last_peered":"2023-08-25T09:49:40.609696+0000","last_clean":"2023-08-25T09:49:40.609696+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:49:40.609696+0000","last_undegraded":"2023-08-25T09:49:40.609696+0000","last_fullsized":"2023-08-25T09:49:40.609696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:16:55.451701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610899+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.610899+0000","last_peered":"2023-08-25T09:49:40.610899+0000","last_clean":"2023-08-25T09:49:40.610899+0000","last_became_active":"2023-08-25T09:49:34.630012+0000","last_became_peered":"2023-08-25T09:49:34.630012+0000","last_unstale":"2023-08-25T09:49:40.610899+0000","last_undegraded":"2023-08-25T09:49:40.610899+0000","last_fullsized":"2023-08-25T09:49:40.610899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:11:33.237689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610543+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:49:40.610543+0000","last_peered":"2023-08-25T09:49:40.610543+0000","last_clean":"2023-08-25T09:49:40.610543+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:49:40.610543+0000","last_undegraded":"2023-08-25T09:49:40.610543+0000","last_fullsized":"2023-08-25T09:49:40.610543+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:25.560317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671645+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:49:39.671645+0000","last_peered":"2023-08-25T09:49:39.671645+0000","last_clean":"2023-08-25T09:49:39.671645+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:49:39.671645+0000","last_undegraded":"2023-08-25T09:49:39.671645+0000","last_fullsized":"2023-08-25T09:49:39.671645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:37:05.937993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669858+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:49:39.669858+0000","last_peered":"2023-08-25T09:49:39.669858+0000","last_clean":"2023-08-25T09:49:39.669858+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:49:39.669858+0000","last_undegraded":"2023-08-25T09:49:39.669858+0000","last_fullsized":"2023-08-25T09:49:39.669858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:20:55.146183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668790+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:49:39.668790+0000","last_peered":"2023-08-25T09:49:39.668790+0000","last_clean":"2023-08-25T09:49:39.668790+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:49:39.668790+0000","last_undegraded":"2023-08-25T09:49:39.668790+0000","last_fullsized":"2023-08-25T09:49:39.668790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:51.399369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672435+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:49:39.672435+0000","last_peered":"2023-08-25T09:49:39.672435+0000","last_clean":"2023-08-25T09:49:39.672435+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:49:39.672435+0000","last_undegraded":"2023-08-25T09:49:39.672435+0000","last_fullsized":"2023-08-25T09:49:39.672435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:41:04.943976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609915+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:49:40.609915+0000","last_peered":"2023-08-25T09:49:40.609915+0000","last_clean":"2023-08-25T09:49:40.609915+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:49:40.609915+0000","last_undegraded":"2023-08-25T09:49:40.609915+0000","last_fullsized":"2023-08-25T09:49:40.609915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:30:03.672997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670253+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:49:39.670253+0000","last_peered":"2023-08-25T09:49:39.670253+0000","last_clean":"2023-08-25T09:49:39.670253+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:49:39.670253+0000","last_undegraded":"2023-08-25T09:49:39.670253+0000","last_fullsized":"2023-08-25T09:49:39.670253+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.806748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610312+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:49:40.610312+0000","last_peered":"2023-08-25T09:49:40.610312+0000","last_clean":"2023-08-25T09:49:40.610312+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:49:40.610312+0000","last_undegraded":"2023-08-25T09:49:40.610312+0000","last_fullsized":"2023-08-25T09:49:40.610312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:15:05.871192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672132+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:49:39.672132+0000","last_peered":"2023-08-25T09:49:39.672132+0000","last_clean":"2023-08-25T09:49:39.672132+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:49:39.672132+0000","last_undegraded":"2023-08-25T09:49:39.672132+0000","last_fullsized":"2023-08-25T09:49:39.672132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:51:00.162861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609435+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:49:40.609435+0000","last_peered":"2023-08-25T09:49:40.609435+0000","last_clean":"2023-08-25T09:49:40.609435+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:49:40.609435+0000","last_undegraded":"2023-08-25T09:49:40.609435+0000","last_fullsized":"2023-08-25T09:49:40.609435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:54:36.909125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609010+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:49:40.609010+0000","last_peered":"2023-08-25T09:49:40.609010+0000","last_clean":"2023-08-25T09:49:40.609010+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:49:40.609010+0000","last_undegraded":"2023-08-25T09:49:40.609010+0000","last_fullsized":"2023-08-25T09:49:40.609010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:45.655937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671033+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:49:39.671033+0000","last_peered":"2023-08-25T09:49:39.671033+0000","last_clean":"2023-08-25T09:49:39.671033+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:49:39.671033+0000","last_undegraded":"2023-08-25T09:49:39.671033+0000","last_fullsized":"2023-08-25T09:49:39.671033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:19:24.241659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609947+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:49:40.609947+0000","last_peered":"2023-08-25T09:49:40.609947+0000","last_clean":"2023-08-25T09:49:40.609947+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:49:40.609947+0000","last_undegraded":"2023-08-25T09:49:40.609947+0000","last_fullsized":"2023-08-25T09:49:40.609947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:27:06.196565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533764+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:49:40.533764+0000","last_peered":"2023-08-25T09:49:40.533764+0000","last_clean":"2023-08-25T09:49:40.533764+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:49:40.533764+0000","last_undegraded":"2023-08-25T09:49:40.533764+0000","last_fullsized":"2023-08-25T09:49:40.533764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:53:01.633281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672281+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:49:39.672281+0000","last_peered":"2023-08-25T09:49:39.672281+0000","last_clean":"2023-08-25T09:49:39.672281+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:49:39.672281+0000","last_undegraded":"2023-08-25T09:49:39.672281+0000","last_fullsized":"2023-08-25T09:49:39.672281+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:38:19.645378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533734+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:49:40.533734+0000","last_peered":"2023-08-25T09:49:40.533734+0000","last_clean":"2023-08-25T09:49:40.533734+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:49:40.533734+0000","last_undegraded":"2023-08-25T09:49:40.533734+0000","last_fullsized":"2023-08-25T09:49:40.533734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:27:54.228223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533904+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:49:40.533904+0000","last_peered":"2023-08-25T09:49:40.533904+0000","last_clean":"2023-08-25T09:49:40.533904+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:49:40.533904+0000","last_undegraded":"2023-08-25T09:49:40.533904+0000","last_fullsized":"2023-08-25T09:49:40.533904+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:06:19.107446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670060+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:49:39.670060+0000","last_peered":"2023-08-25T09:49:39.670060+0000","last_clean":"2023-08-25T09:49:39.670060+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:49:39.670060+0000","last_undegraded":"2023-08-25T09:49:39.670060+0000","last_fullsized":"2023-08-25T09:49:39.670060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:08:51.960006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608748+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:49:40.608748+0000","last_peered":"2023-08-25T09:49:40.608748+0000","last_clean":"2023-08-25T09:49:40.608748+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:49:40.608748+0000","last_undegraded":"2023-08-25T09:49:40.608748+0000","last_fullsized":"2023-08-25T09:49:40.608748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:35.777074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668458+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:49:39.668458+0000","last_peered":"2023-08-25T09:49:39.668458+0000","last_clean":"2023-08-25T09:49:39.668458+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:49:39.668458+0000","last_undegraded":"2023-08-25T09:49:39.668458+0000","last_fullsized":"2023-08-25T09:49:39.668458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:40:57.260235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669943+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:49:39.669943+0000","last_peered":"2023-08-25T09:49:39.669943+0000","last_clean":"2023-08-25T09:49:39.669943+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:49:39.669943+0000","last_undegraded":"2023-08-25T09:49:39.669943+0000","last_fullsized":"2023-08-25T09:49:39.669943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:42:36.500862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670365+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:49:39.670365+0000","last_peered":"2023-08-25T09:49:39.670365+0000","last_clean":"2023-08-25T09:49:39.670365+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:49:39.670365+0000","last_undegraded":"2023-08-25T09:49:39.670365+0000","last_fullsized":"2023-08-25T09:49:39.670365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:33:44.740556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314206+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:49:40.314206+0000","last_peered":"2023-08-25T09:49:40.314206+0000","last_clean":"2023-08-25T09:49:40.314206+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:49:40.314206+0000","last_undegraded":"2023-08-25T09:49:40.314206+0000","last_fullsized":"2023-08-25T09:49:40.314206+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:35.345143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670439+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:49:39.670439+0000","last_peered":"2023-08-25T09:49:39.670439+0000","last_clean":"2023-08-25T09:49:39.670439+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:49:39.670439+0000","last_undegraded":"2023-08-25T09:49:39.670439+0000","last_fullsized":"2023-08-25T09:49:39.670439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:38.142714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669800+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:49:39.669800+0000","last_peered":"2023-08-25T09:49:39.669800+0000","last_clean":"2023-08-25T09:49:39.669800+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:49:39.669800+0000","last_undegraded":"2023-08-25T09:49:39.669800+0000","last_fullsized":"2023-08-25T09:49:39.669800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:05:15.100928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668185+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:49:39.668185+0000","last_peered":"2023-08-25T09:49:39.668185+0000","last_clean":"2023-08-25T09:49:39.668185+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:49:39.668185+0000","last_undegraded":"2023-08-25T09:49:39.668185+0000","last_fullsized":"2023-08-25T09:49:39.668185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:52.824866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671435+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:49:39.671435+0000","last_peered":"2023-08-25T09:49:39.671435+0000","last_clean":"2023-08-25T09:49:39.671435+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:49:39.671435+0000","last_undegraded":"2023-08-25T09:49:39.671435+0000","last_fullsized":"2023-08-25T09:49:39.671435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:22:34.193671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668773+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:49:39.668773+0000","last_peered":"2023-08-25T09:49:39.668773+0000","last_clean":"2023-08-25T09:49:39.668773+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:49:39.668773+0000","last_undegraded":"2023-08-25T09:49:39.668773+0000","last_fullsized":"2023-08-25T09:49:39.668773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:38:09.260537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313744+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:49:40.313744+0000","last_peered":"2023-08-25T09:49:40.313744+0000","last_clean":"2023-08-25T09:49:40.313744+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:49:40.313744+0000","last_undegraded":"2023-08-25T09:49:40.313744+0000","last_fullsized":"2023-08-25T09:49:40.313744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:47:49.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314502+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:49:40.314502+0000","last_peered":"2023-08-25T09:49:40.314502+0000","last_clean":"2023-08-25T09:49:40.314502+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:49:40.314502+0000","last_undegraded":"2023-08-25T09:49:40.314502+0000","last_fullsized":"2023-08-25T09:49:40.314502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:52.308305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669862+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:49:39.669862+0000","last_peered":"2023-08-25T09:49:39.669862+0000","last_clean":"2023-08-25T09:49:39.669862+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:49:39.669862+0000","last_undegraded":"2023-08-25T09:49:39.669862+0000","last_fullsized":"2023-08-25T09:49:39.669862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:33:20.778440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669203+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:49:39.669203+0000","last_peered":"2023-08-25T09:49:39.669203+0000","last_clean":"2023-08-25T09:49:39.669203+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:49:39.669203+0000","last_undegraded":"2023-08-25T09:49:39.669203+0000","last_fullsized":"2023-08-25T09:49:39.669203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:14.357542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669995+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:49:39.669995+0000","last_peered":"2023-08-25T09:49:39.669995+0000","last_clean":"2023-08-25T09:49:39.669995+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:49:39.669995+0000","last_undegraded":"2023-08-25T09:49:39.669995+0000","last_fullsized":"2023-08-25T09:49:39.669995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:06.555517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668019+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:49:39.668019+0000","last_peered":"2023-08-25T09:49:39.668019+0000","last_clean":"2023-08-25T09:49:39.668019+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:49:39.668019+0000","last_undegraded":"2023-08-25T09:49:39.668019+0000","last_fullsized":"2023-08-25T09:49:39.668019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:35:29.581577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671260+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:49:39.671260+0000","last_peered":"2023-08-25T09:49:39.671260+0000","last_clean":"2023-08-25T09:49:39.671260+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:49:39.671260+0000","last_undegraded":"2023-08-25T09:49:39.671260+0000","last_fullsized":"2023-08-25T09:49:39.671260+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:43.256956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669816+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:49:39.669816+0000","last_peered":"2023-08-25T09:49:39.669816+0000","last_clean":"2023-08-25T09:49:39.669816+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:49:39.669816+0000","last_undegraded":"2023-08-25T09:49:39.669816+0000","last_fullsized":"2023-08-25T09:49:39.669816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:05:53.483156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669716+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:49:39.669716+0000","last_peered":"2023-08-25T09:49:39.669716+0000","last_clean":"2023-08-25T09:49:39.669716+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:49:39.669716+0000","last_undegraded":"2023-08-25T09:49:39.669716+0000","last_fullsized":"2023-08-25T09:49:39.669716+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:17.656387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609913+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:49:40.609913+0000","last_peered":"2023-08-25T09:49:40.609913+0000","last_clean":"2023-08-25T09:49:40.609913+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:49:40.609913+0000","last_undegraded":"2023-08-25T09:49:40.609913+0000","last_fullsized":"2023-08-25T09:49:40.609913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:04:46.102346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669727+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:49:39.669727+0000","last_peered":"2023-08-25T09:49:39.669727+0000","last_clean":"2023-08-25T09:49:39.669727+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:49:39.669727+0000","last_undegraded":"2023-08-25T09:49:39.669727+0000","last_fullsized":"2023-08-25T09:49:39.669727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:03:17.962039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668925+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:49:39.668925+0000","last_peered":"2023-08-25T09:49:39.668925+0000","last_clean":"2023-08-25T09:49:39.668925+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:49:39.668925+0000","last_undegraded":"2023-08-25T09:49:39.668925+0000","last_fullsized":"2023-08-25T09:49:39.668925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:15:02.299452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314453+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:49:40.314453+0000","last_peered":"2023-08-25T09:49:40.314453+0000","last_clean":"2023-08-25T09:49:40.314453+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:49:40.314453+0000","last_undegraded":"2023-08-25T09:49:40.314453+0000","last_fullsized":"2023-08-25T09:49:40.314453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:31.990094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":130128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.851}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":401408,"data_stored":720656,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]}]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.456}]}]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":315392,"data_stored":126883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":126321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":376832,"data_stored":717181,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:03.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.220+0000 7f690b5cf640 1 Processor -- start 2023-08-25T09:50:04.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.220+0000 7f690b5cf640 1 -- start start 2023-08-25T09:50:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.221+0000 7f690b5cf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69040c3a40 0x7f69040c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f690b5cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f690b5cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040c1dc0 con 0x7f69040c3660 2023-08-25T09:50:04.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f690b5cf640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904157a80 con 0x7f69040c3a40 2023-08-25T09:50:04.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f690b5cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6904151360 con 0x7f69041509d0 2023-08-25T09:50:04.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6908b6a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69040c3a40 0x7f69040c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6908b6a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69040c3a40 0x7f69040c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59962/0 (socket says 172.21.15.71:59962) 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6908b6a640 1 -- 172.21.15.71:0/102362188 learned_addr learned my addr 172.21.15.71:0/102362188 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6909b6c640 1 --2- 172.21.15.71:0/102362188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6908b6a640 1 -- 172.21.15.71:0/102362188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 msgr2=0x7f6904150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6908b6a640 1 --2- 172.21.15.71:0/102362188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.222+0000 7f6908b6a640 1 -- 172.21.15.71:0/102362188 >> v1:172.21.15.164:6789/0 conn(0x7f69040c3660 legacy=0x7f690406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.223+0000 7f6908b6a640 1 -- 172.21.15.71:0/102362188 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6904156e00 con 0x7f69040c3a40 2023-08-25T09:50:04.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.224+0000 7f6908b6a640 1 --2- 172.21.15.71:0/102362188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f69040c1710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009870 tx=0x7f68f402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e8667b399f36a2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.224+0000 7f68fbfff640 1 -- 172.21.15.71:0/102362188 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f402f840 con 0x7f69040c3a40 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.224+0000 7f68fbfff640 1 -- 172.21.15.71:0/102362188 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f402f9a0 con 0x7f69040c3a40 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.224+0000 7f68fbfff640 1 -- 172.21.15.71:0/102362188 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f402fd30 con 0x7f69040c3a40 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 -- 172.21.15.71:0/102362188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 msgr2=0x7f69040c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 --2- 172.21.15.71:0/102362188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f69040c1710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009870 tx=0x7f68f402ba20 comp rx=0 tx=0).stop 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 -- 172.21.15.71:0/102362188 shutdown_connections 2023-08-25T09:50:04.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 --2- 172.21.15.71:0/102362188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 --2- 172.21.15.71:0/102362188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f69040c1710 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 -- 172.21.15.71:0/102362188 >> 172.21.15.71:0/102362188 conn(0x7f69040654f0 msgr2=0x7f6904065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:04.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 -- 172.21.15.71:0/102362188 shutdown_connections 2023-08-25T09:50:04.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.225+0000 7f690b5cf640 1 -- 172.21.15.71:0/102362188 wait complete. 2023-08-25T09:50:04.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.227+0000 7f690b5cf640 1 Processor -- start 2023-08-25T09:50:04.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.227+0000 7f690b5cf640 1 -- start start 2023-08-25T09:50:04.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.228+0000 7f690b5cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69040c3660 0x7f69040c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.228+0000 7f690b5cf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f6904145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.228+0000 7f690b5cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.228+0000 7f690b5cf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904159580 con 0x7f69040c3a40 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.228+0000 7f690b5cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904157e20 con 0x7f69040c3660 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.228+0000 7f690b5cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6904156240 con 0x7f69041509d0 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f6904145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f6904145f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59964/0 (socket says 172.21.15.71:59964) 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 -- 172.21.15.71:0/2623064041 learned_addr learned my addr 172.21.15.71:0/2623064041 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:04.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f690936b640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69040c3660 0x7f69040c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6909b6c640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 -- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 msgr2=0x7f6904146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904146460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 -- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69040c3660 msgr2=0x7f69040c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69040c3660 0x7f69040c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.229+0000 7f6908b6a640 1 -- 172.21.15.71:0/2623064041 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f40094d0 con 0x7f69040c3a40 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.230+0000 7f6908b6a640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f6904145f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f68f4000c00 tx=0x7f68f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:04.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.230+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4037c40 con 0x7f69040c3a40 2023-08-25T09:50:04.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.230+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f4037da0 con 0x7f69040c3a40 2023-08-25T09:50:04.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.230+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4041640 con 0x7f69040c3a40 2023-08-25T09:50:04.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.230+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69041580c0 con 0x7f69040c3a40 2023-08-25T09:50:04.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.230+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690406fd70 con 0x7f69040c3a40 2023-08-25T09:50:04.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.231+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040c29c0 con 0x7f69040c3a40 2023-08-25T09:50:04.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.232+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f68f40417a0 con 0x7f69040c3a40 2023-08-25T09:50:04.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.233+0000 7f68f9ffb640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f68e806a080 0x7f68e806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.233+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f68f403c070 con 0x7f69040c3a40 2023-08-25T09:50:04.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.233+0000 7f690936b640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f68e806a080 0x7f68e806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.234+0000 7f690936b640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f68e806a080 0x7f68e806c540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6900000f60 tx=0x7f6900009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:04.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.237+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f68f4076930 con 0x7f69040c3a40 2023-08-25T09:50:04.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.425+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f69041573c0 con 0x7f68e806a080 2023-08-25T09:50:04.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.439+0000 7f68f9ffb640 1 -- 172.21.15.71:0/2623064041 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367292 (secure 0 0 0) 0x7f69000012f0 con 0x7f68e806a080 2023-08-25T09:50:04.442 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:04.446 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:04.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f68e806a080 msgr2=0x7f68e806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f68e806a080 0x7f68e806c540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6900000f60 tx=0x7f6900009040 comp rx=0 tx=0).stop 2023-08-25T09:50:04.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 msgr2=0x7f6904145f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f6904145f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f68f4000c00 tx=0x7f68f4004730 comp rx=0 tx=0).stop 2023-08-25T09:50:04.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 shutdown_connections 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f68e806a080 0x7f68e806c540 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69041509d0 0x7f6904146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69040c3a40 0x7f6904145f20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 --2- 172.21.15.71:0/2623064041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69040c3660 0x7f69040c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 >> 172.21.15.71:0/2623064041 conn(0x7f69040654f0 msgr2=0x7f69040c92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 shutdown_connections 2023-08-25T09:50:04.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.446+0000 7f690b5cf640 1 -- 172.21.15.71:0/2623064041 wait complete. 2023-08-25T09:50:04.467 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2023-08-25T09:50:02.643913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595007,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227080,"kb_used_data":2592,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747640,"statfs":{"total":773094113280,"available":772861583360,"internally_reserved":0,"allocated":2654208,"data_stored":2199862,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004678"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532259+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:49:40.532259+0000","last_peered":"2023-08-25T09:49:40.532259+0000","last_clean":"2023-08-25T09:49:40.532259+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:49:40.532259+0000","last_undegraded":"2023-08-25T09:49:40.532259+0000","last_fullsized":"2023-08-25T09:49:40.532259+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:08:44.154380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670137+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:49:39.670137+0000","last_peered":"2023-08-25T09:49:39.670137+0000","last_clean":"2023-08-25T09:49:39.670137+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:49:39.670137+0000","last_undegraded":"2023-08-25T09:49:39.670137+0000","last_fullsized":"2023-08-25T09:49:39.670137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:19:21.160750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672075+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:49:39.672075+0000","last_peered":"2023-08-25T09:49:39.672075+0000","last_clean":"2023-08-25T09:49:39.672075+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:49:39.672075+0000","last_undegraded":"2023-08-25T09:49:39.672075+0000","last_fullsized":"2023-08-25T09:49:39.672075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:39:44.896839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.667983+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:49:39.667983+0000","last_peered":"2023-08-25T09:49:39.667983+0000","last_clean":"2023-08-25T09:49:39.667983+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:49:39.667983+0000","last_undegraded":"2023-08-25T09:49:39.667983+0000","last_fullsized":"2023-08-25T09:49:39.667983+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:52:21.149049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610317+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:49:40.610317+0000","last_peered":"2023-08-25T09:49:40.610317+0000","last_clean":"2023-08-25T09:49:40.610317+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:49:40.610317+0000","last_undegraded":"2023-08-25T09:49:40.610317+0000","last_fullsized":"2023-08-25T09:49:40.610317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.067887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609551+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:49:40.609551+0000","last_peered":"2023-08-25T09:49:40.609551+0000","last_clean":"2023-08-25T09:49:40.609551+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:49:40.609551+0000","last_undegraded":"2023-08-25T09:49:40.609551+0000","last_fullsized":"2023-08-25T09:49:40.609551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:31:30.676338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669356+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:49:39.669356+0000","last_peered":"2023-08-25T09:49:39.669356+0000","last_clean":"2023-08-25T09:49:39.669356+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:49:39.669356+0000","last_undegraded":"2023-08-25T09:49:39.669356+0000","last_fullsized":"2023-08-25T09:49:39.669356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:52:41.076094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672443+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:49:39.672443+0000","last_peered":"2023-08-25T09:49:39.672443+0000","last_clean":"2023-08-25T09:49:39.672443+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:49:39.672443+0000","last_undegraded":"2023-08-25T09:49:39.672443+0000","last_fullsized":"2023-08-25T09:49:39.672443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:41:06.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670837+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:49:39.670837+0000","last_peered":"2023-08-25T09:49:39.670837+0000","last_clean":"2023-08-25T09:49:39.670837+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:49:39.670837+0000","last_undegraded":"2023-08-25T09:49:39.670837+0000","last_fullsized":"2023-08-25T09:49:39.670837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:43.561619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669087+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:49:39.669087+0000","last_peered":"2023-08-25T09:49:39.669087+0000","last_clean":"2023-08-25T09:49:39.669087+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:49:39.669087+0000","last_undegraded":"2023-08-25T09:49:39.669087+0000","last_fullsized":"2023-08-25T09:49:39.669087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:29:31.719174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668364+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:49:39.668364+0000","last_peered":"2023-08-25T09:49:39.668364+0000","last_clean":"2023-08-25T09:49:39.668364+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:49:39.668364+0000","last_undegraded":"2023-08-25T09:49:39.668364+0000","last_fullsized":"2023-08-25T09:49:39.668364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:54:21.024745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669503+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:49:39.669503+0000","last_peered":"2023-08-25T09:49:39.669503+0000","last_clean":"2023-08-25T09:49:39.669503+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:49:39.669503+0000","last_undegraded":"2023-08-25T09:49:39.669503+0000","last_fullsized":"2023-08-25T09:49:39.669503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:38.132579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533392+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:49:40.533392+0000","last_peered":"2023-08-25T09:49:40.533392+0000","last_clean":"2023-08-25T09:49:40.533392+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:49:40.533392+0000","last_undegraded":"2023-08-25T09:49:40.533392+0000","last_fullsized":"2023-08-25T09:49:40.533392+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:44:04.188030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669599+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:49:39.669599+0000","last_peered":"2023-08-25T09:49:39.669599+0000","last_clean":"2023-08-25T09:49:39.669599+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:49:39.669599+0000","last_undegraded":"2023-08-25T09:49:39.669599+0000","last_fullsized":"2023-08-25T09:49:39.669599+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:36:03.677003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314444+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:49:40.314444+0000","last_peered":"2023-08-25T09:49:40.314444+0000","last_clean":"2023-08-25T09:49:40.314444+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:49:40.314444+0000","last_undegraded":"2023-08-25T09:49:40.314444+0000","last_fullsized":"2023-08-25T09:49:40.314444+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:24:33.940678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670152+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:49:39.670152+0000","last_peered":"2023-08-25T09:49:39.670152+0000","last_clean":"2023-08-25T09:49:39.670152+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:49:39.670152+0000","last_undegraded":"2023-08-25T09:49:39.670152+0000","last_fullsized":"2023-08-25T09:49:39.670152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:04.589805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671210+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:49:39.671210+0000","last_peered":"2023-08-25T09:49:39.671210+0000","last_clean":"2023-08-25T09:49:39.671210+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:49:39.671210+0000","last_undegraded":"2023-08-25T09:49:39.671210+0000","last_fullsized":"2023-08-25T09:49:39.671210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:21.872702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610243+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:49:40.610243+0000","last_peered":"2023-08-25T09:49:40.610243+0000","last_clean":"2023-08-25T09:49:40.610243+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:49:40.610243+0000","last_undegraded":"2023-08-25T09:49:40.610243+0000","last_fullsized":"2023-08-25T09:49:40.610243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:40:11.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610311+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:49:40.610311+0000","last_peered":"2023-08-25T09:49:40.610311+0000","last_clean":"2023-08-25T09:49:40.610311+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:49:40.610311+0000","last_undegraded":"2023-08-25T09:49:40.610311+0000","last_fullsized":"2023-08-25T09:49:40.610311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:24:36.237719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532845+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:49:40.532845+0000","last_peered":"2023-08-25T09:49:40.532845+0000","last_clean":"2023-08-25T09:49:40.532845+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:49:40.532845+0000","last_undegraded":"2023-08-25T09:49:40.532845+0000","last_fullsized":"2023-08-25T09:49:40.532845+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:17.775607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669479+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:49:39.669479+0000","last_peered":"2023-08-25T09:49:39.669479+0000","last_clean":"2023-08-25T09:49:39.669479+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:49:39.669479+0000","last_undegraded":"2023-08-25T09:49:39.669479+0000","last_fullsized":"2023-08-25T09:49:39.669479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:24:10.790283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669214+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:49:39.669214+0000","last_peered":"2023-08-25T09:49:39.669214+0000","last_clean":"2023-08-25T09:49:39.669214+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:49:39.669214+0000","last_undegraded":"2023-08-25T09:49:39.669214+0000","last_fullsized":"2023-08-25T09:49:39.669214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:45:44.040670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609056+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:49:40.609056+0000","last_peered":"2023-08-25T09:49:40.609056+0000","last_clean":"2023-08-25T09:49:40.609056+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:49:40.609056+0000","last_undegraded":"2023-08-25T09:49:40.609056+0000","last_fullsized":"2023-08-25T09:49:40.609056+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:30:29.949537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610535+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:49:40.610535+0000","last_peered":"2023-08-25T09:49:40.610535+0000","last_clean":"2023-08-25T09:49:40.610535+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:49:40.610535+0000","last_undegraded":"2023-08-25T09:49:40.610535+0000","last_fullsized":"2023-08-25T09:49:40.610535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:33:47.145180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608671+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.608671+0000","last_peered":"2023-08-25T09:49:40.608671+0000","last_clean":"2023-08-25T09:49:40.608671+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:49:40.608671+0000","last_undegraded":"2023-08-25T09:49:40.608671+0000","last_fullsized":"2023-08-25T09:49:40.608671+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:49:14.644166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670296+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:49:39.670296+0000","last_peered":"2023-08-25T09:49:39.670296+0000","last_clean":"2023-08-25T09:49:39.670296+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:49:39.670296+0000","last_undegraded":"2023-08-25T09:49:39.670296+0000","last_fullsized":"2023-08-25T09:49:39.670296+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:33:42.072752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609238+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:49:40.609238+0000","last_peered":"2023-08-25T09:49:40.609238+0000","last_clean":"2023-08-25T09:49:40.609238+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:49:40.609238+0000","last_undegraded":"2023-08-25T09:49:40.609238+0000","last_fullsized":"2023-08-25T09:49:40.609238+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:01:49.724298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609229+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:49:40.609229+0000","last_peered":"2023-08-25T09:49:40.609229+0000","last_clean":"2023-08-25T09:49:40.609229+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:49:40.609229+0000","last_undegraded":"2023-08-25T09:49:40.609229+0000","last_fullsized":"2023-08-25T09:49:40.609229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:35:14.991310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669355+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:49:39.669355+0000","last_peered":"2023-08-25T09:49:39.669355+0000","last_clean":"2023-08-25T09:49:39.669355+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:49:39.669355+0000","last_undegraded":"2023-08-25T09:49:39.669355+0000","last_fullsized":"2023-08-25T09:49:39.669355+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:36:04.594929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669714+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:49:39.669714+0000","last_peered":"2023-08-25T09:49:39.669714+0000","last_clean":"2023-08-25T09:49:39.669714+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:49:39.669714+0000","last_undegraded":"2023-08-25T09:49:39.669714+0000","last_fullsized":"2023-08-25T09:49:39.669714+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:13:29.132994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"28'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671383+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:49:39.671383+0000","last_peered":"2023-08-25T09:49:39.671383+0000","last_clean":"2023-08-25T09:49:39.671383+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:49:39.671383+0000","last_undegraded":"2023-08-25T09:49:39.671383+0000","last_fullsized":"2023-08-25T09:49:39.671383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:08:04.314595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533055+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:49:40.533055+0000","last_peered":"2023-08-25T09:49:40.533055+0000","last_clean":"2023-08-25T09:49:40.533055+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:49:40.533055+0000","last_undegraded":"2023-08-25T09:49:40.533055+0000","last_fullsized":"2023-08-25T09:49:40.533055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:29.319338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609374+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:49:40.609374+0000","last_peered":"2023-08-25T09:49:40.609374+0000","last_clean":"2023-08-25T09:49:40.609374+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:49:40.609374+0000","last_undegraded":"2023-08-25T09:49:40.609374+0000","last_fullsized":"2023-08-25T09:49:40.609374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:11:38.846168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"29'10","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:57.914119+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:49:57.914119+0000","last_peered":"2023-08-25T09:49:57.914119+0000","last_clean":"2023-08-25T09:49:57.914119+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:49:57.914119+0000","last_undegraded":"2023-08-25T09:49:57.914119+0000","last_fullsized":"2023-08-25T09:49:57.914119+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:19:49.118130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669680+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:49:39.669680+0000","last_peered":"2023-08-25T09:49:39.669680+0000","last_clean":"2023-08-25T09:49:39.669680+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:49:39.669680+0000","last_undegraded":"2023-08-25T09:49:39.669680+0000","last_fullsized":"2023-08-25T09:49:39.669680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:15:08.913052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532469+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:49:40.532469+0000","last_peered":"2023-08-25T09:49:40.532469+0000","last_clean":"2023-08-25T09:49:40.532469+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:49:40.532469+0000","last_undegraded":"2023-08-25T09:49:40.532469+0000","last_fullsized":"2023-08-25T09:49:40.532469+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:54:59.656364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669355+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:49:39.669355+0000","last_peered":"2023-08-25T09:49:39.669355+0000","last_clean":"2023-08-25T09:49:39.669355+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:49:39.669355+0000","last_undegraded":"2023-08-25T09:49:39.669355+0000","last_fullsized":"2023-08-25T09:49:39.669355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:48.989731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610694+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:49:40.610694+0000","last_peered":"2023-08-25T09:49:40.610694+0000","last_clean":"2023-08-25T09:49:40.610694+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:49:40.610694+0000","last_undegraded":"2023-08-25T09:49:40.610694+0000","last_fullsized":"2023-08-25T09:49:40.610694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:46.063058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670003+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:49:39.670003+0000","last_peered":"2023-08-25T09:49:39.670003+0000","last_clean":"2023-08-25T09:49:39.670003+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:49:39.670003+0000","last_undegraded":"2023-08-25T09:49:39.670003+0000","last_fullsized":"2023-08-25T09:49:39.670003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:13:11.987428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608483+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:49:40.608483+0000","last_peered":"2023-08-25T09:49:40.608483+0000","last_clean":"2023-08-25T09:49:40.608483+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:49:40.608483+0000","last_undegraded":"2023-08-25T09:49:40.608483+0000","last_fullsized":"2023-08-25T09:49:40.608483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:28:33.537079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608779+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:49:40.608779+0000","last_peered":"2023-08-25T09:49:40.608779+0000","last_clean":"2023-08-25T09:49:40.608779+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:49:40.608779+0000","last_undegraded":"2023-08-25T09:49:40.608779+0000","last_fullsized":"2023-08-25T09:49:40.608779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:01:10.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.534056+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:49:40.534056+0000","last_peered":"2023-08-25T09:49:40.534056+0000","last_clean":"2023-08-25T09:49:40.534056+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:49:40.534056+0000","last_undegraded":"2023-08-25T09:49:40.534056+0000","last_fullsized":"2023-08-25T09:49:40.534056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:30.208912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.611163+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:49:40.611163+0000","last_peered":"2023-08-25T09:49:40.611163+0000","last_clean":"2023-08-25T09:49:40.611163+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:49:40.611163+0000","last_undegraded":"2023-08-25T09:49:40.611163+0000","last_fullsized":"2023-08-25T09:49:40.611163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:46:45.992240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668225+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:49:39.668225+0000","last_peered":"2023-08-25T09:49:39.668225+0000","last_clean":"2023-08-25T09:49:39.668225+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:49:39.668225+0000","last_undegraded":"2023-08-25T09:49:39.668225+0000","last_fullsized":"2023-08-25T09:49:39.668225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:00:13.974928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669196+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:49:39.669196+0000","last_peered":"2023-08-25T09:49:39.669196+0000","last_clean":"2023-08-25T09:49:39.669196+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:49:39.669196+0000","last_undegraded":"2023-08-25T09:49:39.669196+0000","last_fullsized":"2023-08-25T09:49:39.669196+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:28:03.992015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671806+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:49:39.671806+0000","last_peered":"2023-08-25T09:49:39.671806+0000","last_clean":"2023-08-25T09:49:39.671806+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:49:39.671806+0000","last_undegraded":"2023-08-25T09:49:39.671806+0000","last_fullsized":"2023-08-25T09:49:39.671806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:14:52.844903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533580+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:49:40.533580+0000","last_peered":"2023-08-25T09:49:40.533580+0000","last_clean":"2023-08-25T09:49:40.533580+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:49:40.533580+0000","last_undegraded":"2023-08-25T09:49:40.533580+0000","last_fullsized":"2023-08-25T09:49:40.533580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:00.063674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609540+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:49:40.609540+0000","last_peered":"2023-08-25T09:49:40.609540+0000","last_clean":"2023-08-25T09:49:40.609540+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:49:40.609540+0000","last_undegraded":"2023-08-25T09:49:40.609540+0000","last_fullsized":"2023-08-25T09:49:40.609540+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:59:18.199533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532919+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:49:40.532919+0000","last_peered":"2023-08-25T09:49:40.532919+0000","last_clean":"2023-08-25T09:49:40.532919+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:49:40.532919+0000","last_undegraded":"2023-08-25T09:49:40.532919+0000","last_fullsized":"2023-08-25T09:49:40.532919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:00:24.110176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533867+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:49:40.533867+0000","last_peered":"2023-08-25T09:49:40.533867+0000","last_clean":"2023-08-25T09:49:40.533867+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:49:40.533867+0000","last_undegraded":"2023-08-25T09:49:40.533867+0000","last_fullsized":"2023-08-25T09:49:40.533867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:25:26.896240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672580+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:49:39.672580+0000","last_peered":"2023-08-25T09:49:39.672580+0000","last_clean":"2023-08-25T09:49:39.672580+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:49:39.672580+0000","last_undegraded":"2023-08-25T09:49:39.672580+0000","last_fullsized":"2023-08-25T09:49:39.672580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:23.747043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668857+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:49:39.668857+0000","last_peered":"2023-08-25T09:49:39.668857+0000","last_clean":"2023-08-25T09:49:39.668857+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:49:39.668857+0000","last_undegraded":"2023-08-25T09:49:39.668857+0000","last_fullsized":"2023-08-25T09:49:39.668857+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:12:48.827644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533435+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:49:40.533435+0000","last_peered":"2023-08-25T09:49:40.533435+0000","last_clean":"2023-08-25T09:49:40.533435+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:49:40.533435+0000","last_undegraded":"2023-08-25T09:49:40.533435+0000","last_fullsized":"2023-08-25T09:49:40.533435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:07:36.521502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669650+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:49:39.669650+0000","last_peered":"2023-08-25T09:49:39.669650+0000","last_clean":"2023-08-25T09:49:39.669650+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:49:39.669650+0000","last_undegraded":"2023-08-25T09:49:39.669650+0000","last_fullsized":"2023-08-25T09:49:39.669650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:26:07.776398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668924+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:49:39.668924+0000","last_peered":"2023-08-25T09:49:39.668924+0000","last_clean":"2023-08-25T09:49:39.668924+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:49:39.668924+0000","last_undegraded":"2023-08-25T09:49:39.668924+0000","last_fullsized":"2023-08-25T09:49:39.668924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:22:13.379657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669047+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:49:39.669047+0000","last_peered":"2023-08-25T09:49:39.669047+0000","last_clean":"2023-08-25T09:49:39.669047+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:49:39.669047+0000","last_undegraded":"2023-08-25T09:49:39.669047+0000","last_fullsized":"2023-08-25T09:49:39.669047+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:36:02.088533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533994+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:49:40.533994+0000","last_peered":"2023-08-25T09:49:40.533994+0000","last_clean":"2023-08-25T09:49:40.533994+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:49:40.533994+0000","last_undegraded":"2023-08-25T09:49:40.533994+0000","last_fullsized":"2023-08-25T09:49:40.533994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:34:02.269298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532502+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:49:40.532502+0000","last_peered":"2023-08-25T09:49:40.532502+0000","last_clean":"2023-08-25T09:49:40.532502+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:49:40.532502+0000","last_undegraded":"2023-08-25T09:49:40.532502+0000","last_fullsized":"2023-08-25T09:49:40.532502+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:52:34.012911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313601+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:49:40.313601+0000","last_peered":"2023-08-25T09:49:40.313601+0000","last_clean":"2023-08-25T09:49:40.313601+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:49:40.313601+0000","last_undegraded":"2023-08-25T09:49:40.313601+0000","last_fullsized":"2023-08-25T09:49:40.313601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:28.801938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314730+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:49:40.314730+0000","last_peered":"2023-08-25T09:49:40.314730+0000","last_clean":"2023-08-25T09:49:40.314730+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:49:40.314730+0000","last_undegraded":"2023-08-25T09:49:40.314730+0000","last_fullsized":"2023-08-25T09:49:40.314730+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:17:34.425122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669544+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:49:39.669544+0000","last_peered":"2023-08-25T09:49:39.669544+0000","last_clean":"2023-08-25T09:49:39.669544+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:49:39.669544+0000","last_undegraded":"2023-08-25T09:49:39.669544+0000","last_fullsized":"2023-08-25T09:49:39.669544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:44.776136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668099+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:49:39.668099+0000","last_peered":"2023-08-25T09:49:39.668099+0000","last_clean":"2023-08-25T09:49:39.668099+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:49:39.668099+0000","last_undegraded":"2023-08-25T09:49:39.668099+0000","last_fullsized":"2023-08-25T09:49:39.668099+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:08.959803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670236+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:49:39.670236+0000","last_peered":"2023-08-25T09:49:39.670236+0000","last_clean":"2023-08-25T09:49:39.670236+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:49:39.670236+0000","last_undegraded":"2023-08-25T09:49:39.670236+0000","last_fullsized":"2023-08-25T09:49:39.670236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:04.664807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668818+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:49:39.668818+0000","last_peered":"2023-08-25T09:49:39.668818+0000","last_clean":"2023-08-25T09:49:39.668818+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:49:39.668818+0000","last_undegraded":"2023-08-25T09:49:39.668818+0000","last_fullsized":"2023-08-25T09:49:39.668818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:45:13.832806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610040+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:49:40.610040+0000","last_peered":"2023-08-25T09:49:40.610040+0000","last_clean":"2023-08-25T09:49:40.610040+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:49:40.610040+0000","last_undegraded":"2023-08-25T09:49:40.610040+0000","last_fullsized":"2023-08-25T09:49:40.610040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:08.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609484+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:49:40.609484+0000","last_peered":"2023-08-25T09:49:40.609484+0000","last_clean":"2023-08-25T09:49:40.609484+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:49:40.609484+0000","last_undegraded":"2023-08-25T09:49:40.609484+0000","last_fullsized":"2023-08-25T09:49:40.609484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:02.892745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671904+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:49:39.671904+0000","last_peered":"2023-08-25T09:49:39.671904+0000","last_clean":"2023-08-25T09:49:39.671904+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:49:39.671904+0000","last_undegraded":"2023-08-25T09:49:39.671904+0000","last_fullsized":"2023-08-25T09:49:39.671904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:12:08.551735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":183,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670426+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:49:39.670426+0000","last_peered":"2023-08-25T09:49:39.670426+0000","last_clean":"2023-08-25T09:49:39.670426+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:49:39.670426+0000","last_undegraded":"2023-08-25T09:49:39.670426+0000","last_fullsized":"2023-08-25T09:49:39.670426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:14:32.439794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670825+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:49:39.670825+0000","last_peered":"2023-08-25T09:49:39.670825+0000","last_clean":"2023-08-25T09:49:39.670825+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:49:39.670825+0000","last_undegraded":"2023-08-25T09:49:39.670825+0000","last_fullsized":"2023-08-25T09:49:39.670825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:14:42.235888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533862+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:49:40.533862+0000","last_peered":"2023-08-25T09:49:40.533862+0000","last_clean":"2023-08-25T09:49:40.533862+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:49:40.533862+0000","last_undegraded":"2023-08-25T09:49:40.533862+0000","last_fullsized":"2023-08-25T09:49:40.533862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:47:24.625104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609113+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:49:40.609113+0000","last_peered":"2023-08-25T09:49:40.609113+0000","last_clean":"2023-08-25T09:49:40.609113+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:49:40.609113+0000","last_undegraded":"2023-08-25T09:49:40.609113+0000","last_fullsized":"2023-08-25T09:49:40.609113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:34:01.834958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670693+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:49:39.670693+0000","last_peered":"2023-08-25T09:49:39.670693+0000","last_clean":"2023-08-25T09:49:39.670693+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:49:39.670693+0000","last_undegraded":"2023-08-25T09:49:39.670693+0000","last_fullsized":"2023-08-25T09:49:39.670693+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:53:52.223404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670888+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:49:39.670888+0000","last_peered":"2023-08-25T09:49:39.670888+0000","last_clean":"2023-08-25T09:49:39.670888+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:49:39.670888+0000","last_undegraded":"2023-08-25T09:49:39.670888+0000","last_fullsized":"2023-08-25T09:49:39.670888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:52.646017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669643+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:49:39.669643+0000","last_peered":"2023-08-25T09:49:39.669643+0000","last_clean":"2023-08-25T09:49:39.669643+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:49:39.669643+0000","last_undegraded":"2023-08-25T09:49:39.669643+0000","last_fullsized":"2023-08-25T09:49:39.669643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:07:22.260320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668612+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:49:39.668612+0000","last_peered":"2023-08-25T09:49:39.668612+0000","last_clean":"2023-08-25T09:49:39.668612+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:49:39.668612+0000","last_undegraded":"2023-08-25T09:49:39.668612+0000","last_fullsized":"2023-08-25T09:49:39.668612+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:48:39.386372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670019+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:49:39.670019+0000","last_peered":"2023-08-25T09:49:39.670019+0000","last_clean":"2023-08-25T09:49:39.670019+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:49:39.670019+0000","last_undegraded":"2023-08-25T09:49:39.670019+0000","last_fullsized":"2023-08-25T09:49:39.670019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:29.348922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610153+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:49:40.610153+0000","last_peered":"2023-08-25T09:49:40.610153+0000","last_clean":"2023-08-25T09:49:40.610153+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:49:40.610153+0000","last_undegraded":"2023-08-25T09:49:40.610153+0000","last_fullsized":"2023-08-25T09:49:40.610153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:50:39.530928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532880+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:49:40.532880+0000","last_peered":"2023-08-25T09:49:40.532880+0000","last_clean":"2023-08-25T09:49:40.532880+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:49:40.532880+0000","last_undegraded":"2023-08-25T09:49:40.532880+0000","last_fullsized":"2023-08-25T09:49:40.532880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:08:17.159830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670546+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:49:39.670546+0000","last_peered":"2023-08-25T09:49:39.670546+0000","last_clean":"2023-08-25T09:49:39.670546+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:49:39.670546+0000","last_undegraded":"2023-08-25T09:49:39.670546+0000","last_fullsized":"2023-08-25T09:49:39.670546+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:36.871310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610203+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:49:40.610203+0000","last_peered":"2023-08-25T09:49:40.610203+0000","last_clean":"2023-08-25T09:49:40.610203+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:49:40.610203+0000","last_undegraded":"2023-08-25T09:49:40.610203+0000","last_fullsized":"2023-08-25T09:49:40.610203+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:13:22.282204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314057+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:49:40.314057+0000","last_peered":"2023-08-25T09:49:40.314057+0000","last_clean":"2023-08-25T09:49:40.314057+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:49:40.314057+0000","last_undegraded":"2023-08-25T09:49:40.314057+0000","last_fullsized":"2023-08-25T09:49:40.314057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:32:23.040483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668795+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:49:39.668795+0000","last_peered":"2023-08-25T09:49:39.668795+0000","last_clean":"2023-08-25T09:49:39.668795+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:49:39.668795+0000","last_undegraded":"2023-08-25T09:49:39.668795+0000","last_fullsized":"2023-08-25T09:49:39.668795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:21:47.452907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668565+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:49:39.668565+0000","last_peered":"2023-08-25T09:49:39.668565+0000","last_clean":"2023-08-25T09:49:39.668565+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:49:39.668565+0000","last_undegraded":"2023-08-25T09:49:39.668565+0000","last_fullsized":"2023-08-25T09:49:39.668565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:12.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313625+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:49:40.313625+0000","last_peered":"2023-08-25T09:49:40.313625+0000","last_clean":"2023-08-25T09:49:40.313625+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:49:40.313625+0000","last_undegraded":"2023-08-25T09:49:40.313625+0000","last_fullsized":"2023-08-25T09:49:40.313625+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:04:41.693477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609626+0000","last_change":"2023-08-25T09:49:34.628078+0000","last_active":"2023-08-25T09:49:40.609626+0000","last_peered":"2023-08-25T09:49:40.609626+0000","last_clean":"2023-08-25T09:49:40.609626+0000","last_became_active":"2023-08-25T09:49:34.627744+0000","last_became_peered":"2023-08-25T09:49:34.627744+0000","last_unstale":"2023-08-25T09:49:40.609626+0000","last_undegraded":"2023-08-25T09:49:40.609626+0000","last_fullsized":"2023-08-25T09:49:40.609626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:15.076203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671559+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:49:39.671559+0000","last_peered":"2023-08-25T09:49:39.671559+0000","last_clean":"2023-08-25T09:49:39.671559+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:49:39.671559+0000","last_undegraded":"2023-08-25T09:49:39.671559+0000","last_fullsized":"2023-08-25T09:49:39.671559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:50:31.566722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313898+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:49:40.313898+0000","last_peered":"2023-08-25T09:49:40.313898+0000","last_clean":"2023-08-25T09:49:40.313898+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:49:40.313898+0000","last_undegraded":"2023-08-25T09:49:40.313898+0000","last_fullsized":"2023-08-25T09:49:40.313898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:29:27.667548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668920+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:49:39.668920+0000","last_peered":"2023-08-25T09:49:39.668920+0000","last_clean":"2023-08-25T09:49:39.668920+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:49:39.668920+0000","last_undegraded":"2023-08-25T09:49:39.668920+0000","last_fullsized":"2023-08-25T09:49:39.668920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:58.529027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671825+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:49:39.671825+0000","last_peered":"2023-08-25T09:49:39.671825+0000","last_clean":"2023-08-25T09:49:39.671825+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:49:39.671825+0000","last_undegraded":"2023-08-25T09:49:39.671825+0000","last_fullsized":"2023-08-25T09:49:39.671825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:53:27.311764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313371+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:49:40.313371+0000","last_peered":"2023-08-25T09:49:40.313371+0000","last_clean":"2023-08-25T09:49:40.313371+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:49:40.313371+0000","last_undegraded":"2023-08-25T09:49:40.313371+0000","last_fullsized":"2023-08-25T09:49:40.313371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:23:01.195200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668159+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:49:39.668159+0000","last_peered":"2023-08-25T09:49:39.668159+0000","last_clean":"2023-08-25T09:49:39.668159+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:49:39.668159+0000","last_undegraded":"2023-08-25T09:49:39.668159+0000","last_fullsized":"2023-08-25T09:49:39.668159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:32:15.403697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672436+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:49:39.672436+0000","last_peered":"2023-08-25T09:49:39.672436+0000","last_clean":"2023-08-25T09:49:39.672436+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:49:39.672436+0000","last_undegraded":"2023-08-25T09:49:39.672436+0000","last_fullsized":"2023-08-25T09:49:39.672436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:53.368049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671121+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:49:39.671121+0000","last_peered":"2023-08-25T09:49:39.671121+0000","last_clean":"2023-08-25T09:49:39.671121+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:49:39.671121+0000","last_undegraded":"2023-08-25T09:49:39.671121+0000","last_fullsized":"2023-08-25T09:49:39.671121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:05:58.232199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671520+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:49:39.671520+0000","last_peered":"2023-08-25T09:49:39.671520+0000","last_clean":"2023-08-25T09:49:39.671520+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:49:39.671520+0000","last_undegraded":"2023-08-25T09:49:39.671520+0000","last_fullsized":"2023-08-25T09:49:39.671520+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:08:33.021697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"28'8","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609696+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:49:40.609696+0000","last_peered":"2023-08-25T09:49:40.609696+0000","last_clean":"2023-08-25T09:49:40.609696+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:49:40.609696+0000","last_undegraded":"2023-08-25T09:49:40.609696+0000","last_fullsized":"2023-08-25T09:49:40.609696+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:16:55.451701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610899+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.610899+0000","last_peered":"2023-08-25T09:49:40.610899+0000","last_clean":"2023-08-25T09:49:40.610899+0000","last_became_active":"2023-08-25T09:49:34.630012+0000","last_became_peered":"2023-08-25T09:49:34.630012+0000","last_unstale":"2023-08-25T09:49:40.610899+0000","last_undegraded":"2023-08-25T09:49:40.610899+0000","last_fullsized":"2023-08-25T09:49:40.610899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:11:33.237689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610543+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:49:40.610543+0000","last_peered":"2023-08-25T09:49:40.610543+0000","last_clean":"2023-08-25T09:49:40.610543+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:49:40.610543+0000","last_undegraded":"2023-08-25T09:49:40.610543+0000","last_fullsized":"2023-08-25T09:49:40.610543+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:25.560317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671645+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:49:39.671645+0000","last_peered":"2023-08-25T09:49:39.671645+0000","last_clean":"2023-08-25T09:49:39.671645+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:49:39.671645+0000","last_undegraded":"2023-08-25T09:49:39.671645+0000","last_fullsized":"2023-08-25T09:49:39.671645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:37:05.937993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669858+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:49:39.669858+0000","last_peered":"2023-08-25T09:49:39.669858+0000","last_clean":"2023-08-25T09:49:39.669858+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:49:39.669858+0000","last_undegraded":"2023-08-25T09:49:39.669858+0000","last_fullsized":"2023-08-25T09:49:39.669858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:20:55.146183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668790+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:49:39.668790+0000","last_peered":"2023-08-25T09:49:39.668790+0000","last_clean":"2023-08-25T09:49:39.668790+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:49:39.668790+0000","last_undegraded":"2023-08-25T09:49:39.668790+0000","last_fullsized":"2023-08-25T09:49:39.668790+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:51.399369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672435+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:49:39.672435+0000","last_peered":"2023-08-25T09:49:39.672435+0000","last_clean":"2023-08-25T09:49:39.672435+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:49:39.672435+0000","last_undegraded":"2023-08-25T09:49:39.672435+0000","last_fullsized":"2023-08-25T09:49:39.672435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:41:04.943976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609915+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:49:40.609915+0000","last_peered":"2023-08-25T09:49:40.609915+0000","last_clean":"2023-08-25T09:49:40.609915+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:49:40.609915+0000","last_undegraded":"2023-08-25T09:49:40.609915+0000","last_fullsized":"2023-08-25T09:49:40.609915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:30:03.672997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670253+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:49:39.670253+0000","last_peered":"2023-08-25T09:49:39.670253+0000","last_clean":"2023-08-25T09:49:39.670253+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:49:39.670253+0000","last_undegraded":"2023-08-25T09:49:39.670253+0000","last_fullsized":"2023-08-25T09:49:39.670253+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.806748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610312+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:49:40.610312+0000","last_peered":"2023-08-25T09:49:40.610312+0000","last_clean":"2023-08-25T09:49:40.610312+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:49:40.610312+0000","last_undegraded":"2023-08-25T09:49:40.610312+0000","last_fullsized":"2023-08-25T09:49:40.610312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:15:05.871192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672132+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:49:39.672132+0000","last_peered":"2023-08-25T09:49:39.672132+0000","last_clean":"2023-08-25T09:49:39.672132+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:49:39.672132+0000","last_undegraded":"2023-08-25T09:49:39.672132+0000","last_fullsized":"2023-08-25T09:49:39.672132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:51:00.162861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609435+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:49:40.609435+0000","last_peered":"2023-08-25T09:49:40.609435+0000","last_clean":"2023-08-25T09:49:40.609435+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:49:40.609435+0000","last_undegraded":"2023-08-25T09:49:40.609435+0000","last_fullsized":"2023-08-25T09:49:40.609435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:54:36.909125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609010+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:49:40.609010+0000","last_peered":"2023-08-25T09:49:40.609010+0000","last_clean":"2023-08-25T09:49:40.609010+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:49:40.609010+0000","last_undegraded":"2023-08-25T09:49:40.609010+0000","last_fullsized":"2023-08-25T09:49:40.609010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:45.655937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671033+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:49:39.671033+0000","last_peered":"2023-08-25T09:49:39.671033+0000","last_clean":"2023-08-25T09:49:39.671033+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:49:39.671033+0000","last_undegraded":"2023-08-25T09:49:39.671033+0000","last_fullsized":"2023-08-25T09:49:39.671033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:19:24.241659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609947+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:49:40.609947+0000","last_peered":"2023-08-25T09:49:40.609947+0000","last_clean":"2023-08-25T09:49:40.609947+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:49:40.609947+0000","last_undegraded":"2023-08-25T09:49:40.609947+0000","last_fullsized":"2023-08-25T09:49:40.609947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:27:06.196565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533764+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:49:40.533764+0000","last_peered":"2023-08-25T09:49:40.533764+0000","last_clean":"2023-08-25T09:49:40.533764+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:49:40.533764+0000","last_undegraded":"2023-08-25T09:49:40.533764+0000","last_fullsized":"2023-08-25T09:49:40.533764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:53:01.633281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672281+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:49:39.672281+0000","last_peered":"2023-08-25T09:49:39.672281+0000","last_clean":"2023-08-25T09:49:39.672281+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:49:39.672281+0000","last_undegraded":"2023-08-25T09:49:39.672281+0000","last_fullsized":"2023-08-25T09:49:39.672281+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:38:19.645378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533734+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:49:40.533734+0000","last_peered":"2023-08-25T09:49:40.533734+0000","last_clean":"2023-08-25T09:49:40.533734+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:49:40.533734+0000","last_undegraded":"2023-08-25T09:49:40.533734+0000","last_fullsized":"2023-08-25T09:49:40.533734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:27:54.228223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533904+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:49:40.533904+0000","last_peered":"2023-08-25T09:49:40.533904+0000","last_clean":"2023-08-25T09:49:40.533904+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:49:40.533904+0000","last_undegraded":"2023-08-25T09:49:40.533904+0000","last_fullsized":"2023-08-25T09:49:40.533904+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:06:19.107446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670060+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:49:39.670060+0000","last_peered":"2023-08-25T09:49:39.670060+0000","last_clean":"2023-08-25T09:49:39.670060+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:49:39.670060+0000","last_undegraded":"2023-08-25T09:49:39.670060+0000","last_fullsized":"2023-08-25T09:49:39.670060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:08:51.960006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608748+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:49:40.608748+0000","last_peered":"2023-08-25T09:49:40.608748+0000","last_clean":"2023-08-25T09:49:40.608748+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:49:40.608748+0000","last_undegraded":"2023-08-25T09:49:40.608748+0000","last_fullsized":"2023-08-25T09:49:40.608748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:35.777074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668458+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:49:39.668458+0000","last_peered":"2023-08-25T09:49:39.668458+0000","last_clean":"2023-08-25T09:49:39.668458+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:49:39.668458+0000","last_undegraded":"2023-08-25T09:49:39.668458+0000","last_fullsized":"2023-08-25T09:49:39.668458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:40:57.260235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669943+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:49:39.669943+0000","last_peered":"2023-08-25T09:49:39.669943+0000","last_clean":"2023-08-25T09:49:39.669943+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:49:39.669943+0000","last_undegraded":"2023-08-25T09:49:39.669943+0000","last_fullsized":"2023-08-25T09:49:39.669943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:42:36.500862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670365+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:49:39.670365+0000","last_peered":"2023-08-25T09:49:39.670365+0000","last_clean":"2023-08-25T09:49:39.670365+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:49:39.670365+0000","last_undegraded":"2023-08-25T09:49:39.670365+0000","last_fullsized":"2023-08-25T09:49:39.670365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:33:44.740556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314206+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:49:40.314206+0000","last_peered":"2023-08-25T09:49:40.314206+0000","last_clean":"2023-08-25T09:49:40.314206+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:49:40.314206+0000","last_undegraded":"2023-08-25T09:49:40.314206+0000","last_fullsized":"2023-08-25T09:49:40.314206+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:35.345143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670439+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:49:39.670439+0000","last_peered":"2023-08-25T09:49:39.670439+0000","last_clean":"2023-08-25T09:49:39.670439+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:49:39.670439+0000","last_undegraded":"2023-08-25T09:49:39.670439+0000","last_fullsized":"2023-08-25T09:49:39.670439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:38.142714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669800+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:49:39.669800+0000","last_peered":"2023-08-25T09:49:39.669800+0000","last_clean":"2023-08-25T09:49:39.669800+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:49:39.669800+0000","last_undegraded":"2023-08-25T09:49:39.669800+0000","last_fullsized":"2023-08-25T09:49:39.669800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:05:15.100928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668185+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:49:39.668185+0000","last_peered":"2023-08-25T09:49:39.668185+0000","last_clean":"2023-08-25T09:49:39.668185+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:49:39.668185+0000","last_undegraded":"2023-08-25T09:49:39.668185+0000","last_fullsized":"2023-08-25T09:49:39.668185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:52.824866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671435+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:49:39.671435+0000","last_peered":"2023-08-25T09:49:39.671435+0000","last_clean":"2023-08-25T09:49:39.671435+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:49:39.671435+0000","last_undegraded":"2023-08-25T09:49:39.671435+0000","last_fullsized":"2023-08-25T09:49:39.671435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:22:34.193671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668773+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:49:39.668773+0000","last_peered":"2023-08-25T09:49:39.668773+0000","last_clean":"2023-08-25T09:49:39.668773+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:49:39.668773+0000","last_undegraded":"2023-08-25T09:49:39.668773+0000","last_fullsized":"2023-08-25T09:49:39.668773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:38:09.260537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313744+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:49:40.313744+0000","last_peered":"2023-08-25T09:49:40.313744+0000","last_clean":"2023-08-25T09:49:40.313744+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:49:40.313744+0000","last_undegraded":"2023-08-25T09:49:40.313744+0000","last_fullsized":"2023-08-25T09:49:40.313744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:47:49.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314502+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:49:40.314502+0000","last_peered":"2023-08-25T09:49:40.314502+0000","last_clean":"2023-08-25T09:49:40.314502+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:49:40.314502+0000","last_undegraded":"2023-08-25T09:49:40.314502+0000","last_fullsized":"2023-08-25T09:49:40.314502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:52.308305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669862+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:49:39.669862+0000","last_peered":"2023-08-25T09:49:39.669862+0000","last_clean":"2023-08-25T09:49:39.669862+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:49:39.669862+0000","last_undegraded":"2023-08-25T09:49:39.669862+0000","last_fullsized":"2023-08-25T09:49:39.669862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:33:20.778440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669203+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:49:39.669203+0000","last_peered":"2023-08-25T09:49:39.669203+0000","last_clean":"2023-08-25T09:49:39.669203+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:49:39.669203+0000","last_undegraded":"2023-08-25T09:49:39.669203+0000","last_fullsized":"2023-08-25T09:49:39.669203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:14.357542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669995+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:49:39.669995+0000","last_peered":"2023-08-25T09:49:39.669995+0000","last_clean":"2023-08-25T09:49:39.669995+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:49:39.669995+0000","last_undegraded":"2023-08-25T09:49:39.669995+0000","last_fullsized":"2023-08-25T09:49:39.669995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:06.555517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668019+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:49:39.668019+0000","last_peered":"2023-08-25T09:49:39.668019+0000","last_clean":"2023-08-25T09:49:39.668019+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:49:39.668019+0000","last_undegraded":"2023-08-25T09:49:39.668019+0000","last_fullsized":"2023-08-25T09:49:39.668019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:35:29.581577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671260+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:49:39.671260+0000","last_peered":"2023-08-25T09:49:39.671260+0000","last_clean":"2023-08-25T09:49:39.671260+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:49:39.671260+0000","last_undegraded":"2023-08-25T09:49:39.671260+0000","last_fullsized":"2023-08-25T09:49:39.671260+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:43.256956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669816+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:49:39.669816+0000","last_peered":"2023-08-25T09:49:39.669816+0000","last_clean":"2023-08-25T09:49:39.669816+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:49:39.669816+0000","last_undegraded":"2023-08-25T09:49:39.669816+0000","last_fullsized":"2023-08-25T09:49:39.669816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:05:53.483156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669716+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:49:39.669716+0000","last_peered":"2023-08-25T09:49:39.669716+0000","last_clean":"2023-08-25T09:49:39.669716+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:49:39.669716+0000","last_undegraded":"2023-08-25T09:49:39.669716+0000","last_fullsized":"2023-08-25T09:49:39.669716+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:17.656387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609913+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:49:40.609913+0000","last_peered":"2023-08-25T09:49:40.609913+0000","last_clean":"2023-08-25T09:49:40.609913+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:49:40.609913+0000","last_undegraded":"2023-08-25T09:49:40.609913+0000","last_fullsized":"2023-08-25T09:49:40.609913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:04:46.102346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669727+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:49:39.669727+0000","last_peered":"2023-08-25T09:49:39.669727+0000","last_clean":"2023-08-25T09:49:39.669727+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:49:39.669727+0000","last_undegraded":"2023-08-25T09:49:39.669727+0000","last_fullsized":"2023-08-25T09:49:39.669727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:03:17.962039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668925+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:49:39.668925+0000","last_peered":"2023-08-25T09:49:39.668925+0000","last_clean":"2023-08-25T09:49:39.668925+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:49:39.668925+0000","last_undegraded":"2023-08-25T09:49:39.668925+0000","last_fullsized":"2023-08-25T09:49:39.668925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:15:02.299452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314453+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:49:40.314453+0000","last_peered":"2023-08-25T09:49:40.314453+0000","last_clean":"2023-08-25T09:49:40.314453+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:49:40.314453+0000","last_undegraded":"2023-08-25T09:49:40.314453+0000","last_fullsized":"2023-08-25T09:49:40.314453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:31.990094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4639,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":130128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.851}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4950000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":401408,"data_stored":720656,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.482}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4379999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]}]},{"osd":6,"up_from":15,"seq":64424509451,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.456}]}]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":315392,"data_stored":126883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":311296,"data_stored":126321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542154,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":376832,"data_stored":717181,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:04.479 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-08-25T09:50:04.480 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-08-25T09:50:04.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-08-25T09:50:04.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-08-25T09:50:04.481 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-08-25T09:50:04.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.726+0000 7f0270614640 1 Processor -- start 2023-08-25T09:50:04.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.726+0000 7f0270614640 1 -- start start 2023-08-25T09:50:04.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f0270614640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0268156640 0x7f02680c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f0270614640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f026815c6c0 0x7f026815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f0270614640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02680c4ae0 con 0x7f02680cbdf0 2023-08-25T09:50:04.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f0270614640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02680cad80 con 0x7f026815c6c0 2023-08-25T09:50:04.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f0270614640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02680c3c30 con 0x7f0268156640 2023-08-25T09:50:04.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f026ebb1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f026815c6c0 0x7f026815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f026dbaf640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f02680cbdf0 0x7f026806b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46364/0 (socket says 172.21.15.71:46364) 2023-08-25T09:50:04.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.728+0000 7f026dbaf640 1 -- 172.21.15.71:0/1797083906 learned_addr learned my addr 172.21.15.71:0/1797083906 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:04.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.729+0000 7f026e3b0640 1 --2- 172.21.15.71:0/1797083906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0268156640 0x7f02680c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.729+0000 7f026ebb1640 1 -- 172.21.15.71:0/1797083906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0268156640 msgr2=0x7f02680c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:04.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.729+0000 7f026ebb1640 1 --2- 172.21.15.71:0/1797083906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0268156640 0x7f02680c9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.729+0000 7f026ebb1640 1 -- 172.21.15.71:0/1797083906 >> v1:172.21.15.71:6789/0 conn(0x7f02680cbdf0 legacy=0x7f026806b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.729+0000 7f026d3ae640 1 -- 172.21.15.71:0/1797083906 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492464514 0 0) 0x7f02680c4ae0 con 0x7f02680cbdf0 2023-08-25T09:50:04.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.729+0000 7f026ebb1640 1 -- 172.21.15.71:0/1797083906 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680c9550 con 0x7f026815c6c0 2023-08-25T09:50:04.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.730+0000 7f026ebb1640 1 --2- 172.21.15.71:0/1797083906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f026815c6c0 0x7f026815cb50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0260009d90 tx=0x7f026002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd20f1588708dc45 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:04.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.730+0000 7f026d3ae640 1 -- 172.21.15.71:0/1797083906 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026002cbd0 con 0x7f026815c6c0 2023-08-25T09:50:04.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.730+0000 7f026d3ae640 1 -- 172.21.15.71:0/1797083906 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026002cd30 con 0x7f026815c6c0 2023-08-25T09:50:04.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.730+0000 7f026d3ae640 1 -- 172.21.15.71:0/1797083906 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026002cea0 con 0x7f026815c6c0 2023-08-25T09:50:04.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 -- 172.21.15.71:0/1797083906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f026815c6c0 msgr2=0x7f026815cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 --2- 172.21.15.71:0/1797083906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f026815c6c0 0x7f026815cb50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0260009d90 tx=0x7f026002be20 comp rx=0 tx=0).stop 2023-08-25T09:50:04.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 -- 172.21.15.71:0/1797083906 shutdown_connections 2023-08-25T09:50:04.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 --2- 172.21.15.71:0/1797083906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f026815c6c0 0x7f026815cb50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 --2- 172.21.15.71:0/1797083906 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0268156640 0x7f02680c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 -- 172.21.15.71:0/1797083906 >> 172.21.15.71:0/1797083906 conn(0x7f026806a5b0 msgr2=0x7f0268070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:04.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 -- 172.21.15.71:0/1797083906 shutdown_connections 2023-08-25T09:50:04.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.731+0000 7f0270614640 1 -- 172.21.15.71:0/1797083906 wait complete. 2023-08-25T09:50:04.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.732+0000 7f0270614640 1 Processor -- start 2023-08-25T09:50:04.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 -- start start 2023-08-25T09:50:04.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02680cbdf0 0x7f0268152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 0x7f02680c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f026815c6c0 0x7f0268155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026814bfd0 con 0x7f0268156640 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02681ef8e0 con 0x7f02680cbdf0 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.733+0000 7f0270614640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02681ef5b0 con 0x7f026815c6c0 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 0x7f02680c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 0x7f02680c6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59988/0 (socket says 172.21.15.71:59988) 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 -- 172.21.15.71:0/3158001349 learned_addr learned my addr 172.21.15.71:0/3158001349 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026e3b0640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02680cbdf0 0x7f0268152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026ebb1640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f026815c6c0 0x7f0268155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 -- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f026815c6c0 msgr2=0x7f0268155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f026815c6c0 0x7f0268155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 -- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02680cbdf0 msgr2=0x7f0268152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02680cbdf0 0x7f0268152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 -- 172.21.15.71:0/3158001349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0260009a70 con 0x7f0268156640 2023-08-25T09:50:04.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.734+0000 7f026dbaf640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 0x7f02680c6b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f025800c430 tx=0x7f025800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:04.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.735+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02580074e0 con 0x7f0268156640 2023-08-25T09:50:04.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.735+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0258007660 con 0x7f0268156640 2023-08-25T09:50:04.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.735+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258004a20 con 0x7f0268156640 2023-08-25T09:50:04.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.735+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026814fc30 con 0x7f0268156640 2023-08-25T09:50:04.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.735+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02680c74f0 con 0x7f0268156640 2023-08-25T09:50:04.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.736+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02681efa50 con 0x7f0268156640 2023-08-25T09:50:04.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.737+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0258004b80 con 0x7f0268156640 2023-08-25T09:50:04.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.738+0000 7f025effd640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f024c06a050 0x7f024c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:04.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.738+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0258088740 con 0x7f0268156640 2023-08-25T09:50:04.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.738+0000 7f026e3b0640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f024c06a050 0x7f024c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:04.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.739+0000 7f026e3b0640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f024c06a050 0x7f024c06c510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0264006fd0 tx=0x7f0264008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:04.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.743+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0258053aa0 con 0x7f0268156640 2023-08-25T09:50:04.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.967+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0268069160 con 0x7f0268156640 2023-08-25T09:50:04.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.975+0000 7f025effd640 1 -- 172.21.15.71:0/3158001349 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+169489 (secure 0 0 0) 0x7f02580570c0 con 0x7f0268156640 2023-08-25T09:50:04.978 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:04.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f024c06a050 msgr2=0x7f024c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f024c06a050 0x7f024c06c510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0264006fd0 tx=0x7f0264008040 comp rx=0 tx=0).stop 2023-08-25T09:50:04.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 msgr2=0x7f02680c6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:04.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 0x7f02680c6b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f025800c430 tx=0x7f025800c8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 shutdown_connections 2023-08-25T09:50:04.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f024c06a050 0x7f024c06c510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f026815c6c0 0x7f0268155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0268156640 0x7f02680c6b60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 --2- 172.21.15.71:0/3158001349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02680cbdf0 0x7f0268152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 >> 172.21.15.71:0/3158001349 conn(0x7f026806a5b0 msgr2=0x7f02680c3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 shutdown_connections 2023-08-25T09:50:04.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:04.979+0000 7f0270614640 1 -- 172.21.15.71:0/3158001349 wait complete. 2023-08-25T09:50:04.998 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":5,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":786313351},{"type":"v1","addr":"172.21.15.71:6833","nonce":786313351}]},"active_addr":"172.21.15.71:6833/786313351","active_change":"2023-08-25T09:49:10.610048+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":1135389859}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2230515392}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":2170776043}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.71:0","nonce":3517306118}]}]} 2023-08-25T09:50:05.002 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-08-25T09:50:05.002 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-08-25T09:50:05.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:05.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.215+0000 7f1dfef8b640 1 Processor -- start 2023-08-25T09:50:05.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.216+0000 7f1dfef8b640 1 -- start start 2023-08-25T09:50:05.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.217+0000 7f1dfef8b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1df8150c50 0x7f1df80c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.217+0000 7f1dfef8b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1df80c76b0 0x7f1df81613f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.217+0000 7f1dfef8b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df80c2b80 con 0x7f1df80c58c0 2023-08-25T09:50:05.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.217+0000 7f1dfef8b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df80c1770 con 0x7f1df80c76b0 2023-08-25T09:50:05.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.217+0000 7f1dfef8b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df80c1f30 con 0x7f1df8150c50 2023-08-25T09:50:05.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfd528640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1df80c76b0 0x7f1df81613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1deffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1df80c58c0 0x7f1df80c34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46380/0 (socket says 172.21.15.71:46380) 2023-08-25T09:50:05.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1deffff640 1 -- 172.21.15.71:0/1272308008 learned_addr learned my addr 172.21.15.71:0/1272308008 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:05.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfcd27640 1 --2- 172.21.15.71:0/1272308008 >> v2:172.21.15.164:3300/0 conn(0x7f1df8150c50 0x7f1df80c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1def7fe640 1 -- 172.21.15.71:0/1272308008 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892071162 0 0) 0x7f1df80c2b80 con 0x7f1df80c58c0 2023-08-25T09:50:05.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1def7fe640 1 -- 172.21.15.71:0/1272308008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1de0003520 con 0x7f1df80c58c0 2023-08-25T09:50:05.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfd528640 1 -- 172.21.15.71:0/1272308008 >> v2:172.21.15.164:3300/0 conn(0x7f1df8150c50 msgr2=0x7f1df80c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:05.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfd528640 1 --2- 172.21.15.71:0/1272308008 >> v2:172.21.15.164:3300/0 conn(0x7f1df8150c50 0x7f1df80c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfd528640 1 -- 172.21.15.71:0/1272308008 >> v1:172.21.15.71:6789/0 conn(0x7f1df80c58c0 legacy=0x7f1df80c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfd528640 1 -- 172.21.15.71:0/1272308008 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df80c6400 con 0x7f1df80c76b0 2023-08-25T09:50:05.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.218+0000 7f1dfcd27640 1 --2- 172.21.15.71:0/1272308008 >> v2:172.21.15.164:3300/0 conn(0x7f1df8150c50 0x7f1df80c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:05.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.219+0000 7f1dfd528640 1 --2- 172.21.15.71:0/1272308008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c76b0 0x7f1df81613f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1df8074090 tx=0x7f1df002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8e7a02a99f2cd0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:05.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1def7fe640 1 -- 172.21.15.71:0/1272308008 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0030610 con 0x7f1df80c76b0 2023-08-25T09:50:05.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1def7fe640 1 -- 172.21.15.71:0/1272308008 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df0030770 con 0x7f1df80c76b0 2023-08-25T09:50:05.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1def7fe640 1 -- 172.21.15.71:0/1272308008 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0030ae0 con 0x7f1df80c76b0 2023-08-25T09:50:05.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 -- 172.21.15.71:0/1272308008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c76b0 msgr2=0x7f1df81613f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/1272308008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c76b0 0x7f1df81613f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1df8074090 tx=0x7f1df002b960 comp rx=0 tx=0).stop 2023-08-25T09:50:05.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 -- 172.21.15.71:0/1272308008 shutdown_connections 2023-08-25T09:50:05.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/1272308008 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c76b0 0x7f1df81613f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/1272308008 >> v2:172.21.15.164:3300/0 conn(0x7f1df8150c50 0x7f1df80c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 -- 172.21.15.71:0/1272308008 >> 172.21.15.71:0/1272308008 conn(0x7f1df806bff0 msgr2=0x7f1df806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:05.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 -- 172.21.15.71:0/1272308008 shutdown_connections 2023-08-25T09:50:05.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.220+0000 7f1dfef8b640 1 -- 172.21.15.71:0/1272308008 wait complete. 2023-08-25T09:50:05.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.222+0000 7f1dfef8b640 1 Processor -- start 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.222+0000 7f1dfef8b640 1 -- start start 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.223+0000 7f1dfef8b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c58c0 0x7f1df806ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.223+0000 7f1dfef8b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df80c76b0 0x7f1df806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.223+0000 7f1dfef8b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 0x7f1df806f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.223+0000 7f1dfef8b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80ca9b0 con 0x7f1df8150c50 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.223+0000 7f1dfef8b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df814a720 con 0x7f1df80c76b0 2023-08-25T09:50:05.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.223+0000 7f1dfef8b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80cb140 con 0x7f1df80c58c0 2023-08-25T09:50:05.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 0x7f1df806f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 0x7f1df806f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60008/0 (socket says 172.21.15.71:60008) 2023-08-25T09:50:05.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 -- 172.21.15.71:0/4028166464 learned_addr learned my addr 172.21.15.71:0/4028166464 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1deffff640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df80c76b0 0x7f1df806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfcd27640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c58c0 0x7f1df806ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 -- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c58c0 msgr2=0x7f1df806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c58c0 0x7f1df806ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 -- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df80c76b0 msgr2=0x7f1df806ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df80c76b0 0x7f1df806ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 -- 172.21.15.71:0/4028166464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df0009400 con 0x7f1df8150c50 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1deffff640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df80c76b0 0x7f1df806ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.224+0000 7f1dfd528640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 0x7f1df806f3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009890 tx=0x7f1df00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.225+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df003f070 con 0x7f1df8150c50 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.225+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df0006e10 con 0x7f1df8150c50 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.225+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df0030ea0 con 0x7f1df8150c50 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.225+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df80704f0 con 0x7f1df8150c50 2023-08-25T09:50:05.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.225+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df80caba0 con 0x7f1df8150c50 2023-08-25T09:50:05.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.226+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df814a890 con 0x7f1df8150c50 2023-08-25T09:50:05.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.227+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1df0003830 con 0x7f1df8150c50 2023-08-25T09:50:05.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.228+0000 7f1ded7fa640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1de406a050 0x7f1de406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.228+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1df00aaa20 con 0x7f1df8150c50 2023-08-25T09:50:05.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.228+0000 7f1dfcd27640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1de406a050 0x7f1de406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.229+0000 7f1dfcd27640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1de406a050 0x7f1de406c510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1df4003fd0 tx=0x7f1df40039f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:05.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.232+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1df0075ea0 con 0x7f1df8150c50 2023-08-25T09:50:05.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.421+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1df804e420 con 0x7f1df8150c50 2023-08-25T09:50:05.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.424+0000 7f1ded7fa640 1 -- 172.21.15.71:0/4028166464 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7f1df00794c0 con 0x7f1df8150c50 2023-08-25T09:50:05.427 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:05.427 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:05.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1de406a050 msgr2=0x7f1de406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1de406a050 0x7f1de406c510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1df4003fd0 tx=0x7f1df40039f0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 msgr2=0x7f1df806f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 0x7f1df806f3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009890 tx=0x7f1df00046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 shutdown_connections 2023-08-25T09:50:05.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df80c76b0 0x7f1df806ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1de406a050 0x7f1de406c510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df8150c50 0x7f1df806f3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 --2- 172.21.15.71:0/4028166464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c58c0 0x7f1df806ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 >> 172.21.15.71:0/4028166464 conn(0x7f1df806bff0 msgr2=0x7f1df814e420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:05.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 shutdown_connections 2023-08-25T09:50:05.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.427+0000 7f1dfef8b640 1 -- 172.21.15.71:0/4028166464 wait complete. 2023-08-25T09:50:05.447 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-08-25T09:50:05.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:05.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.662+0000 7ff3418bf640 1 Processor -- start 2023-08-25T09:50:05.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.662+0000 7ff3418bf640 1 -- start start 2023-08-25T09:50:05.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.663+0000 7ff3418bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff33c1594d0 0x7ff33c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.663+0000 7ff3418bf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff33c06cc70 0x7ff33c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.663+0000 7ff3418bf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff33c06a960 con 0x7ff33c0c3940 2023-08-25T09:50:05.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.663+0000 7ff3418bf640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c06ac20 con 0x7ff33c06cc70 2023-08-25T09:50:05.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.663+0000 7ff3418bf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c06af00 con 0x7ff33c1594d0 2023-08-25T09:50:05.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff33c06cc70 0x7ff33c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff33c06cc70 0x7ff33c14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60016/0 (socket says 172.21.15.71:60016) 2023-08-25T09:50:05.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 -- 172.21.15.71:0/3762902396 learned_addr learned my addr 172.21.15.71:0/3762902396 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:05.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff33affd640 1 --2- 172.21.15.71:0/3762902396 >> v2:172.21.15.164:3300/0 conn(0x7ff33c1594d0 0x7ff33c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 -- 172.21.15.71:0/3762902396 >> v2:172.21.15.164:3300/0 conn(0x7ff33c1594d0 msgr2=0x7ff33c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:05.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 --2- 172.21.15.71:0/3762902396 >> v2:172.21.15.164:3300/0 conn(0x7ff33c1594d0 0x7ff33c0c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 -- 172.21.15.71:0/3762902396 >> v1:172.21.15.164:6789/0 conn(0x7ff33c0c3940 legacy=0x7ff33c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:05.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.664+0000 7ff333fff640 1 -- 172.21.15.71:0/3762902396 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c0c2e60 con 0x7ff33c06cc70 2023-08-25T09:50:05.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.665+0000 7ff333fff640 1 --2- 172.21.15.71:0/3762902396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c14f850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff33c071d10 tx=0x7ff32402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7006171cff984089 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.666+0000 7ff33a7fc640 1 -- 172.21.15.71:0/3762902396 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32402eea0 con 0x7ff33c06cc70 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.666+0000 7ff33a7fc640 1 -- 172.21.15.71:0/3762902396 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324036440 con 0x7ff33c06cc70 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.666+0000 7ff33a7fc640 1 -- 172.21.15.71:0/3762902396 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240367d0 con 0x7ff33c06cc70 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.666+0000 7ff3418bf640 1 -- 172.21.15.71:0/3762902396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 msgr2=0x7ff33c14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.666+0000 7ff3418bf640 1 --2- 172.21.15.71:0/3762902396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c14f850 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff33c071d10 tx=0x7ff32402e0f0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.667+0000 7ff3418bf640 1 -- 172.21.15.71:0/3762902396 shutdown_connections 2023-08-25T09:50:05.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.667+0000 7ff3418bf640 1 --2- 172.21.15.71:0/3762902396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c14f850 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.667+0000 7ff3418bf640 1 --2- 172.21.15.71:0/3762902396 >> v2:172.21.15.164:3300/0 conn(0x7ff33c1594d0 0x7ff33c0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.667+0000 7ff3418bf640 1 -- 172.21.15.71:0/3762902396 >> 172.21.15.71:0/3762902396 conn(0x7ff33c0654f0 msgr2=0x7ff33c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:05.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.667+0000 7ff3418bf640 1 -- 172.21.15.71:0/3762902396 shutdown_connections 2023-08-25T09:50:05.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.667+0000 7ff3418bf640 1 -- 172.21.15.71:0/3762902396 wait complete. 2023-08-25T09:50:05.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.668+0000 7ff3418bf640 1 Processor -- start 2023-08-25T09:50:05.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.669+0000 7ff3418bf640 1 -- start start 2023-08-25T09:50:05.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff3418bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff3418bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff33c0c3940 0x7ff33c14e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff3418bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff33c1594d0 0x7ff33c14eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff3418bf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c1407e0 con 0x7ff33c06cc70 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff3418bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c1ee050 con 0x7ff33c0c3940 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff3418bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c158df0 con 0x7ff33c1594d0 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c1588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60032/0 (socket says 172.21.15.71:60032) 2023-08-25T09:50:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 -- 172.21.15.71:0/2257491822 learned_addr learned my addr 172.21.15.71:0/2257491822 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 -- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff33c1594d0 msgr2=0x7ff33c14eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff33c1594d0 0x7ff33c14eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff333fff640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff33c0c3940 0x7ff33c14e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 -- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff33c0c3940 msgr2=0x7ff33c14e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff33c0c3940 0x7ff33c14e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.670+0000 7ff33affd640 1 -- 172.21.15.71:0/2257491822 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32400b770 con 0x7ff33c06cc70 2023-08-25T09:50:05.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.671+0000 7ff33affd640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c1588b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00c7f0 tx=0x7ff32c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:05.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.671+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c004220 con 0x7ff33c06cc70 2023-08-25T09:50:05.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.671+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c0043a0 con 0x7ff33c06cc70 2023-08-25T09:50:05.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.671+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c003850 con 0x7ff33c06cc70 2023-08-25T09:50:05.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.671+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c0c6220 con 0x7ff33c06cc70 2023-08-25T09:50:05.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.671+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c1ee540 con 0x7ff33c06cc70 2023-08-25T09:50:05.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.673+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33c066510 con 0x7ff33c06cc70 2023-08-25T09:50:05.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.674+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff32c003bc0 con 0x7ff33c06cc70 2023-08-25T09:50:05.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.675+0000 7ff338ff9640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff31c06a050 0x7ff31c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:05.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.675+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff32c088930 con 0x7ff33c06cc70 2023-08-25T09:50:05.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.675+0000 7ff333fff640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff31c06a050 0x7ff31c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:05.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.676+0000 7ff333fff640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff31c06a050 0x7ff31c06c510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff324038000 tx=0x7ff324008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:05.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.679+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff32c053ca0 con 0x7ff33c06cc70 2023-08-25T09:50:05.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.874+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff33c069160 con 0x7ff33c06cc70 2023-08-25T09:50:05.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.878+0000 7ff338ff9640 1 -- 172.21.15.71:0/2257491822 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+20120 (secure 0 0 0) 0x7ff32c01e020 con 0x7ff33c06cc70 2023-08-25T09:50:05.880 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:05.880 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":29,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:49:39.129269+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:05.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff31c06a050 msgr2=0x7ff31c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff31c06a050 0x7ff31c06c510 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff324038000 tx=0x7ff324008040 comp rx=0 tx=0).stop 2023-08-25T09:50:05.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 msgr2=0x7ff33c1588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:05.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c1588b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00c7f0 tx=0x7ff32c00ccb0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 shutdown_connections 2023-08-25T09:50:05.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff33c0c3940 0x7ff33c14e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff31c06a050 0x7ff31c06c510 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff33c1594d0 0x7ff33c14eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 --2- 172.21.15.71:0/2257491822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff33c06cc70 0x7ff33c1588b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:05.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 >> 172.21.15.71:0/2257491822 conn(0x7ff33c0654f0 msgr2=0x7ff33c0c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:05.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 shutdown_connections 2023-08-25T09:50:05.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:05.880+0000 7ff3418bf640 1 -- 172.21.15.71:0/2257491822 wait complete. 2023-08-25T09:50:05.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-08-25T09:50:05.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-08-25T09:50:05.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-08-25T09:50:05.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-08-25T09:50:05.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-08-25T09:50:05.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-08-25T09:50:05.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-08-25T09:50:05.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-08-25T09:50:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.201+0000 7f8b7edd6640 1 Processor -- start 2023-08-25T09:50:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.201+0000 7f8b7edd6640 1 -- start start 2023-08-25T09:50:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.204+0000 7f8b7edd6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b7806abc0 0x7f8b78069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.204+0000 7f8b7edd6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b78155da0 con 0x7f8b780acd20 2023-08-25T09:50:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.204+0000 7f8b7edd6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b78152860 con 0x7f8b78069d90 2023-08-25T09:50:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.204+0000 7f8b7edd6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b78155980 con 0x7f8b7806abc0 2023-08-25T09:50:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.204+0000 7f8b7cb72640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8b780acd20 0x7f8b780c0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46396/0 (socket says 172.21.15.71:46396) 2023-08-25T09:50:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.204+0000 7f8b7cb72640 1 -- 172.21.15.71:0/3064199900 learned_addr learned my addr 172.21.15.71:0/3064199900 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.205+0000 7f8b77fff640 1 --2- 172.21.15.71:0/3064199900 >> v2:172.21.15.164:3300/0 conn(0x7f8b7806abc0 0x7f8b78069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.205+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769955324 0 0) 0x7f8b78155da0 con 0x7f8b780acd20 2023-08-25T09:50:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.206+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b60003520 con 0x7f8b780acd20 2023-08-25T09:50:06.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.206+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106421478 0 0) 0x7f8b78152860 con 0x7f8b78069d90 2023-08-25T09:50:06.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.206+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b78155da0 con 0x7f8b78069d90 2023-08-25T09:50:06.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.206+0000 7f8b77fff640 1 -- 172.21.15.71:0/3064199900 >> v1:172.21.15.164:6789/0 conn(0x7f8b78069d90 legacy=0x7f8b780bf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.206+0000 7f8b77fff640 1 -- 172.21.15.71:0/3064199900 >> v1:172.21.15.71:6789/0 conn(0x7f8b780acd20 legacy=0x7f8b780c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.206+0000 7f8b77fff640 1 -- 172.21.15.71:0/3064199900 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b780bfd70 con 0x7f8b7806abc0 2023-08-25T09:50:06.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.208+0000 7fd8d7539640 1 Processor -- start 2023-08-25T09:50:06.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.208+0000 7f8b77fff640 1 --2- 172.21.15.71:0/3064199900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b7806abc0 0x7f8b78069850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8b680097a0 tx=0x7f8b6802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616400d55356e8b6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.208+0000 7fd8d7539640 1 -- start start 2023-08-25T09:50:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.209+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68030840 con 0x7f8b7806abc0 2023-08-25T09:50:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.209+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b680309a0 con 0x7f8b7806abc0 2023-08-25T09:50:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.209+0000 7f8b777fe640 1 -- 172.21.15.71:0/3064199900 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68030d70 con 0x7f8b7806abc0 2023-08-25T09:50:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 -- 172.21.15.71:0/3064199900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b7806abc0 msgr2=0x7f8b78069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 --2- 172.21.15.71:0/3064199900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b7806abc0 0x7f8b78069850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8b680097a0 tx=0x7f8b6802b980 comp rx=0 tx=0).stop 2023-08-25T09:50:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 -- 172.21.15.71:0/3064199900 shutdown_connections 2023-08-25T09:50:06.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 --2- 172.21.15.71:0/3064199900 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b7806abc0 0x7f8b78069850 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 -- 172.21.15.71:0/3064199900 >> 172.21.15.71:0/3064199900 conn(0x7f8b780c6cc0 msgr2=0x7f8b780c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 -- 172.21.15.71:0/3064199900 shutdown_connections 2023-08-25T09:50:06.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.210+0000 7f8b7edd6640 1 -- 172.21.15.71:0/3064199900 wait complete. 2023-08-25T09:50:06.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.211+0000 7fd8d7539640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd8c80d9340 0x7fd8c80df840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.211+0000 7fd8d7539640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd8c800edc0 0x7fd8c8038900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.213+0000 7f8b7edd6640 1 Processor -- start 2023-08-25T09:50:06.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7f1201287640 1 Processor -- start 2023-08-25T09:50:06.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7f1201287640 1 -- start start 2023-08-25T09:50:06.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7f1201287640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11fc158d40 0x7f11fc0c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7f1201287640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f11fc15d090 0x7f11fc0c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7f1201287640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0ca050 con 0x7f11fc0c9730 2023-08-25T09:50:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7f1201287640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc159d30 con 0x7f11fc15d090 2023-08-25T09:50:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7f1201287640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc1591d0 con 0x7f11fc158d40 2023-08-25T09:50:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7fd8d5ad6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd8c800edc0 0x7fd8c8038900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7fd8d5ad6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd8c800edc0 0x7fd8c8038900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60040/0 (socket says 172.21.15.71:60040) 2023-08-25T09:50:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7fd8d5ad6640 1 -- 172.21.15.71:0/2473874931 learned_addr learned my addr 172.21.15.71:0/2473874931 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7fd8d4ad4640 1 --2- 172.21.15.71:0/2473874931 >> v2:172.21.15.164:3300/0 conn(0x7fd8c80d9340 0x7fd8c80df840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.212+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80dfd80 con 0x7fd8c80d8f60 2023-08-25T09:50:06.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.213+0000 7fd8c7fff640 1 -- 172.21.15.71:0/2473874931 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647176944 0 0) 0x7fd8cc001110 con 0x7fd8c80d8f60 2023-08-25T09:50:06.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.213+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c804ffb0 con 0x7fd8c800edc0 2023-08-25T09:50:06.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.213+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c804f050 con 0x7fd8c80d9340 2023-08-25T09:50:06.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.213+0000 7fd8c7fff640 1 -- 172.21.15.71:0/2473874931 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc003520 con 0x7fd8c80d8f60 2023-08-25T09:50:06.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7fd8d5ad6640 1 -- 172.21.15.71:0/2473874931 >> v2:172.21.15.164:3300/0 conn(0x7fd8c80d9340 msgr2=0x7fd8c80df840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.213+0000 7f8b7edd6640 1 -- start start 2023-08-25T09:50:06.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f8b7edd6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b78069d90 0x7f8b781501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fa7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11fc158d40 0x7f11fc0c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fa7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11fc158d40 0x7f11fc0c8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56062/0 (socket says 172.21.15.71:56062) 2023-08-25T09:50:06.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fa7fc640 1 -- 172.21.15.71:0/257958587 learned_addr learned my addr 172.21.15.71:0/257958587 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7fd8d5ad6640 1 --2- 172.21.15.71:0/2473874931 >> v2:172.21.15.164:3300/0 conn(0x7fd8c80d9340 0x7fd8c80df840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7fd8d5ad6640 1 -- 172.21.15.71:0/2473874931 >> v1:172.21.15.71:6789/0 conn(0x7fd8c80d8f60 legacy=0x7fd8c8007160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7fd8d5ad6640 1 -- 172.21.15.71:0/2473874931 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c804f330 con 0x7fd8c800edc0 2023-08-25T09:50:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.214+0000 7fd8d4ad4640 1 --2- 172.21.15.71:0/2473874931 >> v2:172.21.15.164:3300/0 conn(0x7fd8c80d9340 0x7fd8c80df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:50:06.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f8b7edd6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b7806abc0 0x7f8b78156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f8b7edd6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b780acd20 0x7f8b781570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fa7fc640 1 -- 172.21.15.71:0/257958587 >> v2:172.21.15.71:3300/0 conn(0x7f11fc15d090 msgr2=0x7f11fc0c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fb7fe640 1 --2- 172.21.15.71:0/257958587 >> v2:172.21.15.71:3300/0 conn(0x7f11fc15d090 0x7f11fc0c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fa7fc640 1 --2- 172.21.15.71:0/257958587 >> v2:172.21.15.71:3300/0 conn(0x7f11fc15d090 0x7f11fc0c9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.215+0000 7f11fa7fc640 1 -- 172.21.15.71:0/257958587 >> v1:172.21.15.164:6789/0 conn(0x7f11fc0c9730 legacy=0x7f11fc15c980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T09:50:06.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f11fa7fc640 1 -- 172.21.15.71:0/257958587 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc159490 con 0x7f11fc158d40 2023-08-25T09:50:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f8b7cb72640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b78069d90 0x7f8b781501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f8b7cb72640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b78069d90 0x7f8b781501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60066/0 (socket says 172.21.15.71:60066) 2023-08-25T09:50:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b7806abc0 0x7f8b78156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f8b77fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b7806abc0 0x7f8b78156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41364/0 (socket says 172.21.15.71:41364) 2023-08-25T09:50:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7fd8d5ad6640 1 --2- 172.21.15.71:0/2473874931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c800edc0 0x7fd8c8038900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd8d004f120 tx=0x7fd8d00a5d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12204773c1e557c7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7fd8c7fff640 1 -- 172.21.15.71:0/2473874931 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00a6b30 con 0x7fd8c800edc0 2023-08-25T09:50:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7fd8c7fff640 1 -- 172.21.15.71:0/2473874931 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d00a6cd0 con 0x7fd8c800edc0 2023-08-25T09:50:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7fd8c7fff640 1 -- 172.21.15.71:0/2473874931 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00ae640 con 0x7fd8c800edc0 2023-08-25T09:50:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f8b7cb72640 1 -- 172.21.15.71:0/2488574747 learned_addr learned my addr 172.21.15.71:0/2488574747 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.216+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b781445b0 con 0x7f8b78069d90 2023-08-25T09:50:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.217+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b78150b20 con 0x7f8b780acd20 2023-08-25T09:50:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.217+0000 7f8b7d373640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b780acd20 0x7f8b781570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.217+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7814d1a0 con 0x7f8b7806abc0 2023-08-25T09:50:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.217+0000 7f11fb7fe640 1 --2- 172.21.15.71:0/257958587 >> v2:172.21.15.71:3300/0 conn(0x7f11fc15d090 0x7f11fc0c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c800edc0 msgr2=0x7fd8c8038900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2473874931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c800edc0 0x7fd8c8038900 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd8d004f120 tx=0x7fd8d00a5d80 comp rx=0 tx=0).stop 2023-08-25T09:50:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f8b7cb72640 1 -- 172.21.15.71:0/2488574747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b7806abc0 msgr2=0x7f8b78156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f8b7cb72640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b7806abc0 0x7f8b78156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f8b7cb72640 1 -- 172.21.15.71:0/2488574747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b780acd20 msgr2=0x7f8b781570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f8b7cb72640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b780acd20 0x7f8b781570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f8b7cb72640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68009400 con 0x7f8b78069d90 2023-08-25T09:50:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f8b77fff640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b7806abc0 0x7f8b78156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:06.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7f11fa7fc640 1 --2- 172.21.15.71:0/257958587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc158d40 0x7f11fc0c8ec0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f11fc073550 tx=0x7f11f402bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb0bf389cf172302 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 shutdown_connections 2023-08-25T09:50:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2473874931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c800edc0 0x7fd8c8038900 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2473874931 >> v2:172.21.15.164:3300/0 conn(0x7fd8c80d9340 0x7fd8c80df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.218+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 >> 172.21.15.71:0/2473874931 conn(0x7fd8c800e150 msgr2=0x7fd8c800e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f8b7cb72640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b78069d90 0x7f8b781501b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8b70001a70 tx=0x7f8b70001f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 shutdown_connections 2023-08-25T09:50:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f11f9ffb640 1 -- 172.21.15.71:0/257958587 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f402caa0 con 0x7f11fc158d40 2023-08-25T09:50:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000d6c0 con 0x7f8b78069d90 2023-08-25T09:50:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7000d820 con 0x7f8b78069d90 2023-08-25T09:50:06.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000dbf0 con 0x7f8b78069d90 2023-08-25T09:50:06.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f11f9ffb640 1 -- 172.21.15.71:0/257958587 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f402cc00 con 0x7f11fc158d40 2023-08-25T09:50:06.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f11f9ffb640 1 -- 172.21.15.71:0/257958587 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f40343f0 con 0x7f11fc158d40 2023-08-25T09:50:06.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7fd8d7539640 1 -- 172.21.15.71:0/2473874931 wait complete. 2023-08-25T09:50:06.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.219+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b78156720 con 0x7f8b78069d90 2023-08-25T09:50:06.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b781442d0 con 0x7f8b78069d90 2023-08-25T09:50:06.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 -- 172.21.15.71:0/257958587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc158d40 msgr2=0x7f11fc0c8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 --2- 172.21.15.71:0/257958587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc158d40 0x7f11fc0c8ec0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f11fc073550 tx=0x7f11f402bf00 comp rx=0 tx=0).stop 2023-08-25T09:50:06.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 -- 172.21.15.71:0/257958587 shutdown_connections 2023-08-25T09:50:06.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 --2- 172.21.15.71:0/257958587 >> v2:172.21.15.71:3300/0 conn(0x7f11fc15d090 0x7f11fc0c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 --2- 172.21.15.71:0/257958587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc158d40 0x7f11fc0c8ec0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 -- 172.21.15.71:0/257958587 >> 172.21.15.71:0/257958587 conn(0x7f11fc0654f0 msgr2=0x7f11fc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 -- 172.21.15.71:0/257958587 shutdown_connections 2023-08-25T09:50:06.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.220+0000 7f1201287640 1 -- 172.21.15.71:0/257958587 wait complete. 2023-08-25T09:50:06.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.222+0000 7fd8d7539640 1 Processor -- start 2023-08-25T09:50:06.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.222+0000 7fd8d7539640 1 -- start start 2023-08-25T09:50:06.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.223+0000 7f1201287640 1 Processor -- start 2023-08-25T09:50:06.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.224+0000 7f1201287640 1 -- start start 2023-08-25T09:50:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.224+0000 7fd8d7539640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c800edc0 0x7fd8c8179790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d7539640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c80d8f60 0x7fd8c80c3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d7539640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8c80069e0 0x7fd8c817cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d7539640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c8006e70 con 0x7fd8c80d8f60 2023-08-25T09:50:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d4ad4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c80d8f60 0x7fd8c80c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d4ad4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c80d8f60 0x7fd8c80c3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60080/0 (socket says 172.21.15.71:60080) 2023-08-25T09:50:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d4ad4640 1 -- 172.21.15.71:0/2140637585 learned_addr learned my addr 172.21.15.71:0/2140637585 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d52d5640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c800edc0 0x7fd8c8179790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7fd8d5ad6640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8c80069e0 0x7fd8c817cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.223+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8b7000dd50 con 0x7f8b78069d90 2023-08-25T09:50:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.224+0000 7f8b757fa640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8b6006a620 0x7f8b6006cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.224+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f8b700883a0 con 0x7f8b78069d90 2023-08-25T09:50:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.225+0000 7f8b77fff640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8b6006a620 0x7f8b6006cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.226+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c817d100 con 0x7fd8c800edc0 2023-08-25T09:50:06.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.226+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c80cd100 con 0x7fd8c80069e0 2023-08-25T09:50:06.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.226+0000 7f1201287640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11fc0c9730 0x7f11fc1f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f8b77fff640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8b6006a620 0x7f8b6006cae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b68036000 tx=0x7f8b68008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f8b7edd6640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] conn(0x7f8b780ac6d0 0x7f8b78153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f11faffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11fc0c9730 0x7f11fc1f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f11faffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11fc0c9730 0x7f11fc1f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60088/0 (socket says 172.21.15.71:60088) 2023-08-25T09:50:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f11faffd640 1 -- 172.21.15.71:0/3622927953 learned_addr learned my addr 172.21.15.71:0/3622927953 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b78154fe0 con 0x7f8b780ac6d0 2023-08-25T09:50:06.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f1201287640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc0c5b80 0x7f11fc1ef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d4ad4640 1 -- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8c80069e0 msgr2=0x7fd8c817cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d4ad4640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8c80069e0 0x7fd8c817cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d4ad4640 1 -- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c800edc0 msgr2=0x7fd8c8179790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d4ad4640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c800edc0 0x7fd8c8179790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d4ad4640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0009400 con 0x7fd8c80d8f60 2023-08-25T09:50:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f1201287640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11fc14b350 0x7f11fc15b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7f1201287640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc14b7d0 con 0x7f11fc0c9730 2023-08-25T09:50:06.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d52d5640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c800edc0 0x7fd8c8179790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7fd8d5ad6640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8c80069e0 0x7fd8c817cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.227+0000 7f8b7d373640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] conn(0x7f8b780ac6d0 0x7f8b78153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7f1201287640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc146d00 con 0x7f11fc0c5b80 2023-08-25T09:50:06.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7f1201287640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1eefb0 con 0x7f11fc14b350 2023-08-25T09:50:06.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7f11fa7fc640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc0c5b80 0x7f11fc1ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7fd8d4ad4640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c80d8f60 0x7fd8c80c3760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000e910 tx=0x7fd8c000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7f11fb7fe640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11fc14b350 0x7f11fc15b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f8b7d373640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] conn(0x7f8b780ac6d0 0x7f8b78153f00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c000c9a0 con 0x7fd8c80d8f60 2023-08-25T09:50:06.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c000cb00 con 0x7fd8c80d8f60 2023-08-25T09:50:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.228+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c000ce70 con 0x7fd8c80d8f60 2023-08-25T09:50:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.230+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d00721d0 con 0x7fd8c80d8f60 2023-08-25T09:50:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.230+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c80cd3a0 con 0x7fd8c80d8f60 2023-08-25T09:50:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.230+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== osd.0 v2:172.21.15.71:6808/555709626 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f8b6c001330 con 0x7f8b780ac6d0 2023-08-25T09:50:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f11faffd640 1 -- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11fc14b350 msgr2=0x7f11fc15b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f11faffd640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f11fc14b350 0x7f11fc15b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f11faffd640 1 -- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc0c5b80 msgr2=0x7f11fc1ef440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f11faffd640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f11fc0c5b80 0x7f11fc1ef440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f11faffd640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f4009b30 con 0x7f11fc0c9730 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.230+0000 7f11faffd640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11fc0c9730 0x7f11fc1f2840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f000b480 tx=0x7f11f000b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0010870 con 0x7f11fc0c9730 2023-08-25T09:50:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f00109d0 con 0x7f11fc0c9730 2023-08-25T09:50:06.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0010d80 con 0x7f11fc0c9730 2023-08-25T09:50:06.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f1201287640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc1ee9d0 con 0x7f11fc0c9730 2023-08-25T09:50:06.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f0d94c83640 1 Processor -- start 2023-08-25T09:50:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.229+0000 7f0d94c83640 1 -- start start 2023-08-25T09:50:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f0d94c83640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0d9015dd50 0x7f0d9006a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f0d94c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d90069850 0x7f0d90069c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f0d94c83640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d900c55d0 con 0x7f0d900bffa0 2023-08-25T09:50:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f0d94c83640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d900bfb10 con 0x7f0d9015dd50 2023-08-25T09:50:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.231+0000 7f0d94c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900adf80 con 0x7f0d90069850 2023-08-25T09:50:06.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.232+0000 7f72357ab640 1 Processor -- start 2023-08-25T09:50:06.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.232+0000 7f72357ab640 1 -- start start 2023-08-25T09:50:06.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.232+0000 7f1201287640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc1eec40 con 0x7f11fc0c9730 2023-08-25T09:50:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.232+0000 7f0d8ed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0d900bffa0 0x7f0d900c0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:57148/0 (socket says 172.21.15.71:57148) 2023-08-25T09:50:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.232+0000 7f0d8ed9d640 1 -- 172.21.15.71:0/2073167822 learned_addr learned my addr 172.21.15.71:0/2073167822 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.233+0000 7f0d8dd9b640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d90069850 0x7f0d90069c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.233+0000 7f0d8e59c640 1 --2- 172.21.15.71:0/2073167822 >> v2:172.21.15.164:3300/0 conn(0x7f0d9015dd50 0x7f0d9006a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.233+0000 7f6b8ebd6640 1 Processor -- start 2023-08-25T09:50:06.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11fc159a50 con 0x7f11fc0c9730 2023-08-25T09:50:06.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f72357ab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300ad090 0x7f72300c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f72357ab640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f723006b700 0x7f7230163850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f1c4a3b5640 1 Processor -- start 2023-08-25T09:50:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f11f000f3c0 con 0x7f11fc0c9730 2023-08-25T09:50:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f11e37fe640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f11e806a0a0 0x7f11e806c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f11fa7fc640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f11e806a0a0 0x7f11e806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd8c0013470 con 0x7fd8c80d8f60 2023-08-25T09:50:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7fd8c5ffb640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd8b406a150 0x7fd8b406c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd8c0088aa0 con 0x7fd8c80d8f60 2023-08-25T09:50:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7fd8d52d5640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd8b406a150 0x7fd8b406c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f6b8ebd6640 1 -- start start 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f6b8ebd6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b800dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f6b8ebd6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6b80034730 0x7f6b80034ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f6b8ebd6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b80006670 con 0x7f6b80005ec0 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f6b8ebd6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b80033e80 con 0x7f6b80034730 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f6b8ebd6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b80006910 con 0x7f6b800d4d70 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f6b8c972640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b800dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f6b8c972640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b800dabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41414/0 (socket says 172.21.15.71:41414) 2023-08-25T09:50:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f6b8c972640 1 -- 172.21.15.71:0/1801091135 learned_addr learned my addr 172.21.15.71:0/1801091135 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f6b8d173640 1 --2- 172.21.15.71:0/1801091135 >> v2:172.21.15.164:3300/0 conn(0x7f6b80034730 0x7f6b80034ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.233+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398868648 0 0) 0x7f0d900c55d0 con 0x7f0d900bffa0 2023-08-25T09:50:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.233+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d78003520 con 0x7f0d900bffa0 2023-08-25T09:50:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2927670660 0 0) 0x7f0d78003520 con 0x7f0d900bffa0 2023-08-25T09:50:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d90069850 msgr2=0x7f0d90069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d90069850 0x7f0d90069c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 >> v2:172.21.15.164:3300/0 conn(0x7f0d9015dd50 msgr2=0x7f0d9006a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 --2- 172.21.15.71:0/2073167822 >> v2:172.21.15.164:3300/0 conn(0x7f0d9015dd50 0x7f0d9006a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9015e130 con 0x7f0d900bffa0 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0d880019f0 con 0x7f0d900bffa0 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 >> v1:172.21.15.164:6789/0 conn(0x7f0d900bffa0 legacy=0x7f0d900c0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78003970 0x7f0d78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d780047c0 0x7f0d78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900bfb10 con 0x7f0d780047c0 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900adf80 con 0x7f0d78003970 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.234+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7800c000 con 0x7f0d78005180 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f0d8ed9d640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f0d8e59c640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78003970 0x7f0d78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f0d8ed9d640 1 -- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78003970 msgr2=0x7f0d78003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f0d8ed9d640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78003970 0x7f0d78003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f0d8ed9d640 1 -- 172.21.15.71:0/2073167822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d780047c0 msgr2=0x7f0d78004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-08-25T09:50:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f0d8ed9d640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d780047c0 0x7f0d78004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f0d8ed9d640 1 -- 172.21.15.71:0/2073167822 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78003520 con 0x7f0d78005180 2023-08-25T09:50:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f72357ab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72300c50a0 con 0x7f72300accb0 2023-08-25T09:50:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f72357ab640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7230159b60 con 0x7f723006b700 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f72357ab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723006a940 con 0x7f72300ad090 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f722ffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f72300accb0 0x7f72300bfb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46408/0 (socket says 172.21.15.71:46408) 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.236+0000 7f722ffff640 1 -- 172.21.15.71:0/612275477 learned_addr learned my addr 172.21.15.71:0/612275477 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f7234faa640 1 --2- 172.21.15.71:0/612275477 >> v2:172.21.15.71:3300/0 conn(0x7f723006b700 0x7f7230163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471803503 0 0) 0x7f72300c50a0 con 0x7f72300accb0 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7218003520 con 0x7f72300accb0 2023-08-25T09:50:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f722f7fe640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300ad090 0x7f72300c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2293868668 0 0) 0x7f7218003520 con 0x7f72300accb0 2023-08-25T09:50:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300ad090 msgr2=0x7f72300c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.235+0000 7f1c4a3b5640 1 -- start start 2023-08-25T09:50:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f1c4a3b5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1c44069850 0x7f1c44069c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f1c4a3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c440c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f1c4a3b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440c55d0 con 0x7f1c4415dd50 2023-08-25T09:50:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f1c4a3b5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440bfb10 con 0x7f1c44069850 2023-08-25T09:50:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.237+0000 7f1c4a3b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440adf80 con 0x7f1c440bffa0 2023-08-25T09:50:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f11f005ab60 con 0x7f11fc0c9730 2023-08-25T09:50:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300ad090 0x7f72300c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 >> v2:172.21.15.71:3300/0 conn(0x7f723006b700 msgr2=0x7f7230163850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 --2- 172.21.15.71:0/612275477 >> v2:172.21.15.71:3300/0 conn(0x7f723006b700 0x7f7230163850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723006ac20 con 0x7f72300accb0 2023-08-25T09:50:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f11e37fe640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] conn(0x7f11e8073d80 0x7f11e80761c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11e8076890 con 0x7f11e8073d80 2023-08-25T09:50:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f0053190 con 0x7f11fc0c9730 2023-08-25T09:50:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f6b877fe640 1 -- 172.21.15.71:0/1801091135 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384845116 0 0) 0x7f6b80006670 con 0x7f6b80005ec0 2023-08-25T09:50:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f6b877fe640 1 -- 172.21.15.71:0/1801091135 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78003520 con 0x7f6b80005ec0 2023-08-25T09:50:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f0d8ed9d640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d7800b980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800a7b0 tx=0x7f0d780057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9e279d09fcff547 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88034630 con 0x7f0d78005180 2023-08-25T09:50:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8802f070 con 0x7f0d78005180 2023-08-25T09:50:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f0d8d59a640 1 -- 172.21.15.71:0/2073167822 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d880349f0 con 0x7f0d78005180 2023-08-25T09:50:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f7220001670 con 0x7f72300accb0 2023-08-25T09:50:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 >> v1:172.21.15.71:6789/0 conn(0x7f72300accb0 legacy=0x7f72300bfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f1c437fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1c44069850 0x7f1c44069c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f1c437fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1c44069850 0x7f1c44069c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60110/0 (socket says 172.21.15.71:60110) 2023-08-25T09:50:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f1c437fe640 1 -- 172.21.15.71:0/2844522321 learned_addr learned my addr 172.21.15.71:0/2844522321 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f1c48952640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c440c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f1c437fe640 1 -- 172.21.15.71:0/2844522321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 msgr2=0x7f1c440c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7fd8d52d5640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd8b406a150 0x7fd8b406c610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc001030 tx=0x7fd8cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f6b8d173640 1 -- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 msgr2=0x7f6b800dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f6b8d173640 1 --2- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b800dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f6b8d173640 1 -- 172.21.15.71:0/1801091135 >> v1:172.21.15.164:6789/0 conn(0x7f6b80005ec0 legacy=0x7f6b8004b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.238+0000 7f6b8d173640 1 -- 172.21.15.71:0/1801091135 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b80006bb0 con 0x7f6b80034730 2023-08-25T09:50:06.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f1c437fe640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c440c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72180039e0 0x7f7218003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72180047c0 0x7f7218004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218005180 0x7f721800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f1c437fe640 1 -- 172.21.15.71:0/2844522321 >> v1:172.21.15.164:6789/0 conn(0x7f1c4415dd50 legacy=0x7f1c4406a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7230159b60 con 0x7f7218005180 2023-08-25T09:50:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723006a940 con 0x7f72180047c0 2023-08-25T09:50:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.239+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721800c050 con 0x7f72180039e0 2023-08-25T09:50:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d94c83640 1 -- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 msgr2=0x7f0d7800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d94c83640 1 --2- 172.21.15.71:0/2073167822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d7800b980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800a7b0 tx=0x7f0d780057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f1c437fe640 1 -- 172.21.15.71:0/2844522321 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440c0950 con 0x7f1c44069850 2023-08-25T09:50:06.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d8ed9d640 1 -- 172.21.15.71:0/2073167822 reap_dead start 2023-08-25T09:50:06.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f722f7fe640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300ad090 0x7f72300c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f11fb7fe640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] conn(0x7f11e8073d80 0x7f11e80761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f1c48952640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c440c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] conn(0x7fd8c8007540 0x7fd8c80d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8c804f610 con 0x7fd8c8007540 2023-08-25T09:50:06.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f7234faa640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218005180 0x7f721800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d94c83640 1 -- 172.21.15.71:0/2073167822 shutdown_connections 2023-08-25T09:50:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d94c83640 1 -- 172.21.15.71:0/2073167822 >> 172.21.15.71:0/2073167822 conn(0x7f0d900c6cc0 msgr2=0x7f0d900c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d94c83640 1 -- 172.21.15.71:0/2073167822 shutdown_connections 2023-08-25T09:50:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7f0d94c83640 1 -- 172.21.15.71:0/2073167822 wait complete. 2023-08-25T09:50:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f11fa7fc640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f11e806a0a0 0x7f11e806c560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f11f4035000 tx=0x7f11f4043040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f6b8d173640 1 --2- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 0x7f6b80034ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0097a0 tx=0x7f6b7c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=464b9b2782415e9f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.240+0000 7fd8d5ad6640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] conn(0x7fd8c8007540 0x7fd8c80d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f11fb7fe640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] conn(0x7f11e8073d80 0x7f11e80761c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7fd8d5ad6640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] conn(0x7fd8c8007540 0x7fd8c80d2fe0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f7234faa640 1 -- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72180039e0 msgr2=0x7f7218003e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T09:50:06.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f7234faa640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72180039e0 0x7f7218003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f7234faa640 1 -- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72180047c0 msgr2=0x7f7218004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-08-25T09:50:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f7234faa640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72180047c0 0x7f7218004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.241+0000 7f7234faa640 1 -- 172.21.15.71:0/612275477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723006ac20 con 0x7f7218005180 2023-08-25T09:50:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.242+0000 7f6b877fe640 1 -- 172.21.15.71:0/1801091135 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c031830 con 0x7f6b80034730 2023-08-25T09:50:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.242+0000 7f6b877fe640 1 -- 172.21.15.71:0/1801091135 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c031990 con 0x7f6b80034730 2023-08-25T09:50:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.242+0000 7f6b877fe640 1 -- 172.21.15.71:0/1801091135 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c031d60 con 0x7f6b80034730 2023-08-25T09:50:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.242+0000 7f1c437fe640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c44069850 0x7f1c44069c90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ff514e3a6abb6f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 msgr2=0x7f6b80034ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f6b8ebd6640 1 --2- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 0x7f6b80034ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0097a0 tx=0x7f6b7c02b8d0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.242+0000 7f7234faa640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218005180 0x7f721800b980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7230073ab0 tx=0x7f72180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50e2b24f62f1e31 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== osd.2 v2:172.21.15.71:6800/3037968157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fd8d00a69b0 con 0x7fd8c8007540 2023-08-25T09:50:06.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f1c42ffd640 1 -- 172.21.15.71:0/2844522321 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802f840 con 0x7f1c44069850 2023-08-25T09:50:06.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722802cbd0 con 0x7f7218005180 2023-08-25T09:50:06.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f0d94c83640 1 Processor -- start 2023-08-25T09:50:06.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722800f040 con 0x7f7218005180 2023-08-25T09:50:06.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f1c42ffd640 1 -- 172.21.15.71:0/2844522321 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c3802f9a0 con 0x7f1c44069850 2023-08-25T09:50:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== osd.1 v2:172.21.15.71:6804/3242507061 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f11e8076890 con 0x7f11e8073d80 2023-08-25T09:50:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1801091135 shutdown_connections 2023-08-25T09:50:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f6b8ebd6640 1 --2- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 0x7f6b80034ba0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f6b8ebd6640 1 --2- 172.21.15.71:0/1801091135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b800dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1801091135 >> 172.21.15.71:0/1801091135 conn(0x7f6b800501b0 msgr2=0x7f6b800505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f722effd640 1 -- 172.21.15.71:0/612275477 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228034440 con 0x7f7218005180 2023-08-25T09:50:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f72357ab640 1 -- 172.21.15.71:0/612275477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218005180 msgr2=0x7f721800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f72357ab640 1 --2- 172.21.15.71:0/612275477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7218005180 0x7f721800b980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7230073ab0 tx=0x7f72180057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c42ffd640 1 -- 172.21.15.71:0/2844522321 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3802f840 con 0x7f1c44069850 2023-08-25T09:50:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2844522321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c44069850 msgr2=0x7f1c44069c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c4a3b5640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c44069850 0x7f1c44069c90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802ba20 comp rx=0 tx=0).stop 2023-08-25T09:50:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f7234faa640 1 -- 172.21.15.71:0/612275477 reap_dead start 2023-08-25T09:50:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f72357ab640 1 -- 172.21.15.71:0/612275477 shutdown_connections 2023-08-25T09:50:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f72357ab640 1 -- 172.21.15.71:0/612275477 >> 172.21.15.71:0/612275477 conn(0x7f723006c280 msgr2=0x7f723006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2844522321 shutdown_connections 2023-08-25T09:50:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c4a3b5640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c440c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c4a3b5640 1 --2- 172.21.15.71:0/2844522321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c44069850 0x7f1c44069c90 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1c38009870 tx=0x7f1c3802ba20 comp rx=0 tx=0).stop 2023-08-25T09:50:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.244+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2844522321 >> 172.21.15.71:0/2844522321 conn(0x7f1c440c6cc0 msgr2=0x7f1c440c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f72357ab640 1 -- 172.21.15.71:0/612275477 shutdown_connections 2023-08-25T09:50:06.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1801091135 shutdown_connections 2023-08-25T09:50:06.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.243+0000 7f0d94c83640 1 -- start start 2023-08-25T09:50:06.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2844522321 shutdown_connections 2023-08-25T09:50:06.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2844522321 wait complete. 2023-08-25T09:50:06.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f4ed52e7640 1 Processor -- start 2023-08-25T09:50:06.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f4ed52e7640 1 -- start start 2023-08-25T09:50:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f0d94c83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d9013b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.246+0000 7f0d94c83640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d9014c6b0 0x7f0d90148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.246+0000 7f0d94c83640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d9013b730 0x7f0d901491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.246+0000 7f0d94c83640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900aa660 con 0x7f0d9014c6b0 2023-08-25T09:50:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.246+0000 7f0d94c83640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9013a470 con 0x7f0d9013b730 2023-08-25T09:50:06.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.246+0000 7f0d94c83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9014cb40 con 0x7f0d78005180 2023-08-25T09:50:06.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1801091135 wait complete. 2023-08-25T09:50:06.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.247+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d9013b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.247+0000 7f0d8e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d9013b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41458/0 (socket says 172.21.15.71:41458) 2023-08-25T09:50:06.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.247+0000 7f0d8e59c640 1 -- 172.21.15.71:0/1803605650 learned_addr learned my addr 172.21.15.71:0/1803605650 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.247+0000 7f1c4a3b5640 1 Processor -- start 2023-08-25T09:50:06.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f6b8ebd6640 1 Processor -- start 2023-08-25T09:50:06.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.245+0000 7f72357ab640 1 -- 172.21.15.71:0/612275477 wait complete. 2023-08-25T09:50:06.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f72357ab640 1 Processor -- start 2023-08-25T09:50:06.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f0d8e59c640 1 -- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d9013b730 msgr2=0x7f0d901491b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-08-25T09:50:06.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f0d8e59c640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d9013b730 0x7f0d901491b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f0d8e59c640 1 -- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d9014c6b0 msgr2=0x7f0d90148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f0d8e59c640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d9014c6b0 0x7f0d90148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f0d8e59c640 1 -- 172.21.15.71:0/1803605650 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9014ce00 con 0x7f0d78005180 2023-08-25T09:50:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f4ed52e7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ed0152cd0 0x7f4ed015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f0d8e59c640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d9013b1f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400b7f0 tx=0x7f0d8400bcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f1c4a3b5640 1 -- start start 2023-08-25T09:50:06.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f72357ab640 1 -- start start 2023-08-25T09:50:06.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84010830 con 0x7f0d78005180 2023-08-25T09:50:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f4ed52e7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ed00ca5a0 0x7f4ed00c9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d84010990 con 0x7f0d78005180 2023-08-25T09:50:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84010d20 con 0x7f0d78005180 2023-08-25T09:50:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9013a710 con 0x7f0d78005180 2023-08-25T09:50:06.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.248+0000 7f6b8ebd6640 1 -- start start 2023-08-25T09:50:06.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f6b8ebd6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b80005ec0 0x7f6b800d5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9013acd0 con 0x7f0d78005180 2023-08-25T09:50:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f4eceffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ed0152cd0 0x7f4ed015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f4eceffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ed0152cd0 0x7f4ed015cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56118/0 (socket says 172.21.15.71:56118) 2023-08-25T09:50:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f4eceffd640 1 -- 172.21.15.71:0/893970841 learned_addr learned my addr 172.21.15.71:0/893970841 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.249+0000 7f4ece7fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4ed00c3470 0x7f4ed0158c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:57184/0 (socket says 172.21.15.71:57184) 2023-08-25T09:50:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f6b8c972640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b80005ec0 0x7f6b800d5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f6b8c972640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b80005ec0 0x7f6b800d5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60132/0 (socket says 172.21.15.71:60132) 2023-08-25T09:50:06.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f1c4a3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c4415c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f1c4a3b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c4415dd50 0x7f1c440aa570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed015c210 con 0x7f4ed00c3470 2023-08-25T09:50:06.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed015c7d0 con 0x7f4ed00ca5a0 2023-08-25T09:50:06.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed015d1a0 con 0x7f4ed0152cd0 2023-08-25T09:50:06.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.250+0000 7f1c4a3b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c44161020 0x7f1c440aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/893970841 >> v2:172.21.15.71:3300/0 conn(0x7f4ed00ca5a0 0x7f4ed00c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c4a3b5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440a9cf0 con 0x7f1c4415dd50 2023-08-25T09:50:06.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860853052 0 0) 0x7f4ed015c210 con 0x7f4ed00c3470 2023-08-25T09:50:06.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb8003520 con 0x7f4ed00c3470 2023-08-25T09:50:06.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f6b8c972640 1 -- 172.21.15.71:0/1831980605 learned_addr learned my addr 172.21.15.71:0/1831980605 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d900c5b00 con 0x7f0d78005180 2023-08-25T09:50:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c437fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c4415dd50 0x7f1c440aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c437fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c4415dd50 0x7f1c440aa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60148/0 (socket says 172.21.15.71:60148) 2023-08-25T09:50:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c437fe640 1 -- 172.21.15.71:0/2779091893 learned_addr learned my addr 172.21.15.71:0/2779091893 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440c80e0 con 0x7f1c44161020 2023-08-25T09:50:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c440a9a10 con 0x7f1c440bffa0 2023-08-25T09:50:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f1c48952640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c44161020 0x7f1c440aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.251+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593997467 0 0) 0x7f4eb8003520 con 0x7f4ed00c3470 2023-08-25T09:50:06.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f72357ab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300adc20 0x7f72300c2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f72357ab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72300ae000 0x7f72300c25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 >> v2:172.21.15.164:3300/0 conn(0x7f4ed0152cd0 msgr2=0x7f4ed015cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 --2- 172.21.15.71:0/893970841 >> v2:172.21.15.164:3300/0 conn(0x7f4ed0152cd0 0x7f4ed015cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 >> v2:172.21.15.71:3300/0 conn(0x7f4ed00ca5a0 msgr2=0x7f4ed00c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 --2- 172.21.15.71:0/893970841 >> v2:172.21.15.71:3300/0 conn(0x7f4ed00ca5a0 0x7f4ed00c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed01530b0 con 0x7f4ed00c3470 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4ec0001670 con 0x7f4ed00c3470 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 >> v1:172.21.15.164:6789/0 conn(0x7f4ed00c3470 legacy=0x7f4ed0158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4eb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb80047c0 0x7f4eb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb8005180 0x7f4eb800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed015c7d0 con 0x7f4eb8003970 2023-08-25T09:50:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed015d1a0 con 0x7f4eb80047c0 2023-08-25T09:50:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb800c050 con 0x7f4eb8005180 2023-08-25T09:50:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/893970841 >> v2:172.21.15.71:3300/0 conn(0x7f4ed00ca5a0 0x7f4ed00c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:50:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f72357ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72300abc70 0x7f723014ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f72357ab640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72300ac580 con 0x7f72300ae000 2023-08-25T09:50:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f1c437fe640 1 -- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 msgr2=0x7f1c4415c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f1c437fe640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c440bffa0 0x7f1c4415c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f1c437fe640 1 -- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c44161020 msgr2=0x7f1c440aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f1c437fe640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c44161020 0x7f1c440aaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f1c437fe640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c380094d0 con 0x7f1c4415dd50 2023-08-25T09:50:06.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.252+0000 7f1c437fe640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c4415dd50 0x7f1c440aa570 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c380090b0 tx=0x7f1c380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f72357ab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72300c15c0 con 0x7f72300abc70 2023-08-25T09:50:06.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f6b8ebd6640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 0x7f6b800d3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f722ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300adc20 0x7f72300c2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f722ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300adc20 0x7f72300c2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41466/0 (socket says 172.21.15.71:41466) 2023-08-25T09:50:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f722ffff640 1 -- 172.21.15.71:0/2619725463 learned_addr learned my addr 172.21.15.71:0/2619725463 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb8005180 0x7f4eb800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f6b8ebd6640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b8017bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3803b040 con 0x7f1c4415dd50 2023-08-25T09:50:06.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c38003a40 con 0x7f1c4415dd50 2023-08-25T09:50:06.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38003df0 con 0x7f1c4415dd50 2023-08-25T09:50:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800d5c00 con 0x7f6b80005ec0 2023-08-25T09:50:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b800d5690 con 0x7f6b80034730 2023-08-25T09:50:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b80178850 con 0x7f6b800d4d70 2023-08-25T09:50:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b87fff640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 0x7f6b800d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b8d173640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b8017bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b8c972640 1 -- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 msgr2=0x7f6b8017bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b8c972640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b800d4d70 0x7f6b8017bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b8c972640 1 -- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 msgr2=0x7f6b800d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b8c972640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b80034730 0x7f6b800d3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f6b8c972640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7c009400 con 0x7f6b80005ec0 2023-08-25T09:50:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f6b8c972640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b80005ec0 0x7f6b800d5150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400b6c0 tx=0x7f6b7400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f722f7fe640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72300ae000 0x7f72300c25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f7234faa640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72300abc70 0x7f723014ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72301f3150 con 0x7f72300adc20 2023-08-25T09:50:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c44144aa0 con 0x7f1c4415dd50 2023-08-25T09:50:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f1c4a3b5640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c440c8310 con 0x7f1c4415dd50 2023-08-25T09:50:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c440c5820 con 0x7f1c4415dd50 2023-08-25T09:50:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f4eceffd640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4eb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.253+0000 7f4ece7fc640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb80047c0 0x7f4eb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f4eceffd640 1 -- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb8005180 msgr2=0x7f4eb800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f4eceffd640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb8005180 0x7f4eb800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f4eceffd640 1 -- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb80047c0 msgr2=0x7f4eb8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f4eceffd640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb80047c0 0x7f4eb8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f4eceffd640 1 -- 172.21.15.71:0/893970841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8003520 con 0x7f4eb8003970 2023-08-25T09:50:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f4ecf7fe640 1 -- 172.21.15.71:0/893970841 reap_dead start 2023-08-25T09:50:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f7234faa640 1 -- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300adc20 msgr2=0x7f72300c2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f7234faa640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300adc20 0x7f72300c2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f7234faa640 1 -- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72300ae000 msgr2=0x7f72300c25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f7234faa640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72300ae000 0x7f72300c25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.255+0000 7f7234faa640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72300c1860 con 0x7f72300abc70 2023-08-25T09:50:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f722ffff640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300adc20 0x7f72300c2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f722f7fe640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f72300ae000 0x7f72300c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74010040 con 0x7f6b80005ec0 2023-08-25T09:50:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7400e760 con 0x7f6b80005ec0 2023-08-25T09:50:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400eaf0 con 0x7f6b80005ec0 2023-08-25T09:50:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.254+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0d84027480 con 0x7f0d78005180 2023-08-25T09:50:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f0d7effd640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d7006a050 0x7f0d7006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0d840895e0 con 0x7f0d78005180 2023-08-25T09:50:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f0d7effd640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] conn(0x7f0d70073d30 0x7f0d70076170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 --> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d70076840 con 0x7f0d70073d30 2023-08-25T09:50:06.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d84089990 con 0x7f0d78005180 2023-08-25T09:50:06.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f4eceffd640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4eb8003d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800e1c0 tx=0x7f4eb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f851885200572e8d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f0d8dd9b640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d7006a050 0x7f0d7006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.256+0000 7f7234faa640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72300abc70 0x7f723014ed90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f721800bfa0 tx=0x7f7228009750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722802dea0 con 0x7f72300abc70 2023-08-25T09:50:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7228035d60 con 0x7f72300abc70 2023-08-25T09:50:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722803e610 con 0x7f72300abc70 2023-08-25T09:50:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f0d8ed9d640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] conn(0x7f0d70073d30 0x7f0d70076170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f0d8dd9b640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d7006a050 0x7f0d7006c510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d900c4830 tx=0x7f0d80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec802caa0 con 0x7f4eb8003970 2023-08-25T09:50:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec800f040 con 0x7f4eb8003970 2023-08-25T09:50:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f4ecdffb640 1 -- 172.21.15.71:0/893970841 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec802ce20 con 0x7f4eb8003970 2023-08-25T09:50:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 msgr2=0x7f4eb8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4eb8003d90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800e1c0 tx=0x7f4eb80057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 shutdown_connections 2023-08-25T09:50:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 --2- 172.21.15.71:0/893970841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4eb8003d90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 >> 172.21.15.71:0/893970841 conn(0x7f4ed006add0 msgr2=0x7f4ed006b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 shutdown_connections 2023-08-25T09:50:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72300c1b40 con 0x7f72300abc70 2023-08-25T09:50:06.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.259+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7230152730 con 0x7f72300abc70 2023-08-25T09:50:06.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f4ed52e7640 1 -- 172.21.15.71:0/893970841 wait complete. 2023-08-25T09:50:06.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.259+0000 7f0d8ed9d640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] conn(0x7f0d70073d30 0x7f0d70076170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.257+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b80178b30 con 0x7f6b80005ec0 2023-08-25T09:50:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f6b8ebd6640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b801790f0 con 0x7f6b80005ec0 2023-08-25T09:50:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.259+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b80006e50 con 0x7f6b80005ec0 2023-08-25T09:50:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.258+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1c38042480 con 0x7f1c4415dd50 2023-08-25T09:50:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.260+0000 7f1c40ff9640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c3006a150 0x7f1c3006c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.260+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f1c380ab6c0 con 0x7f1c4415dd50 2023-08-25T09:50:06.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.260+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f723006b1e0 con 0x7f72300abc70 2023-08-25T09:50:06.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.260+0000 7f1c40ff9640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] conn(0x7f1c30073e30 0x7f1c30076270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.260+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c30076940 con 0x7f1c30073e30 2023-08-25T09:50:06.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.260+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c380aba70 con 0x7f1c4415dd50 2023-08-25T09:50:06.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.261+0000 7f1c43fff640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c3006a150 0x7f1c3006c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.261+0000 7f4ed52e7640 1 Processor -- start 2023-08-25T09:50:06.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.261+0000 7f4ed52e7640 1 -- start start 2023-08-25T09:50:06.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.261+0000 7f1c48952640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] conn(0x7f1c30073e30 0x7f1c30076270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f1c43fff640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c3006a150 0x7f1c3006c610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c34000f60 tx=0x7f1c340079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== osd.4 v2:172.21.15.164:6800/2125881619 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f0d70076840 con 0x7f0d70073d30 2023-08-25T09:50:06.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f1c48952640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] conn(0x7f1c30073e30 0x7f1c30076270 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f4ed52e7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4ed015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f4ed52e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed0151700 0x7f4ed0153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f4ed52e7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed0153a50 0x7f4ed014e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f4ed52e7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed014f490 con 0x7f4eb8003970 2023-08-25T09:50:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f4ed52e7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0158bf0 con 0x7f4ed0153a50 2023-08-25T09:50:06.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.262+0000 7f4ed52e7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0158950 con 0x7f4ed0151700 2023-08-25T09:50:06.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.263+0000 7f4ece7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed0151700 0x7f4ed0153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.263+0000 7f4ece7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed0151700 0x7f4ed0153510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41490/0 (socket says 172.21.15.71:41490) 2023-08-25T09:50:06.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.263+0000 7f4ece7fc640 1 -- 172.21.15.71:0/1196648597 learned_addr learned my addr 172.21.15.71:0/1196648597 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.263+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f722803b050 con 0x7f72300abc70 2023-08-25T09:50:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.263+0000 7f4eceffd640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4ed015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed0153a50 0x7f4ed014e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f4ecf7fe640 1 -- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed0151700 msgr2=0x7f4ed0153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed0151700 0x7f4ed0153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f4ecf7fe640 1 -- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 msgr2=0x7f4ed015b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb8003970 0x7f4ed015b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f4ecf7fe640 1 -- 172.21.15.71:0/1196648597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed014d080 con 0x7f4ed0153a50 2023-08-25T09:50:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.264+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6b7400ec50 con 0x7f6b80005ec0 2023-08-25T09:50:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f722cff9640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a150 0x7f721c06c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f72280ab7a0 con 0x7f72300abc70 2023-08-25T09:50:06.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f6b857fa640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6b6c06a120 0x7f6b6c06c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f6b74089410 con 0x7f6b80005ec0 2023-08-25T09:50:06.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f722cff9640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] conn(0x7f721c073e30 0x7f721c076270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f721c076940 con 0x7f721c073e30 2023-08-25T09:50:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72280abb50 con 0x7f72300abc70 2023-08-25T09:50:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f722ffff640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a150 0x7f721c06c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== osd.6 v2:172.21.15.164:6824/4127242778 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1c30076940 con 0x7f1c30073e30 2023-08-25T09:50:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.265+0000 7f4ecf7fe640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed0153a50 0x7f4ed014e530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00c5820 tx=0x7f4ec4011d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 --> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b781555a0 con 0x7f8b780ac6d0 2023-08-25T09:50:06.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f722f7fe640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] conn(0x7f721c073e30 0x7f721c076270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f6b857fa640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] conn(0x7f6b6c073e00 0x7f6b6c076240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b6c076910 con 0x7f6b6c073e00 2023-08-25T09:50:06.342 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b740897c0 con 0x7f6b80005ec0 2023-08-25T09:50:06.342 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.266+0000 7f6b87fff640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6b6c06a120 0x7f6b6c06c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.342 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.267+0000 7f8b757fa640 1 -- 172.21.15.71:0/2488574747 <== osd.0 v2:172.21.15.71:6808/555709626 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8b6c0017c0 con 0x7f8b780ac6d0 2023-08-25T09:50:06.342 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.267+0000 7f722f7fe640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] conn(0x7f721c073e30 0x7f721c076270 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.267+0000 7f6b8d173640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] conn(0x7f6b6c073e00 0x7f6b6c076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.267+0000 7f6b87fff640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6b6c06a120 0x7f6b6c06c5e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b88000b60 tx=0x7f6b88061360 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.267+0000 7f722ffff640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a150 0x7f721c06c610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7220007c40 tx=0x7f7220009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] conn(0x7f8b780ac6d0 msgr2=0x7f8b78153f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7edd6640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6808/555709626,v1:172.21.15.71:6811/555709626] conn(0x7f8b780ac6d0 0x7f8b78153f00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f6b8d173640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] conn(0x7f6b6c073e00 0x7f6b6c076240 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8b6006a620 msgr2=0x7f8b6006cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7edd6640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8b6006a620 0x7f8b6006cae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b68036000 tx=0x7f8b68008030 comp rx=0 tx=0).stop 2023-08-25T09:50:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b78069d90 msgr2=0x7f8b781501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7edd6640 1 --2- 172.21.15.71:0/2488574747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b78069d90 0x7f8b781501b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8b70001a70 tx=0x7f8b70001f30 comp rx=0 tx=0).stop 2023-08-25T09:50:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.268+0000 7f8b7d373640 1 -- 172.21.15.71:0/2488574747 reap_dead start 2023-08-25T09:50:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400fa40 con 0x7f4ed0153a50 2023-08-25T09:50:06.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 shutdown_connections 2023-08-25T09:50:06.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 >> 172.21.15.71:0/2488574747 conn(0x7f8b780c6cc0 msgr2=0x7f8b780c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed01562b0 con 0x7f4ed0153a50 2023-08-25T09:50:06.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 shutdown_connections 2023-08-25T09:50:06.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed0156590 con 0x7f4ed0153a50 2023-08-25T09:50:06.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f8b7edd6640 1 -- 172.21.15.71:0/2488574747 wait complete. 2023-08-25T09:50:06.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.269+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== osd.5 v2:172.21.15.164:6808/1764416968 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f721c076940 con 0x7f721c073e30 2023-08-25T09:50:06.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.270+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== osd.3 v2:172.21.15.71:6824/1330919310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f6b6c076910 con 0x7f6b6c073e00 2023-08-25T09:50:06.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.270+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec400fba0 con 0x7f4ed0153a50 2023-08-25T09:50:06.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.271+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 --> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11fc159770 con 0x7f11e8073d80 2023-08-25T09:50:06.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.270+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 --> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8c804fbd0 con 0x7fd8c8007540 2023-08-25T09:50:06.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.271+0000 7fd8c5ffb640 1 -- 172.21.15.71:0/2140637585 <== osd.2 v2:172.21.15.71:6800/3037968157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd8d00a6b20 con 0x7fd8c8007540 2023-08-25T09:50:06.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.272+0000 7f11e37fe640 1 -- 172.21.15.71:0/3622927953 <== osd.1 v2:172.21.15.71:6804/3242507061 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f11ec00c040 con 0x7f11e8073d80 2023-08-25T09:50:06.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.271+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4004020 con 0x7f4ed0153a50 2023-08-25T09:50:06.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.272+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4ed014df40 con 0x7f4ed0153a50 2023-08-25T09:50:06.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.272+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] conn(0x7f11e8073d80 msgr2=0x7f11e80761c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.272+0000 7f11e17fa640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6804/3242507061,v1:172.21.15.71:6806/3242507061] conn(0x7f11e8073d80 0x7f11e80761c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.273+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4ec4004200 con 0x7f4ed0153a50 2023-08-25T09:50:06.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.275+0000 7f4eb77fe640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4ebc06a0a0 0x7f4ebc06c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.275+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4ec40887d0 con 0x7f4ed0153a50 2023-08-25T09:50:06.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.275+0000 7f4eb77fe640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] conn(0x7f4ebc073c10 0x7f4ebc076050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.275+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 --> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ebc076720 con 0x7f4ebc073c10 2023-08-25T09:50:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.275+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ec408adb0 con 0x7f4ed0153a50 2023-08-25T09:50:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f11e806a0a0 msgr2=0x7f11e806c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11e17fa640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f11e806a0a0 0x7f11e806c560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f11f4035000 tx=0x7f11f4043040 comp rx=0 tx=0).stop 2023-08-25T09:50:06.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f4eceffd640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4ebc06a0a0 0x7f4ebc06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11fc0c9730 msgr2=0x7f11fc1f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11e17fa640 1 --2- 172.21.15.71:0/3622927953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f11fc0c9730 0x7f11fc1f2840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f11f000b480 tx=0x7f11f000b940 comp rx=0 tx=0).stop 2023-08-25T09:50:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11fb7fe640 1 -- 172.21.15.71:0/3622927953 reap_dead start 2023-08-25T09:50:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 shutdown_connections 2023-08-25T09:50:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 >> 172.21.15.71:0/3622927953 conn(0x7f11fc0654f0 msgr2=0x7f11fc0c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.277+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 shutdown_connections 2023-08-25T09:50:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.276+0000 7f4ece7fc640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] conn(0x7f4ebc073c10 0x7f4ebc076050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.277+0000 7f11e17fa640 1 -- 172.21.15.71:0/3622927953 wait complete. 2023-08-25T09:50:06.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.277+0000 7f4ece7fc640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] conn(0x7f4ebc073c10 0x7f4ebc076050 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.277+0000 7f4eceffd640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4ebc06a0a0 0x7f4ebc06c560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8035000 tx=0x7f4ec802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] conn(0x7fd8c8007540 msgr2=0x7fd8c80d2fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6800/3037968157,v1:172.21.15.71:6801/3037968157] conn(0x7fd8c8007540 0x7fd8c80d2fe0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd8b406a150 msgr2=0x7fd8b406c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd8b406a150 0x7fd8b406c610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc001030 tx=0x7fd8cc009040 comp rx=0 tx=0).stop 2023-08-25T09:50:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c80d8f60 msgr2=0x7fd8c80c3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 --2- 172.21.15.71:0/2140637585 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8c80d8f60 0x7fd8c80c3760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000e910 tx=0x7fd8c000edd0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d5ad6640 1 -- 172.21.15.71:0/2140637585 reap_dead start 2023-08-25T09:50:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 shutdown_connections 2023-08-25T09:50:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.278+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 >> 172.21.15.71:0/2140637585 conn(0x7fd8c800e150 msgr2=0x7fd8c80c4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.279+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 shutdown_connections 2023-08-25T09:50:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.279+0000 7fd8d7539640 1 -- 172.21.15.71:0/2140637585 wait complete. 2023-08-25T09:50:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.280+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== osd.7 v2:172.21.15.164:6815/2744841889 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f4ebc076720 con 0x7f4ebc073c10 2023-08-25T09:50:06.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.289+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 --> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c4415e130 con 0x7f1c30073e30 2023-08-25T09:50:06.358 INFO:teuthology.orchestra.run.smithi071.stdout:60129542156 2023-08-25T09:50:06.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.293+0000 7f1c40ff9640 1 -- 172.21.15.71:0/2779091893 <== osd.6 v2:172.21.15.164:6824/4127242778 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1c3c007ba0 con 0x7f1c30073e30 2023-08-25T09:50:06.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-08-25T09:50:06.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.293+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] conn(0x7f1c30073e30 msgr2=0x7f1c30076270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.293+0000 7f1c2a7fc640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.164:6824/4127242778,v1:172.21.15.164:6825/4127242778] conn(0x7f1c30073e30 0x7f1c30076270 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.294+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c3006a150 msgr2=0x7f1c3006c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.294+0000 7f1c2a7fc640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c3006a150 0x7f1c3006c610 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1c34000f60 tx=0x7f1c340079b0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.294+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c4415dd50 msgr2=0x7f1c440aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.294+0000 7f1c2a7fc640 1 --2- 172.21.15.71:0/2779091893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c4415dd50 0x7f1c440aa570 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c380090b0 tx=0x7f1c380046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.294+0000 7f1c48952640 1 -- 172.21.15.71:0/2779091893 reap_dead start 2023-08-25T09:50:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.294+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 shutdown_connections 2023-08-25T09:50:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.295+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 >> 172.21.15.71:0/2779091893 conn(0x7f1c440c6cc0 msgr2=0x7f1c44072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.295+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 shutdown_connections 2023-08-25T09:50:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.295+0000 7f1c2a7fc640 1 -- 172.21.15.71:0/2779091893 wait complete. 2023-08-25T09:50:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.297+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 --> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f723006af00 con 0x7f721c073e30 2023-08-25T09:50:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.299+0000 7f722cff9640 1 -- 172.21.15.71:0/2619725463 <== osd.5 v2:172.21.15.164:6808/1764416968 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f722400c040 con 0x7f721c073e30 2023-08-25T09:50:06.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.299+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 --> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d900c5820 con 0x7f0d70073d30 2023-08-25T09:50:06.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.299+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] conn(0x7f721c073e30 msgr2=0x7f721c076270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.300+0000 7f72357ab640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:6808/1764416968,v1:172.21.15.164:6809/1764416968] conn(0x7f721c073e30 0x7f721c076270 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.300+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a150 msgr2=0x7f721c06c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.300+0000 7f72357ab640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a150 0x7f721c06c610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7220007c40 tx=0x7f7220009040 comp rx=0 tx=0).stop 2023-08-25T09:50:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.300+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72300abc70 msgr2=0x7f723014ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f0d7effd640 1 -- 172.21.15.71:0/1803605650 <== osd.4 v2:172.21.15.164:6800/2125881619 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0d88001730 con 0x7f0d70073d30 2023-08-25T09:50:06.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.300+0000 7f72357ab640 1 --2- 172.21.15.71:0/2619725463 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72300abc70 0x7f723014ed90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f721800bfa0 tx=0x7f7228009750 comp rx=0 tx=0).stop 2023-08-25T09:50:06.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f7234faa640 1 -- 172.21.15.71:0/2619725463 reap_dead start 2023-08-25T09:50:06.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 --> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b800350e0 con 0x7f6b6c073e00 2023-08-25T09:50:06.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 shutdown_connections 2023-08-25T09:50:06.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 >> 172.21.15.71:0/2619725463 conn(0x7f723006c280 msgr2=0x7f723006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.366 INFO:teuthology.orchestra.run.smithi071.stdout:60129542156 2023-08-25T09:50:06.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-08-25T09:50:06.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] conn(0x7f0d70073d30 msgr2=0x7f0d70076170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f0d94c83640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:6800/2125881619,v1:172.21.15.164:6801/2125881619] conn(0x7f0d70073d30 0x7f0d70076170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.301+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 shutdown_connections 2023-08-25T09:50:06.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f72357ab640 1 -- 172.21.15.71:0/2619725463 wait complete. 2023-08-25T09:50:06.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d7006a050 msgr2=0x7f0d7006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d7006a050 0x7f0d7006c510 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d900c4830 tx=0x7f0d80008040 comp rx=0 tx=0).stop 2023-08-25T09:50:06.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 msgr2=0x7f0d9013b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 --2- 172.21.15.71:0/1803605650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d78005180 0x7f0d9013b1f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400b7f0 tx=0x7f0d8400bcb0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f6b857fa640 1 -- 172.21.15.71:0/1831980605 <== osd.3 v2:172.21.15.71:6824/1330919310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b7c031b00 con 0x7f6b6c073e00 2023-08-25T09:50:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d8ed9d640 1 -- 172.21.15.71:0/1803605650 reap_dead start 2023-08-25T09:50:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 shutdown_connections 2023-08-25T09:50:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 >> 172.21.15.71:0/1803605650 conn(0x7f0d900c6cc0 msgr2=0x7f0d900c0380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 shutdown_connections 2023-08-25T09:50:06.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.302+0000 7f0d94c83640 1 -- 172.21.15.71:0/1803605650 wait complete. 2023-08-25T09:50:06.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] conn(0x7f6b6c073e00 msgr2=0x7f6b6c076240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6824/1330919310,v1:172.21.15.71:6825/1330919310] conn(0x7f6b6c073e00 0x7f6b6c076240 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6b6c06a120 msgr2=0x7f6b6c06c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6b6c06a120 0x7f6b6c06c5e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b88000b60 tx=0x7f6b88061360 comp rx=0 tx=0).stop 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b80005ec0 msgr2=0x7f6b800d5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 --2- 172.21.15.71:0/1831980605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b80005ec0 0x7f6b800d5150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400b6c0 tx=0x7f6b7400bac0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b8d173640 1 -- 172.21.15.71:0/1831980605 reap_dead start 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 shutdown_connections 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.303+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 >> 172.21.15.71:0/1831980605 conn(0x7f6b800501b0 msgr2=0x7f6b80005540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.304+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 shutdown_connections 2023-08-25T09:50:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.304+0000 7f6b72ffd640 1 -- 172.21.15.71:0/1831980605 wait complete. 2023-08-25T09:50:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.307+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 --> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ed0159740 con 0x7f4ebc073c10 2023-08-25T09:50:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.309+0000 7f4eb77fe640 1 -- 172.21.15.71:0/1196648597 <== osd.7 v2:172.21.15.164:6815/2744841889 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4ec0007c10 con 0x7f4ebc073c10 2023-08-25T09:50:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.309+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] conn(0x7f4ebc073c10 msgr2=0x7f4ebc076050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.309+0000 7f4ed52e7640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:6815/2744841889,v1:172.21.15.164:6817/2744841889] conn(0x7f4ebc073c10 0x7f4ebc076050 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.309+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4ebc06a0a0 msgr2=0x7f4ebc06c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.309+0000 7f4ed52e7640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4ebc06a0a0 0x7f4ebc06c560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8035000 tx=0x7f4ec802e000 comp rx=0 tx=0).stop 2023-08-25T09:50:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.309+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed0153a50 msgr2=0x7f4ed014e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.310+0000 7f4ed52e7640 1 --2- 172.21.15.71:0/1196648597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed0153a50 0x7f4ed014e530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00c5820 tx=0x7f4ec4011d60 comp rx=0 tx=0).stop 2023-08-25T09:50:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.310+0000 7f4ecf7fe640 1 -- 172.21.15.71:0/1196648597 reap_dead start 2023-08-25T09:50:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.310+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 shutdown_connections 2023-08-25T09:50:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.310+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 >> 172.21.15.71:0/1196648597 conn(0x7f4ed006add0 msgr2=0x7f4ed00c4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.310+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 shutdown_connections 2023-08-25T09:50:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.310+0000 7f4ed52e7640 1 -- 172.21.15.71:0/1196648597 wait complete. 2023-08-25T09:50:06.376 INFO:teuthology.orchestra.run.smithi071.stdout:60129542156 2023-08-25T09:50:06.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-08-25T09:50:06.377 INFO:teuthology.orchestra.run.smithi071.stdout:64424509453 2023-08-25T09:50:06.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-08-25T09:50:06.378 INFO:teuthology.orchestra.run.smithi071.stdout:64424509452 2023-08-25T09:50:06.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-08-25T09:50:06.379 INFO:teuthology.orchestra.run.smithi071.stdout:64424509452 2023-08-25T09:50:06.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-08-25T09:50:06.380 INFO:teuthology.orchestra.run.smithi071.stdout:60129542155 2023-08-25T09:50:06.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-08-25T09:50:06.381 INFO:teuthology.orchestra.run.smithi071.stdout:64424509452 2023-08-25T09:50:06.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-08-25T09:50:06.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.719+0000 7f43d359e640 1 Processor -- start 2023-08-25T09:50:06.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.719+0000 7f7586149640 1 Processor -- start 2023-08-25T09:50:06.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.719+0000 7f43d359e640 1 -- start start 2023-08-25T09:50:06.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.719+0000 7f7586149640 1 -- start start 2023-08-25T09:50:06.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f43d359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f43d40aea70 0x7f43d4158b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f43d359e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d40c4b60 con 0x7f43d40699e0 2023-08-25T09:50:06.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f43d359e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4069e70 con 0x7f43d40ae150 2023-08-25T09:50:06.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f43d359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d406a150 con 0x7f43d40aea70 2023-08-25T09:50:06.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f7586149640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800ae150 0x7f75800bf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f7586149640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75800699e0 0x7f75800c5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f7586149640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75800c4b60 con 0x7f75800aea70 2023-08-25T09:50:06.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f7586149640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7580069e70 con 0x7f75800699e0 2023-08-25T09:50:06.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.721+0000 7f7586149640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758006a150 con 0x7f75800ae150 2023-08-25T09:50:06.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.722+0000 7f43d259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f43d40ae150 0x7f43d40bf140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:57196/0 (socket says 172.21.15.71:57196) 2023-08-25T09:50:06.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.722+0000 7f43d259c640 1 -- 172.21.15.71:0/291393145 learned_addr learned my addr 172.21.15.71:0/291393145 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.722+0000 7f7585948640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f75800aea70 0x7f7580158b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46422/0 (socket says 172.21.15.71:46422) 2023-08-25T09:50:06.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.722+0000 7f7585948640 1 -- 172.21.15.71:0/1648716149 learned_addr learned my addr 172.21.15.71:0/1648716149 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.722+0000 7f7584946640 1 --2- 172.21.15.71:0/1648716149 >> v2:172.21.15.164:3300/0 conn(0x7f75800699e0 0x7f75800c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.726+0000 7f43d2d9d640 1 --2- 172.21.15.71:0/291393145 >> v2:172.21.15.71:3300/0 conn(0x7f43d40aea70 0x7f43d4158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.725+0000 7f7585147640 1 --2- 172.21.15.71:0/1648716149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800ae150 0x7f75800bf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.726+0000 7f43d2d9d640 1 -- 172.21.15.71:0/291393145 >> v1:172.21.15.164:6789/0 conn(0x7f43d40ae150 legacy=0x7f43d40bf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.726+0000 7f43d2d9d640 1 -- 172.21.15.71:0/291393145 >> v1:172.21.15.71:6789/0 conn(0x7f43d40699e0 legacy=0x7f43d40c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.726+0000 7f43d2d9d640 1 -- 172.21.15.71:0/291393145 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40c4db0 con 0x7f43d40aea70 2023-08-25T09:50:06.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.726+0000 7f7585147640 1 -- 172.21.15.71:0/1648716149 >> v2:172.21.15.164:3300/0 conn(0x7f75800699e0 msgr2=0x7f75800c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.726+0000 7f7585147640 1 --2- 172.21.15.71:0/1648716149 >> v2:172.21.15.164:3300/0 conn(0x7f75800699e0 0x7f75800c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.727+0000 7f7585147640 1 -- 172.21.15.71:0/1648716149 >> v1:172.21.15.71:6789/0 conn(0x7f75800aea70 legacy=0x7f7580158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.727+0000 7f7585147640 1 -- 172.21.15.71:0/1648716149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800c4db0 con 0x7f75800ae150 2023-08-25T09:50:06.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.728+0000 7f43d2d9d640 1 --2- 172.21.15.71:0/291393145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40aea70 0x7f43d4158b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43c40097a0 tx=0x7f43c402cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2163e21bc3bcda2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.729+0000 7f7585147640 1 --2- 172.21.15.71:0/1648716149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800ae150 0x7f75800bf140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f757800bfa0 tx=0x7f757802e0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83cbccde62c124e7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.729+0000 7f43d159a640 1 -- 172.21.15.71:0/291393145 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c402da30 con 0x7f43d40aea70 2023-08-25T09:50:06.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.729+0000 7f43d159a640 1 -- 172.21.15.71:0/291393145 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c402db90 con 0x7f43d40aea70 2023-08-25T09:50:06.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.729+0000 7f43d159a640 1 -- 172.21.15.71:0/291393145 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4035440 con 0x7f43d40aea70 2023-08-25T09:50:06.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f7577fff640 1 -- 172.21.15.71:0/1648716149 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757802ee50 con 0x7f75800ae150 2023-08-25T09:50:06.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f7577fff640 1 -- 172.21.15.71:0/1648716149 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7578032070 con 0x7f75800ae150 2023-08-25T09:50:06.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f7577fff640 1 -- 172.21.15.71:0/1648716149 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578038620 con 0x7f75800ae150 2023-08-25T09:50:06.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f43d359e640 1 -- 172.21.15.71:0/291393145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40aea70 msgr2=0x7f43d4158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f43d359e640 1 --2- 172.21.15.71:0/291393145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40aea70 0x7f43d4158b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43c40097a0 tx=0x7f43c402cc80 comp rx=0 tx=0).stop 2023-08-25T09:50:06.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f43d359e640 1 -- 172.21.15.71:0/291393145 shutdown_connections 2023-08-25T09:50:06.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f43d359e640 1 --2- 172.21.15.71:0/291393145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40aea70 0x7f43d4158b60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f43d359e640 1 -- 172.21.15.71:0/291393145 >> 172.21.15.71:0/291393145 conn(0x7f43d406c280 msgr2=0x7f43d406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f43d359e640 1 -- 172.21.15.71:0/291393145 shutdown_connections 2023-08-25T09:50:06.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f7586149640 1 -- 172.21.15.71:0/1648716149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800ae150 msgr2=0x7f75800bf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f7586149640 1 --2- 172.21.15.71:0/1648716149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800ae150 0x7f75800bf140 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f757800bfa0 tx=0x7f757802e0a0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f7586149640 1 -- 172.21.15.71:0/1648716149 shutdown_connections 2023-08-25T09:50:06.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f7586149640 1 --2- 172.21.15.71:0/1648716149 >> v2:172.21.15.164:3300/0 conn(0x7f75800699e0 0x7f75800c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f7586149640 1 --2- 172.21.15.71:0/1648716149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800ae150 0x7f75800bf140 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f7586149640 1 -- 172.21.15.71:0/1648716149 >> 172.21.15.71:0/1648716149 conn(0x7f758006c280 msgr2=0x7f758006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.732+0000 7f7586149640 1 -- 172.21.15.71:0/1648716149 shutdown_connections 2023-08-25T09:50:06.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.732+0000 7f7586149640 1 -- 172.21.15.71:0/1648716149 wait complete. 2023-08-25T09:50:06.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.730+0000 7f5a1d214640 1 Processor -- start 2023-08-25T09:50:06.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f5a1d214640 1 -- start start 2023-08-25T09:50:06.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7fc37bfff640 1 Processor -- start 2023-08-25T09:50:06.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.732+0000 7fc37bfff640 1 -- start start 2023-08-25T09:50:06.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.731+0000 7f43d359e640 1 -- 172.21.15.71:0/291393145 wait complete. 2023-08-25T09:50:06.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.733+0000 7f43d359e640 1 Processor -- start 2023-08-25T09:50:06.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7f43d359e640 1 -- start start 2023-08-25T09:50:06.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7f43d359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 0x7f43d41e00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7f7586149640 1 Processor -- start 2023-08-25T09:50:06.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.733+0000 7f5a1d214640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 0x7f5a18069c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.733+0000 7f5a1d214640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180c55d0 con 0x7f5a1815dd50 2023-08-25T09:50:06.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.733+0000 7f5a1d214640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180bfb10 con 0x7f5a18069850 2023-08-25T09:50:06.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.733+0000 7f5a1d214640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180adf80 con 0x7f5a180c06b0 2023-08-25T09:50:06.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7f5a1759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 0x7f5a18069c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7f5a1759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 0x7f5a18069c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41518/0 (socket says 172.21.15.71:41518) 2023-08-25T09:50:06.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7f5a1759e640 1 -- 172.21.15.71:0/2309827691 learned_addr learned my addr 172.21.15.71:0/2309827691 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7f5a1759e640 1 -- 172.21.15.71:0/2309827691 >> v1:172.21.15.164:6789/0 conn(0x7f5a18069850 legacy=0x7f5a180bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7f5a1759e640 1 -- 172.21.15.71:0/2309827691 >> v1:172.21.15.71:6789/0 conn(0x7f5a1815dd50 legacy=0x7f5a1806a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc3740e2650 0x7fc37404f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37bfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc374007690 con 0x7fc37404fc70 2023-08-25T09:50:06.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37bfff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3740e30c0 con 0x7fc3740509f0 2023-08-25T09:50:06.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37bfff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc374006d60 con 0x7fc3740e2650 2023-08-25T09:50:06.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc3740e2650 0x7fc37404f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc3740e2650 0x7fc37404f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60216/0 (socket says 172.21.15.71:60216) 2023-08-25T09:50:06.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.734+0000 7fc37affd640 1 -- 172.21.15.71:0/346174672 learned_addr learned my addr 172.21.15.71:0/346174672 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7f7586149640 1 -- start start 2023-08-25T09:50:06.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7f5a1759e640 1 -- 172.21.15.71:0/2309827691 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1815e130 con 0x7f5a180c06b0 2023-08-25T09:50:06.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.736+0000 7ff0ee0e1640 1 Processor -- start 2023-08-25T09:50:06.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.736+0000 7f43d359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d40aea70 0x7f43d4155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.736+0000 7f43d359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40c8940 0x7f43d4156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.736+0000 7ff0ee0e1640 1 -- start start 2023-08-25T09:50:06.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.736+0000 7f43d259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 0x7f43d41e00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f43d259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 0x7f43d41e00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56190/0 (socket says 172.21.15.71:56190) 2023-08-25T09:50:06.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f7586149640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 0x7f75800ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f43d259c640 1 -- 172.21.15.71:0/1507401231 learned_addr learned my addr 172.21.15.71:0/1507401231 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d406e6f0 con 0x7f43d40c8940 2023-08-25T09:50:06.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4070c80 con 0x7f43d40ae150 2023-08-25T09:50:06.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40711b0 con 0x7f43d40aea70 2023-08-25T09:50:06.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7fc37affd640 1 -- 172.21.15.71:0/346174672 >> v1:172.21.15.164:6789/0 conn(0x7fc3740509f0 legacy=0x7fc3740d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7fc37affd640 1 -- 172.21.15.71:0/346174672 >> v1:172.21.15.71:6789/0 conn(0x7fc37404fc70 legacy=0x7fc374049480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.735+0000 7fc37affd640 1 -- 172.21.15.71:0/346174672 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374007040 con 0x7fc3740e2650 2023-08-25T09:50:06.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f7586149640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75800aea70 0x7f75801e2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f7586149640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800af400 0x7f75801e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f7586149640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75800aaac0 con 0x7f75800aea70 2023-08-25T09:50:06.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f7586149640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75801e05a0 con 0x7f75800ae150 2023-08-25T09:50:06.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f7586149640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75801e08d0 con 0x7f75800af400 2023-08-25T09:50:06.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f43d1d9b640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d40aea70 0x7f43d4155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.737+0000 7f5a1759e640 1 --2- 172.21.15.71:0/2309827691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 0x7f5a18069c90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a100098a0 tx=0x7f5a1002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bb873a0267d4e74 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f7585147640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 0x7f75800ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f7585948640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800af400 0x7f75801e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f7585147640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 0x7f75800ab660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56194/0 (socket says 172.21.15.71:56194) 2023-08-25T09:50:06.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f7585147640 1 -- 172.21.15.71:0/3568346600 learned_addr learned my addr 172.21.15.71:0/3568346600 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f7584946640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75800aea70 0x7f75801e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d2d9d640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40c8940 0x7f43d4156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7ff0ee0e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7ff0ee0e1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff0e806abc0 0x7ff0e80bf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7ff0ee0e1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e8155da0 con 0x7ff0e80c0840 2023-08-25T09:50:06.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7ff0ee0e1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e8155980 con 0x7ff0e806abc0 2023-08-25T09:50:06.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7ff0ee0e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80adcb0 con 0x7ff0e80acd20 2023-08-25T09:50:06.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7fc37affd640 1 --2- 172.21.15.71:0/346174672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740e2650 0x7fc37404f6c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009870 tx=0x7fc36c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a28ca066f2d1c75 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f5a1659c640 1 -- 172.21.15.71:0/2309827691 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1002f830 con 0x7f5a180c06b0 2023-08-25T09:50:06.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f5a1659c640 1 -- 172.21.15.71:0/2309827691 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1002f990 con 0x7f5a180c06b0 2023-08-25T09:50:06.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f5a1659c640 1 -- 172.21.15.71:0/2309827691 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1002fd20 con 0x7f5a180c06b0 2023-08-25T09:50:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d2d9d640 1 -- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d40aea70 msgr2=0x7f43d4155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d2d9d640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d40aea70 0x7f43d4155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d2d9d640 1 -- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 msgr2=0x7f43d41e00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d2d9d640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 0x7f43d41e00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d2d9d640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009400 con 0x7f43d40c8940 2023-08-25T09:50:06.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.738+0000 7f43d259c640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 0x7f43d41e00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:06.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7584946640 1 -- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800af400 msgr2=0x7f75801e0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7584946640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800af400 0x7f75801e0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7584946640 1 -- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 msgr2=0x7f75800ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7584946640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 0x7f75800ab660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7584946640 1 -- 172.21.15.71:0/3568346600 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757800bc80 con 0x7f75800aea70 2023-08-25T09:50:06.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7585147640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 0x7f75800ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:50:06.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41548/0 (socket says 172.21.15.71:41548) 2023-08-25T09:50:06.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7ff0e77fe640 1 -- 172.21.15.71:0/2348593658 learned_addr learned my addr 172.21.15.71:0/2348593658 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f43d2d9d640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40c8940 0x7f43d4156210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f43c4034000 tx=0x7f43c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.739+0000 7f7584946640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75800aea70 0x7f75801e2280 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f757c00e880 tx=0x7f757c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f5a1d214640 1 -- 172.21.15.71:0/2309827691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 msgr2=0x7f5a18069c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2309827691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 0x7f5a18069c90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5a100098a0 tx=0x7f5a1002ba10 comp rx=0 tx=0).stop 2023-08-25T09:50:06.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f5a1d214640 1 -- 172.21.15.71:0/2309827691 shutdown_connections 2023-08-25T09:50:06.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2309827691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a180c06b0 0x7f5a18069c90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f5a1d214640 1 -- 172.21.15.71:0/2309827691 >> 172.21.15.71:0/2309827691 conn(0x7f5a180c6cc0 msgr2=0x7f5a180c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f5a1d214640 1 -- 172.21.15.71:0/2309827691 shutdown_connections 2023-08-25T09:50:06.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7f5a1d214640 1 -- 172.21.15.71:0/2309827691 wait complete. 2023-08-25T09:50:06.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7fc379ffb640 1 -- 172.21.15.71:0/346174672 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c030860 con 0x7fc3740e2650 2023-08-25T09:50:06.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7fc379ffb640 1 -- 172.21.15.71:0/346174672 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c0309c0 con 0x7fc3740e2650 2023-08-25T09:50:06.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7fc379ffb640 1 -- 172.21.15.71:0/346174672 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c030d50 con 0x7fc3740e2650 2023-08-25T09:50:06.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7fc37bfff640 1 -- 172.21.15.71:0/346174672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740e2650 msgr2=0x7fc37404f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7fc37bfff640 1 --2- 172.21.15.71:0/346174672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740e2650 0x7fc37404f6c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009870 tx=0x7fc36c02b9a0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7ff0e7fff640 1 --2- 172.21.15.71:0/2348593658 >> v2:172.21.15.164:3300/0 conn(0x7ff0e806abc0 0x7ff0e80bf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c00c940 con 0x7f75800aea70 2023-08-25T09:50:06.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c00caa0 con 0x7f75800aea70 2023-08-25T09:50:06.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c00ce50 con 0x7f75800aea70 2023-08-25T09:50:06.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f7586149640 1 -- 172.21.15.71:0/3568346600 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75800c9560 con 0x7f75800aea70 2023-08-25T09:50:06.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.740+0000 7f7586149640 1 -- 172.21.15.71:0/3568346600 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75800c9b20 con 0x7f75800aea70 2023-08-25T09:50:06.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7ff0e77fe640 1 -- 172.21.15.71:0/2348593658 >> v2:172.21.15.164:3300/0 conn(0x7ff0e806abc0 msgr2=0x7ff0e80bf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7ff0e77fe640 1 --2- 172.21.15.71:0/2348593658 >> v2:172.21.15.164:3300/0 conn(0x7ff0e806abc0 0x7ff0e80bf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7ff0e77fe640 1 -- 172.21.15.71:0/2348593658 >> v1:172.21.15.164:6789/0 conn(0x7ff0e80c0840 legacy=0x7ff0e8069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.741+0000 7ff0e77fe640 1 -- 172.21.15.71:0/2348593658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8152860 con 0x7ff0e80acd20 2023-08-25T09:50:06.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4040670 con 0x7f43d40c8940 2023-08-25T09:50:06.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c40407f0 con 0x7f43d40c8940 2023-08-25T09:50:06.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4040ba0 con 0x7f43d40c8940 2023-08-25T09:50:06.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d40c0540 con 0x7f43d40c8940 2023-08-25T09:50:06.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7fc37bfff640 1 -- 172.21.15.71:0/346174672 shutdown_connections 2023-08-25T09:50:06.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7fc37bfff640 1 --2- 172.21.15.71:0/346174672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740e2650 0x7fc37404f6c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7fc37bfff640 1 -- 172.21.15.71:0/346174672 >> 172.21.15.71:0/346174672 conn(0x7fc3740545e0 msgr2=0x7fc3740549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.742+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d406e380 con 0x7f43d40c8940 2023-08-25T09:50:06.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.743+0000 7ff0e77fe640 1 --2- 172.21.15.71:0/2348593658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c0300 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8008870 tx=0x7ff0d802df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934e470ba2fdce77 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.743+0000 7f5a1d214640 1 Processor -- start 2023-08-25T09:50:06.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.744+0000 7f5a1d214640 1 -- start start 2023-08-25T09:50:06.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.745+0000 7f5a1d214640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a18069850 0x7f5a18157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.745+0000 7f5a1d214640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a180c06b0 0x7f5a181f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.743+0000 7fc37bfff640 1 -- 172.21.15.71:0/346174672 shutdown_connections 2023-08-25T09:50:06.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1d214640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 0x7f5a181f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.743+0000 7fc37bfff640 1 -- 172.21.15.71:0/346174672 wait complete. 2023-08-25T09:50:06.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.744+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d406e8a0 con 0x7f43d40c8940 2023-08-25T09:50:06.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a16d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a18069850 0x7f5a18157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a16d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a18069850 0x7f5a18157880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41550/0 (socket says 172.21.15.71:41550) 2023-08-25T09:50:06.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a180c06b0 0x7f5a181f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a180c06b0 0x7f5a181f0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56204/0 (socket says 172.21.15.71:56204) 2023-08-25T09:50:06.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7fc37bfff640 1 Processor -- start 2023-08-25T09:50:06.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.744+0000 7ff0e6ffd640 1 -- 172.21.15.71:0/2348593658 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d802ed40 con 0x7ff0e80acd20 2023-08-25T09:50:06.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.744+0000 7ff0e6ffd640 1 -- 172.21.15.71:0/2348593658 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8032070 con 0x7ff0e80acd20 2023-08-25T09:50:06.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.744+0000 7ff0e6ffd640 1 -- 172.21.15.71:0/2348593658 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d802eea0 con 0x7ff0e80acd20 2023-08-25T09:50:06.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.745+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/2348593658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 msgr2=0x7ff0e80c0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.745+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/2348593658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c0300 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8008870 tx=0x7ff0d802df90 comp rx=0 tx=0).stop 2023-08-25T09:50:06.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1759e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 0x7f5a181f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1d214640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180c4420 con 0x7f5a1815dd50 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1759e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 0x7f5a181f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60244/0 (socket says 172.21.15.71:60244) 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1759e640 1 -- 172.21.15.71:0/2350614440 learned_addr learned my addr 172.21.15.71:0/2350614440 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a180c3d70 con 0x7f5a180c06b0 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18159280 con 0x7f5a18069850 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f5a1759e640 1 -- 172.21.15.71:0/2350614440 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a18069850 msgr2=0x7f5a18157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f5a1759e640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a18069850 0x7f5a18157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f5a1759e640 1 -- 172.21.15.71:0/2350614440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a180c06b0 msgr2=0x7f5a181f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f5a1759e640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a180c06b0 0x7f5a181f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f5a1759e640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a100094d0 con 0x7f5a1815dd50 2023-08-25T09:50:06.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.746+0000 7fc37bfff640 1 -- start start 2023-08-25T09:50:06.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/2348593658 shutdown_connections 2023-08-25T09:50:06.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/2348593658 >> v2:172.21.15.164:3300/0 conn(0x7ff0e806abc0 0x7ff0e80bf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/2348593658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c0300 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/2348593658 >> 172.21.15.71:0/2348593658 conn(0x7ff0e80c6cc0 msgr2=0x7ff0e80c70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7f5a1759e640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 0x7f5a181f1480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5a100097f0 tx=0x7f5a100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7fc37bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37404fc70 0x7fc3740523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7fc37bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740509f0 0x7fc374052900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7f87fdb12640 1 Processor -- start 2023-08-25T09:50:06.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.745+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f757c013470 con 0x7f75800aea70 2023-08-25T09:50:06.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f7575ffb640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f756406a120 0x7f756406c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.747+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f757c088a40 con 0x7f75800aea70 2023-08-25T09:50:06.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7f7585147640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f756406a120 0x7f756406c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.748+0000 7f7586149640 1 -- 172.21.15.71:0/3568346600 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560005250 con 0x7f75800aea70 2023-08-25T09:50:06.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/2348593658 shutdown_connections 2023-08-25T09:50:06.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/2348593658 wait complete. 2023-08-25T09:50:06.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7f7585147640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f756406a120 0x7f756406c5e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f757800bfa0 tx=0x7f7578034000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37404fc70 0x7fc3740523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37404fc70 0x7fc3740523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41564/0 (socket says 172.21.15.71:41564) 2023-08-25T09:50:06.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37affd640 1 -- 172.21.15.71:0/2134266830 learned_addr learned my addr 172.21.15.71:0/2134266830 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37a7fc640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740509f0 0x7fc374052900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3740e2650 0x7fc37416a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3741693d0 con 0x7fc3740509f0 2023-08-25T09:50:06.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3740cb580 con 0x7fc3740e2650 2023-08-25T09:50:06.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc374054110 con 0x7fc37404fc70 2023-08-25T09:50:06.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.750+0000 7fc37b7fe640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3740e2650 0x7fc37416a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.750+0000 7fc37a7fc640 1 -- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37404fc70 msgr2=0x7fc3740523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.750+0000 7fc37a7fc640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37404fc70 0x7fc3740523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.750+0000 7fc37a7fc640 1 -- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3740e2650 msgr2=0x7fc37416a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.750+0000 7fc37a7fc640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3740e2650 0x7fc37416a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.750+0000 7fc37a7fc640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0094d0 con 0x7fc3740509f0 2023-08-25T09:50:06.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7f87fdb12640 1 -- start start 2023-08-25T09:50:06.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fb4ae54f640 1 Processor -- start 2023-08-25T09:50:06.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.749+0000 7fb4ae54f640 1 -- start start 2023-08-25T09:50:06.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f43c403c070 con 0x7f43d40c8940 2023-08-25T09:50:06.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7fc37a7fc640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740509f0 0x7fc374052900 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc37000c410 tx=0x7fc37000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7fb4ae54f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb4a80c4c90 0x7fb4a8155a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7ff0ee0e1640 1 Processor -- start 2023-08-25T09:50:06.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7fb4ae54f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a8069c60 con 0x7fb4a814f220 2023-08-25T09:50:06.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7fb4ae54f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a80c58c0 con 0x7fb4a8069850 2023-08-25T09:50:06.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.751+0000 7fb4ae54f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a80c5ba0 con 0x7fb4a80c4c90 2023-08-25T09:50:06.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7fb4a7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb4a814f220 0x7fb4a80c6260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46434/0 (socket says 172.21.15.71:46434) 2023-08-25T09:50:06.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7fb4a7fff640 1 -- 172.21.15.71:0/1344751841 learned_addr learned my addr 172.21.15.71:0/1344751841 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7f43c2ffd640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f43b406a120 0x7f43b406c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f43c40abc20 con 0x7f43d40c8940 2023-08-25T09:50:06.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7fb49f7fe640 1 --2- 172.21.15.71:0/1344751841 >> v2:172.21.15.71:3300/0 conn(0x7fb4a80c4c90 0x7fb4a8155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7fad4e9d6640 1 Processor -- start 2023-08-25T09:50:06.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7ff0ee0e1640 1 -- start start 2023-08-25T09:50:06.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7fad4e9d6640 1 -- start start 2023-08-25T09:50:06.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.752+0000 7f87fdb12640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f815d090 0x7f87f80c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400470945 0 0) 0x7fb4a80c58c0 con 0x7fb4a8069850 2023-08-25T09:50:06.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb494003520 con 0x7fb4a8069850 2023-08-25T09:50:06.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f87f77fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f87f80c9730 0x7f87f815c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46448/0 (socket says 172.21.15.71:46448) 2023-08-25T09:50:06.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f87f77fe640 1 -- 172.21.15.71:0/3937149250 learned_addr learned my addr 172.21.15.71:0/3937149250 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f43d259c640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f43b406a120 0x7f43b406c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f8159d30 con 0x7f87f80c9730 2023-08-25T09:50:06.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7ff0ee0e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 0x7ff0e81f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7ff0ee0e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7ff0ee0e1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0e80c0840 0x7ff0e8150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369108711 0 0) 0x7fb494003520 con 0x7fb4a8069850 2023-08-25T09:50:06.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 >> v2:172.21.15.71:3300/0 conn(0x7fb4a80c4c90 msgr2=0x7fb4a8155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 --2- 172.21.15.71:0/1344751841 >> v2:172.21.15.71:3300/0 conn(0x7fb4a80c4c90 0x7fb4a8155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 >> v1:172.21.15.71:6789/0 conn(0x7fb4a814f220 legacy=0x7fb4a80c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8154f40 con 0x7fb4a8069850 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fb4a00019f0 con 0x7fb4a8069850 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 >> v1:172.21.15.164:6789/0 conn(0x7fb4a8069850 legacy=0x7fb4a815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4940039e0 0x7fb494003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7fb4a77fe640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940047c0 0x7fb494004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7fb4a77fe640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb494005180 0x7fb49400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a80c5ba0 con 0x7fb4940039e0 2023-08-25T09:50:06.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49400c090 con 0x7fb494005180 2023-08-25T09:50:06.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49400c250 con 0x7fb4940047c0 2023-08-25T09:50:06.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 0x7ff0e81f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0deffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0deffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41580/0 (socket says 172.21.15.71:41580) 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0e77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 0x7ff0e81f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56230/0 (socket says 172.21.15.71:56230) 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0deffd640 1 -- 172.21.15.71:0/219576894 learned_addr learned my addr 172.21.15.71:0/219576894 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80c1920 con 0x7ff0e80c0840 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81f29b0 con 0x7ff0e806abc0 2023-08-25T09:50:06.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e80c1b10 con 0x7ff0e80acd20 2023-08-25T09:50:06.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f81591d0 con 0x7f87f8158d40 2023-08-25T09:50:06.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f8159490 con 0x7f87f815d090 2023-08-25T09:50:06.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.753+0000 7f87f7fff640 1 --2- 172.21.15.71:0/3937149250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f815d090 0x7f87f80c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7f43d259c640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f43b406a120 0x7f43b406c5e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0012e0 tx=0x7f43cc00a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7fad4e9d6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad4004fc70 0x7fad4004f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7f87f6ffd640 1 -- 172.21.15.71:0/3937149250 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536447199 0 0) 0x7f87f8159d30 con 0x7f87f80c9730 2023-08-25T09:50:06.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7f87f6ffd640 1 -- 172.21.15.71:0/3937149250 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e4003520 con 0x7f87f80c9730 2023-08-25T09:50:06.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3700035f0 con 0x7fc3740509f0 2023-08-25T09:50:06.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fad4e9d6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad40007690 con 0x7fad400509f0 2023-08-25T09:50:06.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fad4e9d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400e30c0 con 0x7fad400e2650 2023-08-25T09:50:06.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc374169160 con 0x7fc3740509f0 2023-08-25T09:50:06.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3740cb250 con 0x7fc3740509f0 2023-08-25T09:50:06.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fad4e9d6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad40006d60 con 0x7fad4004fc70 2023-08-25T09:50:06.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f43c4076f70 con 0x7f43d40c8940 2023-08-25T09:50:06.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1003b040 con 0x7f5a1815dd50 2023-08-25T09:50:06.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a180c40f0 con 0x7f5a1815dd50 2023-08-25T09:50:06.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a181594d0 con 0x7f5a1815dd50 2023-08-25T09:50:06.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fad4cf73640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fad400509f0 0x7fad400d8c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46452/0 (socket says 172.21.15.71:46452) 2023-08-25T09:50:06.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fad4cf73640 1 -- 172.21.15.71:0/2207253567 learned_addr learned my addr 172.21.15.71:0/2207253567 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:06.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7ff0e7fff640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0e80c0840 0x7ff0e8150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7ff0e77fe640 1 -- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 msgr2=0x7ff0e80c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7ff0e77fe640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7ff0e77fe640 1 -- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0e80c0840 msgr2=0x7ff0e8150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7ff0e77fe640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0e80c0840 0x7ff0e8150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7ff0e77fe640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d8008550 con 0x7ff0e806abc0 2023-08-25T09:50:06.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7f87f7fff640 1 -- 172.21.15.71:0/3937149250 >> v1:172.21.15.164:6789/0 conn(0x7f87f8158d40 legacy=0x7f87f81616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7f87f7fff640 1 -- 172.21.15.71:0/3937149250 >> v1:172.21.15.71:6789/0 conn(0x7f87f80c9730 legacy=0x7f87f815c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7f87f7fff640 1 -- 172.21.15.71:0/3937149250 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f8159770 con 0x7f87f815d090 2023-08-25T09:50:06.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.754+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4940039e0 0x7fb494003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fb4acaec640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb494005180 0x7fb49400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.755+0000 7fb49f7fe640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940047c0 0x7fb494004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7ff0e77fe640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 0x7ff0e81f2e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0e806b050 tx=0x7ff0d8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc374005330 con 0x7fc3740509f0 2023-08-25T09:50:06.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7f87f7fff640 1 --2- 172.21.15.71:0/3937149250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f815d090 0x7f87f80c8ec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f87f00097d0 tx=0x7f87f002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2523f309a1c2c6a4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fb4a7fff640 1 -- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940047c0 msgr2=0x7fb494004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940047c0 0x7fb494004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fb4a7fff640 1 -- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb494005180 msgr2=0x7fb49400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb494005180 0x7fb49400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.756+0000 7fb4a7fff640 1 -- 172.21.15.71:0/1344751841 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8154f40 con 0x7fb4940039e0 2023-08-25T09:50:06.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7fb4acaec640 1 -- 172.21.15.71:0/1344751841 reap_dead start 2023-08-25T09:50:06.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028115280 0 0) 0x7fad400e30c0 con 0x7fad400e2650 2023-08-25T09:50:06.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7fad477fe640 1 --2- 172.21.15.71:0/2207253567 >> v2:172.21.15.164:3300/0 conn(0x7fad4004fc70 0x7fad4004f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad38003520 con 0x7fad400e2650 2023-08-25T09:50:06.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800b360 con 0x7ff0e806abc0 2023-08-25T09:50:06.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a180c45d0 con 0x7f5a1815dd50 2023-08-25T09:50:06.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d800b500 con 0x7ff0e806abc0 2023-08-25T09:50:06.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800b8b0 con 0x7ff0e806abc0 2023-08-25T09:50:06.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e80ca080 con 0x7ff0e806abc0 2023-08-25T09:50:06.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7f87f6ffd640 1 -- 172.21.15.71:0/3937149250 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0030800 con 0x7f87f815d090 2023-08-25T09:50:06.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7f87f6ffd640 1 -- 172.21.15.71:0/3937149250 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0030960 con 0x7f87f815d090 2023-08-25T09:50:06.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.757+0000 7f87f6ffd640 1 -- 172.21.15.71:0/3937149250 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0030cf0 con 0x7f87f815d090 2023-08-25T09:50:06.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4940039e0 0x7fb494003e00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb498001650 tx=0x7fb4940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbca1cdcd534d069 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e80ca640 con 0x7ff0e806abc0 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566071907 0 0) 0x7fad38003520 con 0x7fad400e2650 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 >> v2:172.21.15.164:3300/0 conn(0x7fad4004fc70 msgr2=0x7fad4004f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 --2- 172.21.15.71:0/2207253567 >> v2:172.21.15.164:3300/0 conn(0x7fad4004fc70 0x7fad4004f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 >> v1:172.21.15.71:6789/0 conn(0x7fad400509f0 legacy=0x7fad400d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40007040 con 0x7fad400e2650 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fad3c0015a0 con 0x7fad400e2650 2023-08-25T09:50:06.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 >> v1:172.21.15.164:6789/0 conn(0x7fad400e2650 legacy=0x7fad40049480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f815d090 msgr2=0x7f87f80c8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.758+0000 7f87fdb12640 1 --2- 172.21.15.71:0/3937149250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f815d090 0x7f87f80c8ec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f87f00097d0 tx=0x7f87f002b940 comp rx=0 tx=0).stop 2023-08-25T09:50:06.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad46ffd640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad380039c0 0x7fad38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad46ffd640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad380047c0 0x7fad38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad46ffd640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad38005180 0x7fad3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 shutdown_connections 2023-08-25T09:50:06.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7f87fdb12640 1 --2- 172.21.15.71:0/3937149250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f815d090 0x7f87f80c8ec0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 >> 172.21.15.71:0/3937149250 conn(0x7f87f80654f0 msgr2=0x7f87f8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad40006d60 con 0x7fad380047c0 2023-08-25T09:50:06.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3800c060 con 0x7fad38005180 2023-08-25T09:50:06.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3800c220 con 0x7fad380039c0 2023-08-25T09:50:06.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad477fe640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad380047c0 0x7fad38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad47fff640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad380039c0 0x7fad38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 shutdown_connections 2023-08-25T09:50:06.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7f87fdb12640 1 -- 172.21.15.71:0/3937149250 wait complete. 2023-08-25T09:50:06.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.759+0000 7fad4cf73640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad38005180 0x7fad3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.760+0000 7fad477fe640 1 -- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad380039c0 msgr2=0x7fad38003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.760+0000 7fad477fe640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad380039c0 0x7fad38003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.760+0000 7fad477fe640 1 -- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad38005180 msgr2=0x7fad3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:06.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.760+0000 7fad477fe640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad38005180 0x7fad3800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.760+0000 7fad477fe640 1 -- 172.21.15.71:0/2207253567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38003520 con 0x7fad380047c0 2023-08-25T09:50:06.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.760+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e80c1c50 con 0x7ff0e806abc0 2023-08-25T09:50:06.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.761+0000 7fad4cf73640 1 -- 172.21.15.71:0/2207253567 reap_dead start 2023-08-25T09:50:06.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.761+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff0d8038500 con 0x7ff0e806abc0 2023-08-25T09:50:06.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.762+0000 7ff0e4ff9640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff0c806a0d0 0x7ff0c806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.762+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7ff0d80a98a0 con 0x7ff0e806abc0 2023-08-25T09:50:06.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.762+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc370003750 con 0x7fc3740509f0 2023-08-25T09:50:06.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.762+0000 7ff0deffd640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff0c806a0d0 0x7ff0c806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.763+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f757c053e40 con 0x7f75800aea70 2023-08-25T09:50:06.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.763+0000 7ff0deffd640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff0c806a0d0 0x7ff0c806c590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80ca8a0 tx=0x7ff0d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.762+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc370003ac0 con 0x7fc3740509f0 2023-08-25T09:50:06.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.764+0000 7fad477fe640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad380047c0 0x7fad38004c40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fad400122d0 tx=0x7fad380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0864299bb0dcb6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.763+0000 7f87fdb12640 1 Processor -- start 2023-08-25T09:50:06.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.763+0000 7f87fdb12640 1 -- start start 2023-08-25T09:50:06.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.764+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a10037cd0 con 0x7f5a1815dd50 2023-08-25T09:50:06.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.764+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10042680 con 0x7f5a1815dd50 2023-08-25T09:50:06.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87fdb12640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 0x7f87f8150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87fdb12640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f80b3270 0x7f87f81ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87fdb12640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f81f2760 0x7f87f813c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87fdb12640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f81eeb60 con 0x7f87f80b3270 2023-08-25T09:50:06.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87fdb12640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f81515a0 con 0x7f87f81f2760 2023-08-25T09:50:06.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87fdb12640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f8152470 con 0x7f87f80c9730 2023-08-25T09:50:06.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f81f2760 0x7f87f813c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87f77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 0x7f87f8150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87f77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 0x7f87f8150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41604/0 (socket says 172.21.15.71:41604) 2023-08-25T09:50:06.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7f87f77fe640 1 -- 172.21.15.71:0/616212211 learned_addr learned my addr 172.21.15.71:0/616212211 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7f87eeffd640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f80b3270 0x7f87f81ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc370003c20 con 0x7fc3740509f0 2023-08-25T09:50:06.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498004aa0 con 0x7fb4940039e0 2023-08-25T09:50:06.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49802e070 con 0x7fb4940039e0 2023-08-25T09:50:06.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.765+0000 7fb4a77fe640 1 -- 172.21.15.71:0/1344751841 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498004e60 con 0x7fb4940039e0 2023-08-25T09:50:06.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 -- 172.21.15.71:0/1344751841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4940039e0 msgr2=0x7fb494003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4940039e0 0x7fb494003e00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb498001650 tx=0x7fb4940057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 -- 172.21.15.71:0/1344751841 shutdown_connections 2023-08-25T09:50:06.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 --2- 172.21.15.71:0/1344751841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4940039e0 0x7fb494003e00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 -- 172.21.15.71:0/1344751841 >> 172.21.15.71:0/1344751841 conn(0x7fb4a80654f0 msgr2=0x7fb4a8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5a100428a0 con 0x7f5a1815dd50 2023-08-25T09:50:06.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7f87f77fe640 1 -- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f81f2760 msgr2=0x7f87f813c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7f87f77fe640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f81f2760 0x7f87f813c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7f87f77fe640 1 -- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f80b3270 msgr2=0x7f87f81ee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7f87f77fe640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f80b3270 0x7f87f81ee620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7f87f77fe640 1 -- 172.21.15.71:0/616212211 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0009400 con 0x7f87f80c9730 2023-08-25T09:50:06.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 -- 172.21.15.71:0/1344751841 shutdown_connections 2023-08-25T09:50:06.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.766+0000 7fb4ae54f640 1 -- 172.21.15.71:0/1344751841 wait complete. 2023-08-25T09:50:06.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7f87f77fe640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 0x7f87f8150950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e800cc20 tx=0x7f87e800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7fc3637fe640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc36806a050 0x7fc36806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.767+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fc370088530 con 0x7fc3740509f0 2023-08-25T09:50:06.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f5a0ffff640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f59f806a080 0x7f59f806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7fc37affd640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc36806a050 0x7fc36806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e800ace0 con 0x7f87f80c9730 2023-08-25T09:50:06.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e800ae40 con 0x7f87f80c9730 2023-08-25T09:50:06.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8012600 con 0x7f87f80c9730 2023-08-25T09:50:06.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f81529c0 con 0x7f87f80c9730 2023-08-25T09:50:06.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f81526e0 con 0x7f87f80c9730 2023-08-25T09:50:06.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.768+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f5a100ac2e0 con 0x7f5a1815dd50 2023-08-25T09:50:06.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.769+0000 7fb4ae54f640 1 Processor -- start 2023-08-25T09:50:06.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.769+0000 7f5a16d9d640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f59f806a080 0x7f59f806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.769+0000 7fc37affd640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc36806a050 0x7fc36806c510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc36c02b9e0 tx=0x7fc36c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.770+0000 7fb4ae54f640 1 -- start start 2023-08-25T09:50:06.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.770+0000 7f5a16d9d640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f59f806a080 0x7f59f806c540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800a400 tx=0x7f5a0800d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.770+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f804e420 con 0x7f87f80c9730 2023-08-25T09:50:06.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.771+0000 7fb4ae54f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 0x7fb4a814d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.771+0000 7fb4ae54f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4a80c0cd0 0x7fb4a8158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb4ae54f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 0x7fb4a81f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 0x7fb4a814d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 0x7fb4a814d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:41614/0 (socket says 172.21.15.71:41614) 2023-08-25T09:50:06.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f87e8007070 con 0x7f87f80c9730 2023-08-25T09:50:06.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5a100777f0 con 0x7f5a1815dd50 2023-08-25T09:50:06.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb4acaec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 0x7fb4a81f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb49f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4a80c0cd0 0x7fb4a8158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb4acaec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 0x7fb4a81f6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60296/0 (socket says 172.21.15.71:60296) 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.772+0000 7fb4a7fff640 1 -- 172.21.15.71:0/3446591096 learned_addr learned my addr 172.21.15.71:0/3446591096 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.773+0000 7fb4ae54f640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81f6120 con 0x7fb4a8158ed0 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.773+0000 7fb4ae54f640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81586d0 con 0x7fb4a80c0cd0 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.773+0000 7fb4ae54f640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81561b0 con 0x7fb4940039e0 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7fb4acaec640 1 -- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 msgr2=0x7fb4a814d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7fb4acaec640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 0x7fb4a814d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7fb4acaec640 1 -- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4a80c0cd0 msgr2=0x7fb4a8158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7fb4acaec640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4a80c0cd0 0x7fb4a8158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7fb4acaec640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a80c2e60 con 0x7fb4a8158ed0 2023-08-25T09:50:06.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 0x7fb4a814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:50:06.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7f87f4ff9640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d806a0a0 0x7f87d806c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.774+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f87e8088680 con 0x7f87f80c9730 2023-08-25T09:50:06.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.775+0000 7fb4acaec640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 0x7fb4a81f6540 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8071d70 tx=0x7fb4a000df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.775+0000 7f87eeffd640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d806a0a0 0x7f87d806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.775+0000 7f87eeffd640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d806a0a0 0x7f87d806c560 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f87f81ef750 tx=0x7f87e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34005990 con 0x7fad380047c0 2023-08-25T09:50:06.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0003d80 con 0x7fb4a8158ed0 2023-08-25T09:50:06.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc3700538b0 con 0x7fc3740509f0 2023-08-25T09:50:06.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fb4ae54f640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a80c3120 con 0x7fb4a8158ed0 2023-08-25T09:50:06.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad34005af0 con 0x7fad380047c0 2023-08-25T09:50:06.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fad46ffd640 1 -- 172.21.15.71:0/2207253567 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34005ea0 con 0x7fad380047c0 2023-08-25T09:50:06.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.779+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff0d8074ec0 con 0x7ff0e806abc0 2023-08-25T09:50:06.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fb4ae54f640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a80c36e0 con 0x7fb4a8158ed0 2023-08-25T09:50:06.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.778+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a0005070 con 0x7fb4a8158ed0 2023-08-25T09:50:06.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.779+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2207253567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad380047c0 msgr2=0x7fad38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.779+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad380047c0 0x7fad38004c40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fad400122d0 tx=0x7fad380057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.780+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2207253567 shutdown_connections 2023-08-25T09:50:06.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.780+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2207253567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad380047c0 0x7fad38004c40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.780+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2207253567 >> 172.21.15.71:0/2207253567 conn(0x7fad400545e0 msgr2=0x7fad400549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:06.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.780+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00053e0 con 0x7fb4a8158ed0 2023-08-25T09:50:06.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.780+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2207253567 shutdown_connections 2023-08-25T09:50:06.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.780+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2207253567 wait complete. 2023-08-25T09:50:06.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.781+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a8155200 con 0x7fb4a8158ed0 2023-08-25T09:50:06.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.782+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb4a0005600 con 0x7fb4a8158ed0 2023-08-25T09:50:06.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.782+0000 7fad4e9d6640 1 Processor -- start 2023-08-25T09:50:06.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.783+0000 7fad4e9d6640 1 -- start start 2023-08-25T09:50:06.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.784+0000 7fb4a57fa640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb48806a050 0x7fb48806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.784+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fb4a00883c0 con 0x7fb4a8158ed0 2023-08-25T09:50:06.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.784+0000 7fad4e9d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 0x7fad4004bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.784+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb48806a050 0x7fb48806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.784+0000 7fad4e9d6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad4004e810 0x7fad40052450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad4e9d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad40052990 0x7fad40177c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad47fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 0x7fad4004bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad47fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 0x7fad4004bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:56272/0 (socket says 172.21.15.71:56272) 2023-08-25T09:50:06.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad477fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad4004e810 0x7fad40052450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad477fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad4004e810 0x7fad40052450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60302/0 (socket says 172.21.15.71:60302) 2023-08-25T09:50:06.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad40052e10 con 0x7fad4004e810 2023-08-25T09:50:06.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad47fff640 1 -- 172.21.15.71:0/2465226879 learned_addr learned my addr 172.21.15.71:0/2465226879 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:06.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad400d2c10 con 0x7fad380047c0 2023-08-25T09:50:06.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad401747c0 con 0x7fad40052990 2023-08-25T09:50:06.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.785+0000 7fb4a7fff640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb48806a050 0x7fb48806c510 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb498000f60 tx=0x7fb498009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.786+0000 7fad4cf73640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad40052990 0x7fad40177c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.786+0000 7fad47fff640 1 -- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad40052990 msgr2=0x7fad40177c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.786+0000 7fad47fff640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad40052990 0x7fad40177c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.786+0000 7fad47fff640 1 -- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad4004e810 msgr2=0x7fad40052450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:06.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.786+0000 7fad47fff640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad4004e810 0x7fad40052450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:06.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.786+0000 7fad47fff640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40174aa0 con 0x7fad380047c0 2023-08-25T09:50:06.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.787+0000 7fad47fff640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 0x7fad4004bb50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00fb90 tx=0x7fad3c00d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.788+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00fc80 con 0x7fad380047c0 2023-08-25T09:50:06.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.788+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c00fde0 con 0x7fad380047c0 2023-08-25T09:50:06.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.788+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00de70 con 0x7fad380047c0 2023-08-25T09:50:06.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.788+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad40174d80 con 0x7fad380047c0 2023-08-25T09:50:06.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.789+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f87e8053ba0 con 0x7f87f80c9730 2023-08-25T09:50:06.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.789+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad400d00a0 con 0x7fad380047c0 2023-08-25T09:50:06.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.792+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad400e2d60 con 0x7fad380047c0 2023-08-25T09:50:06.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.793+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fad3c0050d0 con 0x7fad380047c0 2023-08-25T09:50:06.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.795+0000 7fad44ff9640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad2c06a080 0x7fad2c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:06.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.795+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fad3c088a90 con 0x7fad380047c0 2023-08-25T09:50:06.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.796+0000 7fad477fe640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad2c06a080 0x7fad2c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:06.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.797+0000 7fad477fe640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad2c06a080 0x7fad2c06c540 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fad34035000 tx=0x7fad34030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:06.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.792+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb4a8155200 con 0x7fb4a8158ed0 2023-08-25T09:50:06.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:06.808+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fad3c053df0 con 0x7fad380047c0 2023-08-25T09:50:07.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.168+0000 7f43d359e640 1 -- 172.21.15.71:0/1507401231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f43d406e8a0 con 0x7f43d40c8940 2023-08-25T09:50:07.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.168+0000 7f43c2ffd640 1 -- 172.21.15.71:0/1507401231 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f43d406e8a0 con 0x7f43d40c8940 2023-08-25T09:50:07.172 INFO:teuthology.orchestra.run.smithi071.stdout:60129542156 2023-08-25T09:50:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 -- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f43b406a120 msgr2=0x7f43b406c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f43b406a120 0x7f43b406c5e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43cc0012e0 tx=0x7f43cc00a040 comp rx=0 tx=0).stop 2023-08-25T09:50:07.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 -- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40c8940 msgr2=0x7f43d4156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40c8940 0x7f43d4156210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f43c4034000 tx=0x7f43c40046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 -- 172.21.15.71:0/1507401231 shutdown_connections 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f43b406a120 0x7f43b406c5e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43d40c8940 0x7f43d4156210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d40aea70 0x7f43d4155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.172+0000 7f43c0ff9640 1 --2- 172.21.15.71:0/1507401231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d40ae150 0x7f43d41e00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.173+0000 7f43c0ff9640 1 -- 172.21.15.71:0/1507401231 >> 172.21.15.71:0/1507401231 conn(0x7f43d406c280 msgr2=0x7f43d40c8230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.173+0000 7f43c0ff9640 1 -- 172.21.15.71:0/1507401231 shutdown_connections 2023-08-25T09:50:07.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.173+0000 7f43c0ff9640 1 -- 172.21.15.71:0/1507401231 wait complete. 2023-08-25T09:50:07.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.183+0000 7f7586149640 1 -- 172.21.15.71:0/3568346600 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7560005500 con 0x7f75800aea70 2023-08-25T09:50:07.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.184+0000 7f7575ffb640 1 -- 172.21.15.71:0/3568346600 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f757c021070 con 0x7f75800aea70 2023-08-25T09:50:07.188 INFO:teuthology.orchestra.run.smithi071.stdout:60129542156 2023-08-25T09:50:07.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.187+0000 7f756b7fe640 1 -- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f756406a120 msgr2=0x7f756406c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.187+0000 7f756b7fe640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f756406a120 0x7f756406c5e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f757800bfa0 tx=0x7f7578034000 comp rx=0 tx=0).stop 2023-08-25T09:50:07.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 -- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75800aea70 msgr2=0x7f75801e2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75800aea70 0x7f75801e2280 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f757c00e880 tx=0x7f757c00ed40 comp rx=0 tx=0).stop 2023-08-25T09:50:07.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 -- 172.21.15.71:0/3568346600 shutdown_connections 2023-08-25T09:50:07.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f756406a120 0x7f756406c5e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75800af400 0x7f75801e0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f75800aea70 0x7f75801e2280 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 --2- 172.21.15.71:0/3568346600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75800ae150 0x7f75800ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 -- 172.21.15.71:0/3568346600 >> 172.21.15.71:0/3568346600 conn(0x7f758006c280 msgr2=0x7f75800a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 -- 172.21.15.71:0/3568346600 shutdown_connections 2023-08-25T09:50:07.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.188+0000 7f756b7fe640 1 -- 172.21.15.71:0/3568346600 wait complete. 2023-08-25T09:50:07.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.190+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc37404f480 con 0x7fc3740509f0 2023-08-25T09:50:07.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.190+0000 7fc3637fe640 1 -- 172.21.15.71:0/2134266830 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc370056ed0 con 0x7fc3740509f0 2023-08-25T09:50:07.193 INFO:teuthology.orchestra.run.smithi071.stdout:64424509453 2023-08-25T09:50:07.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5a180c45d0 con 0x7f5a1815dd50 2023-08-25T09:50:07.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.194+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc36806a050 msgr2=0x7fc36806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.194+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc36806a050 0x7fc36806c510 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc36c02b9e0 tx=0x7fc36c008030 comp rx=0 tx=0).stop 2023-08-25T09:50:07.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.194+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740509f0 msgr2=0x7fc374052900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.194+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740509f0 0x7fc374052900 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc37000c410 tx=0x7fc37000c810 comp rx=0 tx=0).stop 2023-08-25T09:50:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 shutdown_connections 2023-08-25T09:50:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3740e2650 0x7fc37416a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc36806a050 0x7fc36806c510 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc3740509f0 0x7fc374052900 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7fc37bfff640 1 --2- 172.21.15.71:0/2134266830 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc37404fc70 0x7fc3740523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 >> 172.21.15.71:0/2134266830 conn(0x7fc3740545e0 msgr2=0x7fc3740d7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.195+0000 7f5a0ffff640 1 -- 172.21.15.71:0/2350614440 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5a180c45d0 con 0x7f5a1815dd50 2023-08-25T09:50:07.199 INFO:teuthology.orchestra.run.smithi071.stdout:60129542156 2023-08-25T09:50:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.196+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 shutdown_connections 2023-08-25T09:50:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.196+0000 7fc37bfff640 1 -- 172.21.15.71:0/2134266830 wait complete. 2023-08-25T09:50:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.196+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff0e804e420 con 0x7ff0e806abc0 2023-08-25T09:50:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.197+0000 7ff0e4ff9640 1 -- 172.21.15.71:0/219576894 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff0d80784e0 con 0x7ff0e806abc0 2023-08-25T09:50:07.200 INFO:teuthology.orchestra.run.smithi071.stdout:64424509452 2023-08-25T09:50:07.201 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542156 got 60129542156 for osd.0 2023-08-25T09:50:07.202 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f59f806a080 msgr2=0x7f59f806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f59f806a080 0x7f59f806c540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800a400 tx=0x7f5a0800d040 comp rx=0 tx=0).stop 2023-08-25T09:50:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 msgr2=0x7f5a181f1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 0x7f5a181f1480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5a100097f0 tx=0x7f5a100046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb4a81f6260 con 0x7fb4a8158ed0 2023-08-25T09:50:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 shutdown_connections 2023-08-25T09:50:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a180c06b0 0x7f5a181f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f59f806a080 0x7f59f806c540 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a1815dd50 0x7f5a181f1480 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 --2- 172.21.15.71:0/2350614440 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a18069850 0x7f5a18157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 >> 172.21.15.71:0/2350614440 conn(0x7f5a180c6cc0 msgr2=0x7f5a180bffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.199+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 shutdown_connections 2023-08-25T09:50:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.200+0000 7f5a1d214640 1 -- 172.21.15.71:0/2350614440 wait complete. 2023-08-25T09:50:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.200+0000 7fb4a57fa640 1 -- 172.21.15.71:0/3446591096 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb4a81f6260 con 0x7fb4a8158ed0 2023-08-25T09:50:07.206 INFO:teuthology.orchestra.run.smithi071.stdout:64424509452 2023-08-25T09:50:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.201+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff0c806a0d0 msgr2=0x7ff0c806c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.201+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff0c806a0d0 0x7ff0c806c590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80ca8a0 tx=0x7ff0d0009040 comp rx=0 tx=0).stop 2023-08-25T09:50:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 msgr2=0x7ff0e81f2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 0x7ff0e81f2e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0e806b050 tx=0x7ff0d8004730 comp rx=0 tx=0).stop 2023-08-25T09:50:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 shutdown_connections 2023-08-25T09:50:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff0c806a0d0 0x7ff0c806c590 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0e80c0840 0x7ff0e8150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0e80acd20 0x7ff0e80c1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 --2- 172.21.15.71:0/219576894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0e806abc0 0x7ff0e81f2e40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 >> 172.21.15.71:0/219576894 conn(0x7ff0e80c6cc0 msgr2=0x7ff0e8153c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 shutdown_connections 2023-08-25T09:50:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.202+0000 7ff0ee0e1640 1 -- 172.21.15.71:0/219576894 wait complete. 2023-08-25T09:50:07.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.204+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb48806a050 msgr2=0x7fb48806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.204+0000 7fb49e7fc640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb48806a050 0x7fb48806c510 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb498000f60 tx=0x7fb498009030 comp rx=0 tx=0).stop 2023-08-25T09:50:07.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.204+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 msgr2=0x7fb4a81f6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.204+0000 7fb49e7fc640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 0x7fb4a81f6540 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8071d70 tx=0x7fb4a000df70 comp rx=0 tx=0).stop 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.204+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 shutdown_connections 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.205+0000 7fb49e7fc640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4a80c0cd0 0x7fb4a8158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.205+0000 7fb49e7fc640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb48806a050 0x7fb48806c510 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.205+0000 7fb49e7fc640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb4a8158ed0 0x7fb4a81f6540 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.205+0000 7fb49e7fc640 1 --2- 172.21.15.71:0/3446591096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4940039e0 0x7fb4a814d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.205+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 >> 172.21.15.71:0/3446591096 conn(0x7fb4a80654f0 msgr2=0x7fb4a806b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.205+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f87f8156990 con 0x7f87f80c9730 2023-08-25T09:50:07.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.206+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 shutdown_connections 2023-08-25T09:50:07.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.206+0000 7fb49e7fc640 1 -- 172.21.15.71:0/3446591096 wait complete. 2023-08-25T09:50:07.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.206+0000 7f87f4ff9640 1 -- 172.21.15.71:0/616212211 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f87e80571c0 con 0x7f87f80c9730 2023-08-25T09:50:07.213 INFO:teuthology.orchestra.run.smithi071.stdout:60129542155 2023-08-25T09:50:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.209+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d806a0a0 msgr2=0x7f87d806c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.209+0000 7f87fdb12640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d806a0a0 0x7f87d806c560 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f87f81ef750 tx=0x7f87e0009040 comp rx=0 tx=0).stop 2023-08-25T09:50:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.209+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 msgr2=0x7f87f8150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 0x7f87f8150950 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87e800cc20 tx=0x7f87e800a430 comp rx=0 tx=0).stop 2023-08-25T09:50:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 shutdown_connections 2023-08-25T09:50:07.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f81f2760 0x7f87f813c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d806a0a0 0x7f87d806c560 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f80b3270 0x7f87f81ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 --2- 172.21.15.71:0/616212211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f80c9730 0x7f87f8150950 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 >> 172.21.15.71:0/616212211 conn(0x7f87f80654f0 msgr2=0x7f87f80c3640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 shutdown_connections 2023-08-25T09:50:07.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.210+0000 7f87fdb12640 1 -- 172.21.15.71:0/616212211 wait complete. 2023-08-25T09:50:07.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542156 got 60129542156 for osd.2 2023-08-25T09:50:07.217 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.225 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509453 got 64424509453 for osd.6 2023-08-25T09:50:07.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.223+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fad4004f480 con 0x7fad380047c0 2023-08-25T09:50:07.225 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.224+0000 7fad44ff9640 1 -- 172.21.15.71:0/2465226879 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad3c057410 con 0x7fad380047c0 2023-08-25T09:50:07.226 INFO:teuthology.orchestra.run.smithi071.stdout:64424509452 2023-08-25T09:50:07.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad2c06a080 msgr2=0x7fad2c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad2c06a080 0x7fad2c06c540 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fad34035000 tx=0x7fad34030000 comp rx=0 tx=0).stop 2023-08-25T09:50:07.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 msgr2=0x7fad4004bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 0x7fad4004bb50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fad3c00fb90 tx=0x7fad3c00d520 comp rx=0 tx=0).stop 2023-08-25T09:50:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 shutdown_connections 2023-08-25T09:50:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad2c06a080 0x7fad2c06c540 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad40052990 0x7fad40177c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad4004e810 0x7fad40052450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 --2- 172.21.15.71:0/2465226879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad380047c0 0x7fad4004bb50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.226+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 >> 172.21.15.71:0/2465226879 conn(0x7fad400545e0 msgr2=0x7fad400d3120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.228+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 shutdown_connections 2023-08-25T09:50:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.228+0000 7fad4e9d6640 1 -- 172.21.15.71:0/2465226879 wait complete. 2023-08-25T09:50:07.232 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542156 got 60129542156 for osd.1 2023-08-25T09:50:07.232 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509452 got 64424509452 for osd.4 2023-08-25T09:50:07.234 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.234 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542155 got 60129542155 for osd.3 2023-08-25T09:50:07.235 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509452 got 64424509452 for osd.7 2023-08-25T09:50:07.236 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509452 got 64424509452 for osd.5 2023-08-25T09:50:07.246 DEBUG:teuthology.parallel:result is None 2023-08-25T09:50:07.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-08-25T09:50:07.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:07.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.494+0000 7f4ce5870640 1 Processor -- start 2023-08-25T09:50:07.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.495+0000 7f4ce5870640 1 -- start start 2023-08-25T09:50:07.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4ce5870640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ce01594d0 0x7f4ce00c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4ce5870640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce006a960 con 0x7f4ce006cc70 2023-08-25T09:50:07.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4ce5870640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce006ac20 con 0x7f4ce00c3940 2023-08-25T09:50:07.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4ce5870640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce006af00 con 0x7f4ce01594d0 2023-08-25T09:50:07.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4cdeffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ce01594d0 0x7f4ce00c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4cd7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4ce006cc70 0x7f4ce014f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46460/0 (socket says 172.21.15.71:46460) 2023-08-25T09:50:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.496+0000 7f4cd7fff640 1 -- 172.21.15.71:0/1228441164 learned_addr learned my addr 172.21.15.71:0/1228441164 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.497+0000 7f4cdeffd640 1 -- 172.21.15.71:0/1228441164 >> v1:172.21.15.164:6789/0 conn(0x7f4ce00c3940 legacy=0x7f4ce01598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.497+0000 7f4cdeffd640 1 -- 172.21.15.71:0/1228441164 >> v1:172.21.15.71:6789/0 conn(0x7f4ce006cc70 legacy=0x7f4ce014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.497+0000 7f4cde7fc640 1 -- 172.21.15.71:0/1228441164 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744798812 0 0) 0x7f4ce006a960 con 0x7f4ce006cc70 2023-08-25T09:50:07.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.497+0000 7f4cdeffd640 1 -- 172.21.15.71:0/1228441164 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce00c2e60 con 0x7f4ce01594d0 2023-08-25T09:50:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4cdeffd640 1 --2- 172.21.15.71:0/1228441164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce00c4c90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00097a0 tx=0x7f4cd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecefa75d201368d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4cde7fc640 1 -- 172.21.15.71:0/1228441164 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0031830 con 0x7f4ce01594d0 2023-08-25T09:50:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4cde7fc640 1 -- 172.21.15.71:0/1228441164 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd0031990 con 0x7f4ce01594d0 2023-08-25T09:50:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4cde7fc640 1 -- 172.21.15.71:0/1228441164 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0031d20 con 0x7f4ce01594d0 2023-08-25T09:50:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4ce5870640 1 -- 172.21.15.71:0/1228441164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 msgr2=0x7f4ce00c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1228441164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce00c4c90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00097a0 tx=0x7f4cd002b8d0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4ce5870640 1 -- 172.21.15.71:0/1228441164 shutdown_connections 2023-08-25T09:50:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1228441164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce00c4c90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4ce5870640 1 -- 172.21.15.71:0/1228441164 >> 172.21.15.71:0/1228441164 conn(0x7f4ce00654f0 msgr2=0x7f4ce0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.498+0000 7f4ce5870640 1 -- 172.21.15.71:0/1228441164 shutdown_connections 2023-08-25T09:50:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.499+0000 7f4ce5870640 1 -- 172.21.15.71:0/1228441164 wait complete. 2023-08-25T09:50:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.500+0000 7f4ce5870640 1 Processor -- start 2023-08-25T09:50:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.500+0000 7f4ce5870640 1 -- start start 2023-08-25T09:50:07.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4ce5870640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ce006cc70 0x7f4ce01639e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4ce5870640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ce00c3940 0x7f4ce0158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4ce5870640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce0158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4ce5870640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce0157c20 con 0x7f4ce01594d0 2023-08-25T09:50:07.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4ce5870640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce014cdf0 con 0x7f4ce00c3940 2023-08-25T09:50:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4ce5870640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce0164330 con 0x7f4ce006cc70 2023-08-25T09:50:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce0158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4cdf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce0158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60320/0 (socket says 172.21.15.71:60320) 2023-08-25T09:50:07.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.501+0000 7f4cdf7fe640 1 -- 172.21.15.71:0/1645522954 learned_addr learned my addr 172.21.15.71:0/1645522954 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:07.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdeffd640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ce006cc70 0x7f4ce01639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cd7fff640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ce00c3940 0x7f4ce0158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdf7fe640 1 -- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ce006cc70 msgr2=0x7f4ce01639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdf7fe640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ce006cc70 0x7f4ce01639e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdf7fe640 1 -- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ce00c3940 msgr2=0x7f4ce0158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdf7fe640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ce00c3940 0x7f4ce0158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdf7fe640 1 -- 172.21.15.71:0/1645522954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd0009400 con 0x7f4ce01594d0 2023-08-25T09:50:07.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdf7fe640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce0158580 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800b970 tx=0x7f4cd800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:07.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd800cb40 con 0x7f4ce01594d0 2023-08-25T09:50:07.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd800cca0 con 0x7f4ce01594d0 2023-08-25T09:50:07.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce014cb10 con 0x7f4ce01594d0 2023-08-25T09:50:07.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8014680 con 0x7f4ce01594d0 2023-08-25T09:50:07.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.502+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce0157810 con 0x7f4ce01594d0 2023-08-25T09:50:07.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.503+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce0157d60 con 0x7f4ce01594d0 2023-08-25T09:50:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.504+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4cd801d3b0 con 0x7f4ce01594d0 2023-08-25T09:50:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.505+0000 7f4cdcff9640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4cc006a050 0x7f4cc006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.505+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f4cd80887e0 con 0x7f4ce01594d0 2023-08-25T09:50:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.505+0000 7f4cdeffd640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4cc006a050 0x7f4cc006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.506+0000 7f4cdeffd640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4cc006a050 0x7f4cc006c510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0036000 tx=0x7f4cd0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.509+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4ce0157d60 con 0x7f4ce01594d0 2023-08-25T09:50:07.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.697+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ce00c3420 con 0x7f4cc006a050 2023-08-25T09:50:07.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.712+0000 7f4cdcff9640 1 -- 172.21.15.71:0/1645522954 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367369 (secure 0 0 0) 0x7f4cd0031b00 con 0x7f4cc006a050 2023-08-25T09:50:07.720 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.720+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4cc006a050 msgr2=0x7f4cc006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.720+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4cc006a050 0x7f4cc006c510 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0036000 tx=0x7f4cd0008030 comp rx=0 tx=0).stop 2023-08-25T09:50:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.720+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 msgr2=0x7f4ce0158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce0158580 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800b970 tx=0x7f4cd800be30 comp rx=0 tx=0).stop 2023-08-25T09:50:07.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 shutdown_connections 2023-08-25T09:50:07.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ce00c3940 0x7f4ce0158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4cc006a050 0x7f4cc006c510 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ce01594d0 0x7f4ce0158580 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 --2- 172.21.15.71:0/1645522954 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ce006cc70 0x7f4ce01639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 >> 172.21.15.71:0/1645522954 conn(0x7f4ce00654f0 msgr2=0x7f4ce014d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 shutdown_connections 2023-08-25T09:50:07.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.721+0000 7f4ce5870640 1 -- 172.21.15.71:0/1645522954 wait complete. 2023-08-25T09:50:07.741 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2023-08-25T09:50:06.645565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596413,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227088,"kb_used_data":2640,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747632,"statfs":{"total":773094113280,"available":772861575168,"internally_reserved":0,"allocated":2703360,"data_stored":2202674,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3187,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004937"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532259+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:49:40.532259+0000","last_peered":"2023-08-25T09:49:40.532259+0000","last_clean":"2023-08-25T09:49:40.532259+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:49:40.532259+0000","last_undegraded":"2023-08-25T09:49:40.532259+0000","last_fullsized":"2023-08-25T09:49:40.532259+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:08:44.154380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670137+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:49:39.670137+0000","last_peered":"2023-08-25T09:49:39.670137+0000","last_clean":"2023-08-25T09:49:39.670137+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:49:39.670137+0000","last_undegraded":"2023-08-25T09:49:39.670137+0000","last_fullsized":"2023-08-25T09:49:39.670137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:19:21.160750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672075+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:49:39.672075+0000","last_peered":"2023-08-25T09:49:39.672075+0000","last_clean":"2023-08-25T09:49:39.672075+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:49:39.672075+0000","last_undegraded":"2023-08-25T09:49:39.672075+0000","last_fullsized":"2023-08-25T09:49:39.672075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:39:44.896839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.667983+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:49:39.667983+0000","last_peered":"2023-08-25T09:49:39.667983+0000","last_clean":"2023-08-25T09:49:39.667983+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:49:39.667983+0000","last_undegraded":"2023-08-25T09:49:39.667983+0000","last_fullsized":"2023-08-25T09:49:39.667983+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:52:21.149049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610317+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:49:40.610317+0000","last_peered":"2023-08-25T09:49:40.610317+0000","last_clean":"2023-08-25T09:49:40.610317+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:49:40.610317+0000","last_undegraded":"2023-08-25T09:49:40.610317+0000","last_fullsized":"2023-08-25T09:49:40.610317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.067887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609551+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:49:40.609551+0000","last_peered":"2023-08-25T09:49:40.609551+0000","last_clean":"2023-08-25T09:49:40.609551+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:49:40.609551+0000","last_undegraded":"2023-08-25T09:49:40.609551+0000","last_fullsized":"2023-08-25T09:49:40.609551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:31:30.676338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669356+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:49:39.669356+0000","last_peered":"2023-08-25T09:49:39.669356+0000","last_clean":"2023-08-25T09:49:39.669356+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:49:39.669356+0000","last_undegraded":"2023-08-25T09:49:39.669356+0000","last_fullsized":"2023-08-25T09:49:39.669356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:52:41.076094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672443+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:49:39.672443+0000","last_peered":"2023-08-25T09:49:39.672443+0000","last_clean":"2023-08-25T09:49:39.672443+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:49:39.672443+0000","last_undegraded":"2023-08-25T09:49:39.672443+0000","last_fullsized":"2023-08-25T09:49:39.672443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:41:06.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670837+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:49:39.670837+0000","last_peered":"2023-08-25T09:49:39.670837+0000","last_clean":"2023-08-25T09:49:39.670837+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:49:39.670837+0000","last_undegraded":"2023-08-25T09:49:39.670837+0000","last_fullsized":"2023-08-25T09:49:39.670837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:43.561619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669087+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:49:39.669087+0000","last_peered":"2023-08-25T09:49:39.669087+0000","last_clean":"2023-08-25T09:49:39.669087+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:49:39.669087+0000","last_undegraded":"2023-08-25T09:49:39.669087+0000","last_fullsized":"2023-08-25T09:49:39.669087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:29:31.719174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668364+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:49:39.668364+0000","last_peered":"2023-08-25T09:49:39.668364+0000","last_clean":"2023-08-25T09:49:39.668364+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:49:39.668364+0000","last_undegraded":"2023-08-25T09:49:39.668364+0000","last_fullsized":"2023-08-25T09:49:39.668364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:54:21.024745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669503+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:49:39.669503+0000","last_peered":"2023-08-25T09:49:39.669503+0000","last_clean":"2023-08-25T09:49:39.669503+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:49:39.669503+0000","last_undegraded":"2023-08-25T09:49:39.669503+0000","last_fullsized":"2023-08-25T09:49:39.669503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:38.132579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533392+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:49:40.533392+0000","last_peered":"2023-08-25T09:49:40.533392+0000","last_clean":"2023-08-25T09:49:40.533392+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:49:40.533392+0000","last_undegraded":"2023-08-25T09:49:40.533392+0000","last_fullsized":"2023-08-25T09:49:40.533392+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:44:04.188030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"29'5","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.538756+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:50:05.538756+0000","last_peered":"2023-08-25T09:50:05.538756+0000","last_clean":"2023-08-25T09:50:05.538756+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:50:05.538756+0000","last_undegraded":"2023-08-25T09:50:05.538756+0000","last_fullsized":"2023-08-25T09:50:05.538756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:36:03.677003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314444+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:49:40.314444+0000","last_peered":"2023-08-25T09:49:40.314444+0000","last_clean":"2023-08-25T09:49:40.314444+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:49:40.314444+0000","last_undegraded":"2023-08-25T09:49:40.314444+0000","last_fullsized":"2023-08-25T09:49:40.314444+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:24:33.940678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670152+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:49:39.670152+0000","last_peered":"2023-08-25T09:49:39.670152+0000","last_clean":"2023-08-25T09:49:39.670152+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:49:39.670152+0000","last_undegraded":"2023-08-25T09:49:39.670152+0000","last_fullsized":"2023-08-25T09:49:39.670152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:04.589805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671210+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:49:39.671210+0000","last_peered":"2023-08-25T09:49:39.671210+0000","last_clean":"2023-08-25T09:49:39.671210+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:49:39.671210+0000","last_undegraded":"2023-08-25T09:49:39.671210+0000","last_fullsized":"2023-08-25T09:49:39.671210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:21.872702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610243+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:49:40.610243+0000","last_peered":"2023-08-25T09:49:40.610243+0000","last_clean":"2023-08-25T09:49:40.610243+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:49:40.610243+0000","last_undegraded":"2023-08-25T09:49:40.610243+0000","last_fullsized":"2023-08-25T09:49:40.610243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:40:11.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610311+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:49:40.610311+0000","last_peered":"2023-08-25T09:49:40.610311+0000","last_clean":"2023-08-25T09:49:40.610311+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:49:40.610311+0000","last_undegraded":"2023-08-25T09:49:40.610311+0000","last_fullsized":"2023-08-25T09:49:40.610311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:24:36.237719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532845+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:49:40.532845+0000","last_peered":"2023-08-25T09:49:40.532845+0000","last_clean":"2023-08-25T09:49:40.532845+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:49:40.532845+0000","last_undegraded":"2023-08-25T09:49:40.532845+0000","last_fullsized":"2023-08-25T09:49:40.532845+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:17.775607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669479+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:49:39.669479+0000","last_peered":"2023-08-25T09:49:39.669479+0000","last_clean":"2023-08-25T09:49:39.669479+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:49:39.669479+0000","last_undegraded":"2023-08-25T09:49:39.669479+0000","last_fullsized":"2023-08-25T09:49:39.669479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:24:10.790283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669214+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:49:39.669214+0000","last_peered":"2023-08-25T09:49:39.669214+0000","last_clean":"2023-08-25T09:49:39.669214+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:49:39.669214+0000","last_undegraded":"2023-08-25T09:49:39.669214+0000","last_fullsized":"2023-08-25T09:49:39.669214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:45:44.040670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609148+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:49:40.609148+0000","last_peered":"2023-08-25T09:49:40.609148+0000","last_clean":"2023-08-25T09:49:40.609148+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:49:40.609148+0000","last_undegraded":"2023-08-25T09:49:40.609148+0000","last_fullsized":"2023-08-25T09:49:40.609148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:30:29.949537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610535+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:49:40.610535+0000","last_peered":"2023-08-25T09:49:40.610535+0000","last_clean":"2023-08-25T09:49:40.610535+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:49:40.610535+0000","last_undegraded":"2023-08-25T09:49:40.610535+0000","last_fullsized":"2023-08-25T09:49:40.610535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:33:47.145180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608671+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.608671+0000","last_peered":"2023-08-25T09:49:40.608671+0000","last_clean":"2023-08-25T09:49:40.608671+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:49:40.608671+0000","last_undegraded":"2023-08-25T09:49:40.608671+0000","last_fullsized":"2023-08-25T09:49:40.608671+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:49:14.644166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"29'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535841+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:50:05.535841+0000","last_peered":"2023-08-25T09:50:05.535841+0000","last_clean":"2023-08-25T09:50:05.535841+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:50:05.535841+0000","last_undegraded":"2023-08-25T09:50:05.535841+0000","last_fullsized":"2023-08-25T09:50:05.535841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:33:42.072752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609238+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:49:40.609238+0000","last_peered":"2023-08-25T09:49:40.609238+0000","last_clean":"2023-08-25T09:49:40.609238+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:49:40.609238+0000","last_undegraded":"2023-08-25T09:49:40.609238+0000","last_fullsized":"2023-08-25T09:49:40.609238+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:01:49.724298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609229+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:49:40.609229+0000","last_peered":"2023-08-25T09:49:40.609229+0000","last_clean":"2023-08-25T09:49:40.609229+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:49:40.609229+0000","last_undegraded":"2023-08-25T09:49:40.609229+0000","last_fullsized":"2023-08-25T09:49:40.609229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:35:14.991310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"29'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535643+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:50:05.535643+0000","last_peered":"2023-08-25T09:50:05.535643+0000","last_clean":"2023-08-25T09:50:05.535643+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:50:05.535643+0000","last_undegraded":"2023-08-25T09:50:05.535643+0000","last_fullsized":"2023-08-25T09:50:05.535643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:36:04.594929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669714+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:49:39.669714+0000","last_peered":"2023-08-25T09:49:39.669714+0000","last_clean":"2023-08-25T09:49:39.669714+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:49:39.669714+0000","last_undegraded":"2023-08-25T09:49:39.669714+0000","last_fullsized":"2023-08-25T09:49:39.669714+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:13:29.132994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671434+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:49:39.671434+0000","last_peered":"2023-08-25T09:49:39.671434+0000","last_clean":"2023-08-25T09:49:39.671434+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:49:39.671434+0000","last_undegraded":"2023-08-25T09:49:39.671434+0000","last_fullsized":"2023-08-25T09:49:39.671434+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:08:04.314595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533055+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:49:40.533055+0000","last_peered":"2023-08-25T09:49:40.533055+0000","last_clean":"2023-08-25T09:49:40.533055+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:49:40.533055+0000","last_undegraded":"2023-08-25T09:49:40.533055+0000","last_fullsized":"2023-08-25T09:49:40.533055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:29.319338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609374+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:49:40.609374+0000","last_peered":"2023-08-25T09:49:40.609374+0000","last_clean":"2023-08-25T09:49:40.609374+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:49:40.609374+0000","last_undegraded":"2023-08-25T09:49:40.609374+0000","last_fullsized":"2023-08-25T09:49:40.609374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:11:38.846168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"29'12","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.537617+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:50:05.537617+0000","last_peered":"2023-08-25T09:50:05.537617+0000","last_clean":"2023-08-25T09:50:05.537617+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:50:05.537617+0000","last_undegraded":"2023-08-25T09:50:05.537617+0000","last_fullsized":"2023-08-25T09:50:05.537617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:19:49.118130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669680+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:49:39.669680+0000","last_peered":"2023-08-25T09:49:39.669680+0000","last_clean":"2023-08-25T09:49:39.669680+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:49:39.669680+0000","last_undegraded":"2023-08-25T09:49:39.669680+0000","last_fullsized":"2023-08-25T09:49:39.669680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:15:08.913052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532469+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:49:40.532469+0000","last_peered":"2023-08-25T09:49:40.532469+0000","last_clean":"2023-08-25T09:49:40.532469+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:49:40.532469+0000","last_undegraded":"2023-08-25T09:49:40.532469+0000","last_fullsized":"2023-08-25T09:49:40.532469+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:54:59.656364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669355+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:49:39.669355+0000","last_peered":"2023-08-25T09:49:39.669355+0000","last_clean":"2023-08-25T09:49:39.669355+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:49:39.669355+0000","last_undegraded":"2023-08-25T09:49:39.669355+0000","last_fullsized":"2023-08-25T09:49:39.669355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:48.989731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610694+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:49:40.610694+0000","last_peered":"2023-08-25T09:49:40.610694+0000","last_clean":"2023-08-25T09:49:40.610694+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:49:40.610694+0000","last_undegraded":"2023-08-25T09:49:40.610694+0000","last_fullsized":"2023-08-25T09:49:40.610694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:46.063058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670003+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:49:39.670003+0000","last_peered":"2023-08-25T09:49:39.670003+0000","last_clean":"2023-08-25T09:49:39.670003+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:49:39.670003+0000","last_undegraded":"2023-08-25T09:49:39.670003+0000","last_fullsized":"2023-08-25T09:49:39.670003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:13:11.987428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608483+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:49:40.608483+0000","last_peered":"2023-08-25T09:49:40.608483+0000","last_clean":"2023-08-25T09:49:40.608483+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:49:40.608483+0000","last_undegraded":"2023-08-25T09:49:40.608483+0000","last_fullsized":"2023-08-25T09:49:40.608483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:28:33.537079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608779+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:49:40.608779+0000","last_peered":"2023-08-25T09:49:40.608779+0000","last_clean":"2023-08-25T09:49:40.608779+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:49:40.608779+0000","last_undegraded":"2023-08-25T09:49:40.608779+0000","last_fullsized":"2023-08-25T09:49:40.608779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:01:10.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.534056+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:49:40.534056+0000","last_peered":"2023-08-25T09:49:40.534056+0000","last_clean":"2023-08-25T09:49:40.534056+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:49:40.534056+0000","last_undegraded":"2023-08-25T09:49:40.534056+0000","last_fullsized":"2023-08-25T09:49:40.534056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:30.208912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.730451+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:50:05.730451+0000","last_peered":"2023-08-25T09:50:05.730451+0000","last_clean":"2023-08-25T09:50:05.730451+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:50:05.730451+0000","last_undegraded":"2023-08-25T09:50:05.730451+0000","last_fullsized":"2023-08-25T09:50:05.730451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:46:45.992240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668225+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:49:39.668225+0000","last_peered":"2023-08-25T09:49:39.668225+0000","last_clean":"2023-08-25T09:49:39.668225+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:49:39.668225+0000","last_undegraded":"2023-08-25T09:49:39.668225+0000","last_fullsized":"2023-08-25T09:49:39.668225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:00:13.974928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669196+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:49:39.669196+0000","last_peered":"2023-08-25T09:49:39.669196+0000","last_clean":"2023-08-25T09:49:39.669196+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:49:39.669196+0000","last_undegraded":"2023-08-25T09:49:39.669196+0000","last_fullsized":"2023-08-25T09:49:39.669196+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:28:03.992015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671806+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:49:39.671806+0000","last_peered":"2023-08-25T09:49:39.671806+0000","last_clean":"2023-08-25T09:49:39.671806+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:49:39.671806+0000","last_undegraded":"2023-08-25T09:49:39.671806+0000","last_fullsized":"2023-08-25T09:49:39.671806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:14:52.844903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533580+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:49:40.533580+0000","last_peered":"2023-08-25T09:49:40.533580+0000","last_clean":"2023-08-25T09:49:40.533580+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:49:40.533580+0000","last_undegraded":"2023-08-25T09:49:40.533580+0000","last_fullsized":"2023-08-25T09:49:40.533580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:00.063674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609540+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:49:40.609540+0000","last_peered":"2023-08-25T09:49:40.609540+0000","last_clean":"2023-08-25T09:49:40.609540+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:49:40.609540+0000","last_undegraded":"2023-08-25T09:49:40.609540+0000","last_fullsized":"2023-08-25T09:49:40.609540+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:59:18.199533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532919+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:49:40.532919+0000","last_peered":"2023-08-25T09:49:40.532919+0000","last_clean":"2023-08-25T09:49:40.532919+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:49:40.532919+0000","last_undegraded":"2023-08-25T09:49:40.532919+0000","last_fullsized":"2023-08-25T09:49:40.532919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:00:24.110176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533867+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:49:40.533867+0000","last_peered":"2023-08-25T09:49:40.533867+0000","last_clean":"2023-08-25T09:49:40.533867+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:49:40.533867+0000","last_undegraded":"2023-08-25T09:49:40.533867+0000","last_fullsized":"2023-08-25T09:49:40.533867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:25:26.896240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672608+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:49:39.672608+0000","last_peered":"2023-08-25T09:49:39.672608+0000","last_clean":"2023-08-25T09:49:39.672608+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:49:39.672608+0000","last_undegraded":"2023-08-25T09:49:39.672608+0000","last_fullsized":"2023-08-25T09:49:39.672608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:23.747043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668857+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:49:39.668857+0000","last_peered":"2023-08-25T09:49:39.668857+0000","last_clean":"2023-08-25T09:49:39.668857+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:49:39.668857+0000","last_undegraded":"2023-08-25T09:49:39.668857+0000","last_fullsized":"2023-08-25T09:49:39.668857+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:12:48.827644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533435+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:49:40.533435+0000","last_peered":"2023-08-25T09:49:40.533435+0000","last_clean":"2023-08-25T09:49:40.533435+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:49:40.533435+0000","last_undegraded":"2023-08-25T09:49:40.533435+0000","last_fullsized":"2023-08-25T09:49:40.533435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:07:36.521502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"29'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535468+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:50:05.535468+0000","last_peered":"2023-08-25T09:50:05.535468+0000","last_clean":"2023-08-25T09:50:05.535468+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:50:05.535468+0000","last_undegraded":"2023-08-25T09:50:05.535468+0000","last_fullsized":"2023-08-25T09:50:05.535468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:26:07.776398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668924+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:49:39.668924+0000","last_peered":"2023-08-25T09:49:39.668924+0000","last_clean":"2023-08-25T09:49:39.668924+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:49:39.668924+0000","last_undegraded":"2023-08-25T09:49:39.668924+0000","last_fullsized":"2023-08-25T09:49:39.668924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:22:13.379657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669047+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:49:39.669047+0000","last_peered":"2023-08-25T09:49:39.669047+0000","last_clean":"2023-08-25T09:49:39.669047+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:49:39.669047+0000","last_undegraded":"2023-08-25T09:49:39.669047+0000","last_fullsized":"2023-08-25T09:49:39.669047+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:36:02.088533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533994+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:49:40.533994+0000","last_peered":"2023-08-25T09:49:40.533994+0000","last_clean":"2023-08-25T09:49:40.533994+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:49:40.533994+0000","last_undegraded":"2023-08-25T09:49:40.533994+0000","last_fullsized":"2023-08-25T09:49:40.533994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:34:02.269298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532621+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:49:40.532621+0000","last_peered":"2023-08-25T09:49:40.532621+0000","last_clean":"2023-08-25T09:49:40.532621+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:49:40.532621+0000","last_undegraded":"2023-08-25T09:49:40.532621+0000","last_fullsized":"2023-08-25T09:49:40.532621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:52:34.012911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313601+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:49:40.313601+0000","last_peered":"2023-08-25T09:49:40.313601+0000","last_clean":"2023-08-25T09:49:40.313601+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:49:40.313601+0000","last_undegraded":"2023-08-25T09:49:40.313601+0000","last_fullsized":"2023-08-25T09:49:40.313601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:28.801938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314730+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:49:40.314730+0000","last_peered":"2023-08-25T09:49:40.314730+0000","last_clean":"2023-08-25T09:49:40.314730+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:49:40.314730+0000","last_undegraded":"2023-08-25T09:49:40.314730+0000","last_fullsized":"2023-08-25T09:49:40.314730+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:17:34.425122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669544+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:49:39.669544+0000","last_peered":"2023-08-25T09:49:39.669544+0000","last_clean":"2023-08-25T09:49:39.669544+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:49:39.669544+0000","last_undegraded":"2023-08-25T09:49:39.669544+0000","last_fullsized":"2023-08-25T09:49:39.669544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:44.776136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668255+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:49:39.668255+0000","last_peered":"2023-08-25T09:49:39.668255+0000","last_clean":"2023-08-25T09:49:39.668255+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:49:39.668255+0000","last_undegraded":"2023-08-25T09:49:39.668255+0000","last_fullsized":"2023-08-25T09:49:39.668255+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:08.959803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670236+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:49:39.670236+0000","last_peered":"2023-08-25T09:49:39.670236+0000","last_clean":"2023-08-25T09:49:39.670236+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:49:39.670236+0000","last_undegraded":"2023-08-25T09:49:39.670236+0000","last_fullsized":"2023-08-25T09:49:39.670236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:04.664807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668818+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:49:39.668818+0000","last_peered":"2023-08-25T09:49:39.668818+0000","last_clean":"2023-08-25T09:49:39.668818+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:49:39.668818+0000","last_undegraded":"2023-08-25T09:49:39.668818+0000","last_fullsized":"2023-08-25T09:49:39.668818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:45:13.832806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610040+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:49:40.610040+0000","last_peered":"2023-08-25T09:49:40.610040+0000","last_clean":"2023-08-25T09:49:40.610040+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:49:40.610040+0000","last_undegraded":"2023-08-25T09:49:40.610040+0000","last_fullsized":"2023-08-25T09:49:40.610040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:08.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609484+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:49:40.609484+0000","last_peered":"2023-08-25T09:49:40.609484+0000","last_clean":"2023-08-25T09:49:40.609484+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:49:40.609484+0000","last_undegraded":"2023-08-25T09:49:40.609484+0000","last_fullsized":"2023-08-25T09:49:40.609484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:02.892745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671904+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:49:39.671904+0000","last_peered":"2023-08-25T09:49:39.671904+0000","last_clean":"2023-08-25T09:49:39.671904+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:49:39.671904+0000","last_undegraded":"2023-08-25T09:49:39.671904+0000","last_fullsized":"2023-08-25T09:49:39.671904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:12:08.551735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":183,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670426+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:49:39.670426+0000","last_peered":"2023-08-25T09:49:39.670426+0000","last_clean":"2023-08-25T09:49:39.670426+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:49:39.670426+0000","last_undegraded":"2023-08-25T09:49:39.670426+0000","last_fullsized":"2023-08-25T09:49:39.670426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:14:32.439794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670825+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:49:39.670825+0000","last_peered":"2023-08-25T09:49:39.670825+0000","last_clean":"2023-08-25T09:49:39.670825+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:49:39.670825+0000","last_undegraded":"2023-08-25T09:49:39.670825+0000","last_fullsized":"2023-08-25T09:49:39.670825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:14:42.235888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533862+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:49:40.533862+0000","last_peered":"2023-08-25T09:49:40.533862+0000","last_clean":"2023-08-25T09:49:40.533862+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:49:40.533862+0000","last_undegraded":"2023-08-25T09:49:40.533862+0000","last_fullsized":"2023-08-25T09:49:40.533862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:47:24.625104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"29'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.730045+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:50:05.730045+0000","last_peered":"2023-08-25T09:50:05.730045+0000","last_clean":"2023-08-25T09:50:05.730045+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:50:05.730045+0000","last_undegraded":"2023-08-25T09:50:05.730045+0000","last_fullsized":"2023-08-25T09:50:05.730045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:34:01.834958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670693+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:49:39.670693+0000","last_peered":"2023-08-25T09:49:39.670693+0000","last_clean":"2023-08-25T09:49:39.670693+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:49:39.670693+0000","last_undegraded":"2023-08-25T09:49:39.670693+0000","last_fullsized":"2023-08-25T09:49:39.670693+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:53:52.223404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670888+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:49:39.670888+0000","last_peered":"2023-08-25T09:49:39.670888+0000","last_clean":"2023-08-25T09:49:39.670888+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:49:39.670888+0000","last_undegraded":"2023-08-25T09:49:39.670888+0000","last_fullsized":"2023-08-25T09:49:39.670888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:52.646017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"29'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535826+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:50:05.535826+0000","last_peered":"2023-08-25T09:50:05.535826+0000","last_clean":"2023-08-25T09:50:05.535826+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:50:05.535826+0000","last_undegraded":"2023-08-25T09:50:05.535826+0000","last_fullsized":"2023-08-25T09:50:05.535826+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:07:22.260320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668612+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:49:39.668612+0000","last_peered":"2023-08-25T09:49:39.668612+0000","last_clean":"2023-08-25T09:49:39.668612+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:49:39.668612+0000","last_undegraded":"2023-08-25T09:49:39.668612+0000","last_fullsized":"2023-08-25T09:49:39.668612+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:48:39.386372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670019+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:49:39.670019+0000","last_peered":"2023-08-25T09:49:39.670019+0000","last_clean":"2023-08-25T09:49:39.670019+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:49:39.670019+0000","last_undegraded":"2023-08-25T09:49:39.670019+0000","last_fullsized":"2023-08-25T09:49:39.670019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:29.348922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610153+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:49:40.610153+0000","last_peered":"2023-08-25T09:49:40.610153+0000","last_clean":"2023-08-25T09:49:40.610153+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:49:40.610153+0000","last_undegraded":"2023-08-25T09:49:40.610153+0000","last_fullsized":"2023-08-25T09:49:40.610153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:50:39.530928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532880+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:49:40.532880+0000","last_peered":"2023-08-25T09:49:40.532880+0000","last_clean":"2023-08-25T09:49:40.532880+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:49:40.532880+0000","last_undegraded":"2023-08-25T09:49:40.532880+0000","last_fullsized":"2023-08-25T09:49:40.532880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:08:17.159830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670546+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:49:39.670546+0000","last_peered":"2023-08-25T09:49:39.670546+0000","last_clean":"2023-08-25T09:49:39.670546+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:49:39.670546+0000","last_undegraded":"2023-08-25T09:49:39.670546+0000","last_fullsized":"2023-08-25T09:49:39.670546+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:36.871310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.729657+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:50:05.729657+0000","last_peered":"2023-08-25T09:50:05.729657+0000","last_clean":"2023-08-25T09:50:05.729657+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:50:05.729657+0000","last_undegraded":"2023-08-25T09:50:05.729657+0000","last_fullsized":"2023-08-25T09:50:05.729657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:13:22.282204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314057+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:49:40.314057+0000","last_peered":"2023-08-25T09:49:40.314057+0000","last_clean":"2023-08-25T09:49:40.314057+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:49:40.314057+0000","last_undegraded":"2023-08-25T09:49:40.314057+0000","last_fullsized":"2023-08-25T09:49:40.314057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:32:23.040483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668795+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:49:39.668795+0000","last_peered":"2023-08-25T09:49:39.668795+0000","last_clean":"2023-08-25T09:49:39.668795+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:49:39.668795+0000","last_undegraded":"2023-08-25T09:49:39.668795+0000","last_fullsized":"2023-08-25T09:49:39.668795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:21:47.452907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668565+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:49:39.668565+0000","last_peered":"2023-08-25T09:49:39.668565+0000","last_clean":"2023-08-25T09:49:39.668565+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:49:39.668565+0000","last_undegraded":"2023-08-25T09:49:39.668565+0000","last_fullsized":"2023-08-25T09:49:39.668565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:12.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313625+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:49:40.313625+0000","last_peered":"2023-08-25T09:49:40.313625+0000","last_clean":"2023-08-25T09:49:40.313625+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:49:40.313625+0000","last_undegraded":"2023-08-25T09:49:40.313625+0000","last_fullsized":"2023-08-25T09:49:40.313625+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:04:41.693477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609626+0000","last_change":"2023-08-25T09:49:34.628078+0000","last_active":"2023-08-25T09:49:40.609626+0000","last_peered":"2023-08-25T09:49:40.609626+0000","last_clean":"2023-08-25T09:49:40.609626+0000","last_became_active":"2023-08-25T09:49:34.627744+0000","last_became_peered":"2023-08-25T09:49:34.627744+0000","last_unstale":"2023-08-25T09:49:40.609626+0000","last_undegraded":"2023-08-25T09:49:40.609626+0000","last_fullsized":"2023-08-25T09:49:40.609626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:15.076203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671559+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:49:39.671559+0000","last_peered":"2023-08-25T09:49:39.671559+0000","last_clean":"2023-08-25T09:49:39.671559+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:49:39.671559+0000","last_undegraded":"2023-08-25T09:49:39.671559+0000","last_fullsized":"2023-08-25T09:49:39.671559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:50:31.566722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313898+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:49:40.313898+0000","last_peered":"2023-08-25T09:49:40.313898+0000","last_clean":"2023-08-25T09:49:40.313898+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:49:40.313898+0000","last_undegraded":"2023-08-25T09:49:40.313898+0000","last_fullsized":"2023-08-25T09:49:40.313898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:29:27.667548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668920+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:49:39.668920+0000","last_peered":"2023-08-25T09:49:39.668920+0000","last_clean":"2023-08-25T09:49:39.668920+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:49:39.668920+0000","last_undegraded":"2023-08-25T09:49:39.668920+0000","last_fullsized":"2023-08-25T09:49:39.668920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:58.529027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671825+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:49:39.671825+0000","last_peered":"2023-08-25T09:49:39.671825+0000","last_clean":"2023-08-25T09:49:39.671825+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:49:39.671825+0000","last_undegraded":"2023-08-25T09:49:39.671825+0000","last_fullsized":"2023-08-25T09:49:39.671825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:53:27.311764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313371+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:49:40.313371+0000","last_peered":"2023-08-25T09:49:40.313371+0000","last_clean":"2023-08-25T09:49:40.313371+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:49:40.313371+0000","last_undegraded":"2023-08-25T09:49:40.313371+0000","last_fullsized":"2023-08-25T09:49:40.313371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:23:01.195200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668159+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:49:39.668159+0000","last_peered":"2023-08-25T09:49:39.668159+0000","last_clean":"2023-08-25T09:49:39.668159+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:49:39.668159+0000","last_undegraded":"2023-08-25T09:49:39.668159+0000","last_fullsized":"2023-08-25T09:49:39.668159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:32:15.403697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672436+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:49:39.672436+0000","last_peered":"2023-08-25T09:49:39.672436+0000","last_clean":"2023-08-25T09:49:39.672436+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:49:39.672436+0000","last_undegraded":"2023-08-25T09:49:39.672436+0000","last_fullsized":"2023-08-25T09:49:39.672436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:53.368049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671121+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:49:39.671121+0000","last_peered":"2023-08-25T09:49:39.671121+0000","last_clean":"2023-08-25T09:49:39.671121+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:49:39.671121+0000","last_undegraded":"2023-08-25T09:49:39.671121+0000","last_fullsized":"2023-08-25T09:49:39.671121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:05:58.232199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671520+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:49:39.671520+0000","last_peered":"2023-08-25T09:49:39.671520+0000","last_clean":"2023-08-25T09:49:39.671520+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:49:39.671520+0000","last_undegraded":"2023-08-25T09:49:39.671520+0000","last_fullsized":"2023-08-25T09:49:39.671520+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:08:33.021697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"29'10","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.531847+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:50:05.531847+0000","last_peered":"2023-08-25T09:50:05.531847+0000","last_clean":"2023-08-25T09:50:05.531847+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:50:05.531847+0000","last_undegraded":"2023-08-25T09:50:05.531847+0000","last_fullsized":"2023-08-25T09:50:05.531847+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:16:55.451701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610899+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.610899+0000","last_peered":"2023-08-25T09:49:40.610899+0000","last_clean":"2023-08-25T09:49:40.610899+0000","last_became_active":"2023-08-25T09:49:34.630012+0000","last_became_peered":"2023-08-25T09:49:34.630012+0000","last_unstale":"2023-08-25T09:49:40.610899+0000","last_undegraded":"2023-08-25T09:49:40.610899+0000","last_fullsized":"2023-08-25T09:49:40.610899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:11:33.237689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610543+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:49:40.610543+0000","last_peered":"2023-08-25T09:49:40.610543+0000","last_clean":"2023-08-25T09:49:40.610543+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:49:40.610543+0000","last_undegraded":"2023-08-25T09:49:40.610543+0000","last_fullsized":"2023-08-25T09:49:40.610543+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:25.560317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671645+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:49:39.671645+0000","last_peered":"2023-08-25T09:49:39.671645+0000","last_clean":"2023-08-25T09:49:39.671645+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:49:39.671645+0000","last_undegraded":"2023-08-25T09:49:39.671645+0000","last_fullsized":"2023-08-25T09:49:39.671645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:37:05.937993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669858+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:49:39.669858+0000","last_peered":"2023-08-25T09:49:39.669858+0000","last_clean":"2023-08-25T09:49:39.669858+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:49:39.669858+0000","last_undegraded":"2023-08-25T09:49:39.669858+0000","last_fullsized":"2023-08-25T09:49:39.669858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:20:55.146183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"29'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.537356+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:50:05.537356+0000","last_peered":"2023-08-25T09:50:05.537356+0000","last_clean":"2023-08-25T09:50:05.537356+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:50:05.537356+0000","last_undegraded":"2023-08-25T09:50:05.537356+0000","last_fullsized":"2023-08-25T09:50:05.537356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:51.399369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672435+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:49:39.672435+0000","last_peered":"2023-08-25T09:49:39.672435+0000","last_clean":"2023-08-25T09:49:39.672435+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:49:39.672435+0000","last_undegraded":"2023-08-25T09:49:39.672435+0000","last_fullsized":"2023-08-25T09:49:39.672435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:41:04.943976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609915+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:49:40.609915+0000","last_peered":"2023-08-25T09:49:40.609915+0000","last_clean":"2023-08-25T09:49:40.609915+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:49:40.609915+0000","last_undegraded":"2023-08-25T09:49:40.609915+0000","last_fullsized":"2023-08-25T09:49:40.609915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:30:03.672997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.538798+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:50:05.538798+0000","last_peered":"2023-08-25T09:50:05.538798+0000","last_clean":"2023-08-25T09:50:05.538798+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:50:05.538798+0000","last_undegraded":"2023-08-25T09:50:05.538798+0000","last_fullsized":"2023-08-25T09:50:05.538798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.806748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610312+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:49:40.610312+0000","last_peered":"2023-08-25T09:49:40.610312+0000","last_clean":"2023-08-25T09:49:40.610312+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:49:40.610312+0000","last_undegraded":"2023-08-25T09:49:40.610312+0000","last_fullsized":"2023-08-25T09:49:40.610312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:15:05.871192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672132+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:49:39.672132+0000","last_peered":"2023-08-25T09:49:39.672132+0000","last_clean":"2023-08-25T09:49:39.672132+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:49:39.672132+0000","last_undegraded":"2023-08-25T09:49:39.672132+0000","last_fullsized":"2023-08-25T09:49:39.672132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:51:00.162861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609435+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:49:40.609435+0000","last_peered":"2023-08-25T09:49:40.609435+0000","last_clean":"2023-08-25T09:49:40.609435+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:49:40.609435+0000","last_undegraded":"2023-08-25T09:49:40.609435+0000","last_fullsized":"2023-08-25T09:49:40.609435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:54:36.909125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609010+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:49:40.609010+0000","last_peered":"2023-08-25T09:49:40.609010+0000","last_clean":"2023-08-25T09:49:40.609010+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:49:40.609010+0000","last_undegraded":"2023-08-25T09:49:40.609010+0000","last_fullsized":"2023-08-25T09:49:40.609010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:45.655937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671033+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:49:39.671033+0000","last_peered":"2023-08-25T09:49:39.671033+0000","last_clean":"2023-08-25T09:49:39.671033+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:49:39.671033+0000","last_undegraded":"2023-08-25T09:49:39.671033+0000","last_fullsized":"2023-08-25T09:49:39.671033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:19:24.241659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609947+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:49:40.609947+0000","last_peered":"2023-08-25T09:49:40.609947+0000","last_clean":"2023-08-25T09:49:40.609947+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:49:40.609947+0000","last_undegraded":"2023-08-25T09:49:40.609947+0000","last_fullsized":"2023-08-25T09:49:40.609947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:27:06.196565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533764+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:49:40.533764+0000","last_peered":"2023-08-25T09:49:40.533764+0000","last_clean":"2023-08-25T09:49:40.533764+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:49:40.533764+0000","last_undegraded":"2023-08-25T09:49:40.533764+0000","last_fullsized":"2023-08-25T09:49:40.533764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:53:01.633281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672281+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:49:39.672281+0000","last_peered":"2023-08-25T09:49:39.672281+0000","last_clean":"2023-08-25T09:49:39.672281+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:49:39.672281+0000","last_undegraded":"2023-08-25T09:49:39.672281+0000","last_fullsized":"2023-08-25T09:49:39.672281+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:38:19.645378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533734+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:49:40.533734+0000","last_peered":"2023-08-25T09:49:40.533734+0000","last_clean":"2023-08-25T09:49:40.533734+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:49:40.533734+0000","last_undegraded":"2023-08-25T09:49:40.533734+0000","last_fullsized":"2023-08-25T09:49:40.533734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:27:54.228223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533904+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:49:40.533904+0000","last_peered":"2023-08-25T09:49:40.533904+0000","last_clean":"2023-08-25T09:49:40.533904+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:49:40.533904+0000","last_undegraded":"2023-08-25T09:49:40.533904+0000","last_fullsized":"2023-08-25T09:49:40.533904+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:06:19.107446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670060+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:49:39.670060+0000","last_peered":"2023-08-25T09:49:39.670060+0000","last_clean":"2023-08-25T09:49:39.670060+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:49:39.670060+0000","last_undegraded":"2023-08-25T09:49:39.670060+0000","last_fullsized":"2023-08-25T09:49:39.670060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:08:51.960006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608748+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:49:40.608748+0000","last_peered":"2023-08-25T09:49:40.608748+0000","last_clean":"2023-08-25T09:49:40.608748+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:49:40.608748+0000","last_undegraded":"2023-08-25T09:49:40.608748+0000","last_fullsized":"2023-08-25T09:49:40.608748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:35.777074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668458+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:49:39.668458+0000","last_peered":"2023-08-25T09:49:39.668458+0000","last_clean":"2023-08-25T09:49:39.668458+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:49:39.668458+0000","last_undegraded":"2023-08-25T09:49:39.668458+0000","last_fullsized":"2023-08-25T09:49:39.668458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:40:57.260235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669943+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:49:39.669943+0000","last_peered":"2023-08-25T09:49:39.669943+0000","last_clean":"2023-08-25T09:49:39.669943+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:49:39.669943+0000","last_undegraded":"2023-08-25T09:49:39.669943+0000","last_fullsized":"2023-08-25T09:49:39.669943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:42:36.500862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670365+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:49:39.670365+0000","last_peered":"2023-08-25T09:49:39.670365+0000","last_clean":"2023-08-25T09:49:39.670365+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:49:39.670365+0000","last_undegraded":"2023-08-25T09:49:39.670365+0000","last_fullsized":"2023-08-25T09:49:39.670365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:33:44.740556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314206+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:49:40.314206+0000","last_peered":"2023-08-25T09:49:40.314206+0000","last_clean":"2023-08-25T09:49:40.314206+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:49:40.314206+0000","last_undegraded":"2023-08-25T09:49:40.314206+0000","last_fullsized":"2023-08-25T09:49:40.314206+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:35.345143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670439+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:49:39.670439+0000","last_peered":"2023-08-25T09:49:39.670439+0000","last_clean":"2023-08-25T09:49:39.670439+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:49:39.670439+0000","last_undegraded":"2023-08-25T09:49:39.670439+0000","last_fullsized":"2023-08-25T09:49:39.670439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:38.142714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669800+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:49:39.669800+0000","last_peered":"2023-08-25T09:49:39.669800+0000","last_clean":"2023-08-25T09:49:39.669800+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:49:39.669800+0000","last_undegraded":"2023-08-25T09:49:39.669800+0000","last_fullsized":"2023-08-25T09:49:39.669800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:05:15.100928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668185+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:49:39.668185+0000","last_peered":"2023-08-25T09:49:39.668185+0000","last_clean":"2023-08-25T09:49:39.668185+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:49:39.668185+0000","last_undegraded":"2023-08-25T09:49:39.668185+0000","last_fullsized":"2023-08-25T09:49:39.668185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:52.824866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671435+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:49:39.671435+0000","last_peered":"2023-08-25T09:49:39.671435+0000","last_clean":"2023-08-25T09:49:39.671435+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:49:39.671435+0000","last_undegraded":"2023-08-25T09:49:39.671435+0000","last_fullsized":"2023-08-25T09:49:39.671435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:22:34.193671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668773+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:49:39.668773+0000","last_peered":"2023-08-25T09:49:39.668773+0000","last_clean":"2023-08-25T09:49:39.668773+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:49:39.668773+0000","last_undegraded":"2023-08-25T09:49:39.668773+0000","last_fullsized":"2023-08-25T09:49:39.668773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:38:09.260537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313744+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:49:40.313744+0000","last_peered":"2023-08-25T09:49:40.313744+0000","last_clean":"2023-08-25T09:49:40.313744+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:49:40.313744+0000","last_undegraded":"2023-08-25T09:49:40.313744+0000","last_fullsized":"2023-08-25T09:49:40.313744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:47:49.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314502+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:49:40.314502+0000","last_peered":"2023-08-25T09:49:40.314502+0000","last_clean":"2023-08-25T09:49:40.314502+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:49:40.314502+0000","last_undegraded":"2023-08-25T09:49:40.314502+0000","last_fullsized":"2023-08-25T09:49:40.314502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:52.308305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669862+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:49:39.669862+0000","last_peered":"2023-08-25T09:49:39.669862+0000","last_clean":"2023-08-25T09:49:39.669862+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:49:39.669862+0000","last_undegraded":"2023-08-25T09:49:39.669862+0000","last_fullsized":"2023-08-25T09:49:39.669862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:33:20.778440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669203+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:49:39.669203+0000","last_peered":"2023-08-25T09:49:39.669203+0000","last_clean":"2023-08-25T09:49:39.669203+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:49:39.669203+0000","last_undegraded":"2023-08-25T09:49:39.669203+0000","last_fullsized":"2023-08-25T09:49:39.669203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:14.357542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669995+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:49:39.669995+0000","last_peered":"2023-08-25T09:49:39.669995+0000","last_clean":"2023-08-25T09:49:39.669995+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:49:39.669995+0000","last_undegraded":"2023-08-25T09:49:39.669995+0000","last_fullsized":"2023-08-25T09:49:39.669995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:06.555517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668019+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:49:39.668019+0000","last_peered":"2023-08-25T09:49:39.668019+0000","last_clean":"2023-08-25T09:49:39.668019+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:49:39.668019+0000","last_undegraded":"2023-08-25T09:49:39.668019+0000","last_fullsized":"2023-08-25T09:49:39.668019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:35:29.581577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671260+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:49:39.671260+0000","last_peered":"2023-08-25T09:49:39.671260+0000","last_clean":"2023-08-25T09:49:39.671260+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:49:39.671260+0000","last_undegraded":"2023-08-25T09:49:39.671260+0000","last_fullsized":"2023-08-25T09:49:39.671260+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:43.256956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669816+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:49:39.669816+0000","last_peered":"2023-08-25T09:49:39.669816+0000","last_clean":"2023-08-25T09:49:39.669816+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:49:39.669816+0000","last_undegraded":"2023-08-25T09:49:39.669816+0000","last_fullsized":"2023-08-25T09:49:39.669816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:05:53.483156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669716+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:49:39.669716+0000","last_peered":"2023-08-25T09:49:39.669716+0000","last_clean":"2023-08-25T09:49:39.669716+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:49:39.669716+0000","last_undegraded":"2023-08-25T09:49:39.669716+0000","last_fullsized":"2023-08-25T09:49:39.669716+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:17.656387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609913+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:49:40.609913+0000","last_peered":"2023-08-25T09:49:40.609913+0000","last_clean":"2023-08-25T09:49:40.609913+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:49:40.609913+0000","last_undegraded":"2023-08-25T09:49:40.609913+0000","last_fullsized":"2023-08-25T09:49:40.609913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:04:46.102346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669727+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:49:39.669727+0000","last_peered":"2023-08-25T09:49:39.669727+0000","last_clean":"2023-08-25T09:49:39.669727+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:49:39.669727+0000","last_undegraded":"2023-08-25T09:49:39.669727+0000","last_fullsized":"2023-08-25T09:49:39.669727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:03:17.962039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668925+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:49:39.668925+0000","last_peered":"2023-08-25T09:49:39.668925+0000","last_clean":"2023-08-25T09:49:39.668925+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:49:39.668925+0000","last_undegraded":"2023-08-25T09:49:39.668925+0000","last_fullsized":"2023-08-25T09:49:39.668925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:15:02.299452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314453+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:49:40.314453+0000","last_peered":"2023-08-25T09:49:40.314453+0000","last_clean":"2023-08-25T09:49:40.314453+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:49:40.314453+0000","last_undegraded":"2023-08-25T09:49:40.314453+0000","last_fullsized":"2023-08-25T09:49:40.314453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:31.990094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":6045,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509452,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":130840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]}]},{"osd":7,"up_from":15,"seq":64424509452,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":405504,"data_stored":721238,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.423}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509453,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3129999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":126995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":126321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":389120,"data_stored":717893,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:07.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:07.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.989+0000 7f0d87e34640 1 Processor -- start 2023-08-25T09:50:07.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.989+0000 7f0d87e34640 1 -- start start 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d87e34640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0d800c3660 0x7f0d800c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d87e34640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0d801509d0 0x7f0d80150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d87e34640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d800c1dc0 con 0x7f0d800c3a40 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d87e34640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d80157a80 con 0x7f0d801509d0 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d87e34640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d80151320 con 0x7f0d800c3660 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d853cf640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0d800c3a40 0x7f0d8006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46476/0 (socket says 172.21.15.71:46476) 2023-08-25T09:50:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d863d1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0d801509d0 0x7f0d80150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.991+0000 7f0d853cf640 1 -- 172.21.15.71:0/530191679 learned_addr learned my addr 172.21.15.71:0/530191679 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.992+0000 7f0d85bd0640 1 --2- 172.21.15.71:0/530191679 >> v2:172.21.15.164:3300/0 conn(0x7f0d800c3660 0x7f0d800c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.992+0000 7f0d863d1640 1 -- 172.21.15.71:0/530191679 >> v2:172.21.15.164:3300/0 conn(0x7f0d800c3660 msgr2=0x7f0d800c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.992+0000 7f0d863d1640 1 --2- 172.21.15.71:0/530191679 >> v2:172.21.15.164:3300/0 conn(0x7f0d800c3660 0x7f0d800c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.992+0000 7f0d863d1640 1 -- 172.21.15.71:0/530191679 >> v1:172.21.15.71:6789/0 conn(0x7f0d800c3a40 legacy=0x7f0d8006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.992+0000 7f0d863d1640 1 -- 172.21.15.71:0/530191679 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80156e00 con 0x7f0d801509d0 2023-08-25T09:50:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.992+0000 7f0d84bce640 1 -- 172.21.15.71:0/530191679 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166361594 0 0) 0x7f0d800c1dc0 con 0x7f0d800c3a40 2023-08-25T09:50:07.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.993+0000 7f0d863d1640 1 --2- 172.21.15.71:0/530191679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d801509d0 0x7f0d80150de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0d80066440 tx=0x7f0d7802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a84851f7a6df675 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:07.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.993+0000 7f0d84bce640 1 -- 172.21.15.71:0/530191679 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7802cd10 con 0x7f0d801509d0 2023-08-25T09:50:07.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.993+0000 7f0d84bce640 1 -- 172.21.15.71:0/530191679 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7802ce70 con 0x7f0d801509d0 2023-08-25T09:50:07.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.993+0000 7f0d84bce640 1 -- 172.21.15.71:0/530191679 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78034640 con 0x7f0d801509d0 2023-08-25T09:50:07.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.993+0000 7f0d87e34640 1 -- 172.21.15.71:0/530191679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d801509d0 msgr2=0x7f0d80150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:07.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.993+0000 7f0d87e34640 1 --2- 172.21.15.71:0/530191679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d801509d0 0x7f0d80150de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0d80066440 tx=0x7f0d7802bf60 comp rx=0 tx=0).stop 2023-08-25T09:50:07.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.994+0000 7f0d87e34640 1 -- 172.21.15.71:0/530191679 shutdown_connections 2023-08-25T09:50:07.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.994+0000 7f0d87e34640 1 --2- 172.21.15.71:0/530191679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d801509d0 0x7f0d80150de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.994+0000 7f0d87e34640 1 --2- 172.21.15.71:0/530191679 >> v2:172.21.15.164:3300/0 conn(0x7f0d800c3660 0x7f0d800c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:07.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.994+0000 7f0d87e34640 1 -- 172.21.15.71:0/530191679 >> 172.21.15.71:0/530191679 conn(0x7f0d800654f0 msgr2=0x7f0d80065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:07.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.994+0000 7f0d87e34640 1 -- 172.21.15.71:0/530191679 shutdown_connections 2023-08-25T09:50:07.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.994+0000 7f0d87e34640 1 -- 172.21.15.71:0/530191679 wait complete. 2023-08-25T09:50:07.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.995+0000 7f0d87e34640 1 Processor -- start 2023-08-25T09:50:07.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.995+0000 7f0d87e34640 1 -- start start 2023-08-25T09:50:08.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.996+0000 7f0d87e34640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 0x7f0d8006a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.996+0000 7f0d87e34640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d800c3a40 0x7f0d80065960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.996+0000 7f0d87e34640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d801509d0 0x7f0d8006ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.996+0000 7f0d87e34640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d80159820 con 0x7f0d800c3660 2023-08-25T09:50:08.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.996+0000 7f0d87e34640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d80157fe0 con 0x7f0d801509d0 2023-08-25T09:50:08.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.996+0000 7f0d87e34640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8015ada0 con 0x7f0d800c3a40 2023-08-25T09:50:08.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 0x7f0d8006a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 0x7f0d8006a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60334/0 (socket says 172.21.15.71:60334) 2023-08-25T09:50:08.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 -- 172.21.15.71:0/275395566 learned_addr learned my addr 172.21.15.71:0/275395566 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:08.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 -- 172.21.15.71:0/275395566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d800c3a40 msgr2=0x7f0d80065960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d800c3a40 0x7f0d80065960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 -- 172.21.15.71:0/275395566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d801509d0 msgr2=0x7f0d8006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d801509d0 0x7f0d8006ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 -- 172.21.15.71:0/275395566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78009a70 con 0x7f0d800c3660 2023-08-25T09:50:08.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d85bd0640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 0x7f0d8006a7e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00d830 tx=0x7f0d7c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:08.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00b6c0 con 0x7f0d800c3660 2023-08-25T09:50:08.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c00b820 con 0x7f0d800c3660 2023-08-25T09:50:08.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c00bbb0 con 0x7f0d800c3660 2023-08-25T09:50:08.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.997+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d800c8870 con 0x7f0d800c3660 2023-08-25T09:50:08.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.998+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d80150250 con 0x7f0d800c3660 2023-08-25T09:50:08.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.999+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d800c29c0 con 0x7f0d800c3660 2023-08-25T09:50:08.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:07.999+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0d7c01e460 con 0x7f0d800c3660 2023-08-25T09:50:08.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.001+0000 7f0d767fc640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d6406a050 0x7f0d6406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.001+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7f0d7c088fe0 con 0x7f0d800c3660 2023-08-25T09:50:08.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.001+0000 7f0d853cf640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d6406a050 0x7f0d6406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.002+0000 7f0d853cf640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d6406a050 0x7f0d6406c510 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0d70007fd0 tx=0x7f0d70007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:08.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.005+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0d7c0543f0 con 0x7f0d800c3660 2023-08-25T09:50:08.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.179+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d801573c0 con 0x7f0d6406a050 2023-08-25T09:50:08.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.206+0000 7f0d767fc640 1 -- 172.21.15.71:0/275395566 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367369 (secure 0 0 0) 0x7f0d700070d0 con 0x7f0d6406a050 2023-08-25T09:50:08.215 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:08.217 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:08.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d6406a050 msgr2=0x7f0d6406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d6406a050 0x7f0d6406c510 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0d70007fd0 tx=0x7f0d70007920 comp rx=0 tx=0).stop 2023-08-25T09:50:08.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 msgr2=0x7f0d8006a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 0x7f0d8006a7e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00d830 tx=0x7f0d7c00dcf0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 shutdown_connections 2023-08-25T09:50:08.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d801509d0 0x7f0d8006ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0d6406a050 0x7f0d6406c510 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d800c3a40 0x7f0d80065960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 --2- 172.21.15.71:0/275395566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0d800c3660 0x7f0d8006a7e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 >> 172.21.15.71:0/275395566 conn(0x7f0d800654f0 msgr2=0x7f0d800c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.213+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 shutdown_connections 2023-08-25T09:50:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.214+0000 7f0d87e34640 1 -- 172.21.15.71:0/275395566 wait complete. 2023-08-25T09:50:08.232 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2023-08-25T09:50:06.645565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596413,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227088,"kb_used_data":2640,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747632,"statfs":{"total":773094113280,"available":772861575168,"internally_reserved":0,"allocated":2703360,"data_stored":2202674,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3187,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004937"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532259+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:49:40.532259+0000","last_peered":"2023-08-25T09:49:40.532259+0000","last_clean":"2023-08-25T09:49:40.532259+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:49:40.532259+0000","last_undegraded":"2023-08-25T09:49:40.532259+0000","last_fullsized":"2023-08-25T09:49:40.532259+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:08:44.154380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"28'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670137+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:49:39.670137+0000","last_peered":"2023-08-25T09:49:39.670137+0000","last_clean":"2023-08-25T09:49:39.670137+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:49:39.670137+0000","last_undegraded":"2023-08-25T09:49:39.670137+0000","last_fullsized":"2023-08-25T09:49:39.670137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:19:21.160750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672075+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:49:39.672075+0000","last_peered":"2023-08-25T09:49:39.672075+0000","last_clean":"2023-08-25T09:49:39.672075+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:49:39.672075+0000","last_undegraded":"2023-08-25T09:49:39.672075+0000","last_fullsized":"2023-08-25T09:49:39.672075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:39:44.896839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.667983+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:49:39.667983+0000","last_peered":"2023-08-25T09:49:39.667983+0000","last_clean":"2023-08-25T09:49:39.667983+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:49:39.667983+0000","last_undegraded":"2023-08-25T09:49:39.667983+0000","last_fullsized":"2023-08-25T09:49:39.667983+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:52:21.149049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610317+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:49:40.610317+0000","last_peered":"2023-08-25T09:49:40.610317+0000","last_clean":"2023-08-25T09:49:40.610317+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:49:40.610317+0000","last_undegraded":"2023-08-25T09:49:40.610317+0000","last_fullsized":"2023-08-25T09:49:40.610317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.067887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609551+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:49:40.609551+0000","last_peered":"2023-08-25T09:49:40.609551+0000","last_clean":"2023-08-25T09:49:40.609551+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:49:40.609551+0000","last_undegraded":"2023-08-25T09:49:40.609551+0000","last_fullsized":"2023-08-25T09:49:40.609551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:31:30.676338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669356+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:49:39.669356+0000","last_peered":"2023-08-25T09:49:39.669356+0000","last_clean":"2023-08-25T09:49:39.669356+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:49:39.669356+0000","last_undegraded":"2023-08-25T09:49:39.669356+0000","last_fullsized":"2023-08-25T09:49:39.669356+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:52:41.076094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672443+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:49:39.672443+0000","last_peered":"2023-08-25T09:49:39.672443+0000","last_clean":"2023-08-25T09:49:39.672443+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:49:39.672443+0000","last_undegraded":"2023-08-25T09:49:39.672443+0000","last_fullsized":"2023-08-25T09:49:39.672443+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:41:06.425160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670837+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:49:39.670837+0000","last_peered":"2023-08-25T09:49:39.670837+0000","last_clean":"2023-08-25T09:49:39.670837+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:49:39.670837+0000","last_undegraded":"2023-08-25T09:49:39.670837+0000","last_fullsized":"2023-08-25T09:49:39.670837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:43.561619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669087+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:49:39.669087+0000","last_peered":"2023-08-25T09:49:39.669087+0000","last_clean":"2023-08-25T09:49:39.669087+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:49:39.669087+0000","last_undegraded":"2023-08-25T09:49:39.669087+0000","last_fullsized":"2023-08-25T09:49:39.669087+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:29:31.719174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668364+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:49:39.668364+0000","last_peered":"2023-08-25T09:49:39.668364+0000","last_clean":"2023-08-25T09:49:39.668364+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:49:39.668364+0000","last_undegraded":"2023-08-25T09:49:39.668364+0000","last_fullsized":"2023-08-25T09:49:39.668364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:54:21.024745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669503+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:49:39.669503+0000","last_peered":"2023-08-25T09:49:39.669503+0000","last_clean":"2023-08-25T09:49:39.669503+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:49:39.669503+0000","last_undegraded":"2023-08-25T09:49:39.669503+0000","last_fullsized":"2023-08-25T09:49:39.669503+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:38.132579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533392+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:49:40.533392+0000","last_peered":"2023-08-25T09:49:40.533392+0000","last_clean":"2023-08-25T09:49:40.533392+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:49:40.533392+0000","last_undegraded":"2023-08-25T09:49:40.533392+0000","last_fullsized":"2023-08-25T09:49:40.533392+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:44:04.188030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"29'5","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.538756+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:50:05.538756+0000","last_peered":"2023-08-25T09:50:05.538756+0000","last_clean":"2023-08-25T09:50:05.538756+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:50:05.538756+0000","last_undegraded":"2023-08-25T09:50:05.538756+0000","last_fullsized":"2023-08-25T09:50:05.538756+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:36:03.677003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314444+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:49:40.314444+0000","last_peered":"2023-08-25T09:49:40.314444+0000","last_clean":"2023-08-25T09:49:40.314444+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:49:40.314444+0000","last_undegraded":"2023-08-25T09:49:40.314444+0000","last_fullsized":"2023-08-25T09:49:40.314444+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:24:33.940678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670152+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:49:39.670152+0000","last_peered":"2023-08-25T09:49:39.670152+0000","last_clean":"2023-08-25T09:49:39.670152+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:49:39.670152+0000","last_undegraded":"2023-08-25T09:49:39.670152+0000","last_fullsized":"2023-08-25T09:49:39.670152+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:04.589805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671210+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:49:39.671210+0000","last_peered":"2023-08-25T09:49:39.671210+0000","last_clean":"2023-08-25T09:49:39.671210+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:49:39.671210+0000","last_undegraded":"2023-08-25T09:49:39.671210+0000","last_fullsized":"2023-08-25T09:49:39.671210+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:21.872702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610243+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:49:40.610243+0000","last_peered":"2023-08-25T09:49:40.610243+0000","last_clean":"2023-08-25T09:49:40.610243+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:49:40.610243+0000","last_undegraded":"2023-08-25T09:49:40.610243+0000","last_fullsized":"2023-08-25T09:49:40.610243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:40:11.918086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610311+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:49:40.610311+0000","last_peered":"2023-08-25T09:49:40.610311+0000","last_clean":"2023-08-25T09:49:40.610311+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:49:40.610311+0000","last_undegraded":"2023-08-25T09:49:40.610311+0000","last_fullsized":"2023-08-25T09:49:40.610311+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:24:36.237719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532845+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:49:40.532845+0000","last_peered":"2023-08-25T09:49:40.532845+0000","last_clean":"2023-08-25T09:49:40.532845+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:49:40.532845+0000","last_undegraded":"2023-08-25T09:49:40.532845+0000","last_fullsized":"2023-08-25T09:49:40.532845+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:17.775607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669479+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:49:39.669479+0000","last_peered":"2023-08-25T09:49:39.669479+0000","last_clean":"2023-08-25T09:49:39.669479+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:49:39.669479+0000","last_undegraded":"2023-08-25T09:49:39.669479+0000","last_fullsized":"2023-08-25T09:49:39.669479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:24:10.790283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669214+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:49:39.669214+0000","last_peered":"2023-08-25T09:49:39.669214+0000","last_clean":"2023-08-25T09:49:39.669214+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:49:39.669214+0000","last_undegraded":"2023-08-25T09:49:39.669214+0000","last_fullsized":"2023-08-25T09:49:39.669214+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:45:44.040670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609148+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:49:40.609148+0000","last_peered":"2023-08-25T09:49:40.609148+0000","last_clean":"2023-08-25T09:49:40.609148+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:49:40.609148+0000","last_undegraded":"2023-08-25T09:49:40.609148+0000","last_fullsized":"2023-08-25T09:49:40.609148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:30:29.949537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610535+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:49:40.610535+0000","last_peered":"2023-08-25T09:49:40.610535+0000","last_clean":"2023-08-25T09:49:40.610535+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:49:40.610535+0000","last_undegraded":"2023-08-25T09:49:40.610535+0000","last_fullsized":"2023-08-25T09:49:40.610535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:33:47.145180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608671+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.608671+0000","last_peered":"2023-08-25T09:49:40.608671+0000","last_clean":"2023-08-25T09:49:40.608671+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:49:40.608671+0000","last_undegraded":"2023-08-25T09:49:40.608671+0000","last_fullsized":"2023-08-25T09:49:40.608671+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:49:14.644166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"29'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535841+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:50:05.535841+0000","last_peered":"2023-08-25T09:50:05.535841+0000","last_clean":"2023-08-25T09:50:05.535841+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:50:05.535841+0000","last_undegraded":"2023-08-25T09:50:05.535841+0000","last_fullsized":"2023-08-25T09:50:05.535841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:33:42.072752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609238+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:49:40.609238+0000","last_peered":"2023-08-25T09:49:40.609238+0000","last_clean":"2023-08-25T09:49:40.609238+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:49:40.609238+0000","last_undegraded":"2023-08-25T09:49:40.609238+0000","last_fullsized":"2023-08-25T09:49:40.609238+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:01:49.724298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609229+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:49:40.609229+0000","last_peered":"2023-08-25T09:49:40.609229+0000","last_clean":"2023-08-25T09:49:40.609229+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:49:40.609229+0000","last_undegraded":"2023-08-25T09:49:40.609229+0000","last_fullsized":"2023-08-25T09:49:40.609229+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:35:14.991310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"29'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535643+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:50:05.535643+0000","last_peered":"2023-08-25T09:50:05.535643+0000","last_clean":"2023-08-25T09:50:05.535643+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:50:05.535643+0000","last_undegraded":"2023-08-25T09:50:05.535643+0000","last_fullsized":"2023-08-25T09:50:05.535643+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:36:04.594929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669714+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:49:39.669714+0000","last_peered":"2023-08-25T09:49:39.669714+0000","last_clean":"2023-08-25T09:49:39.669714+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:49:39.669714+0000","last_undegraded":"2023-08-25T09:49:39.669714+0000","last_fullsized":"2023-08-25T09:49:39.669714+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:13:29.132994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671434+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:49:39.671434+0000","last_peered":"2023-08-25T09:49:39.671434+0000","last_clean":"2023-08-25T09:49:39.671434+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:49:39.671434+0000","last_undegraded":"2023-08-25T09:49:39.671434+0000","last_fullsized":"2023-08-25T09:49:39.671434+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:08:04.314595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533055+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:49:40.533055+0000","last_peered":"2023-08-25T09:49:40.533055+0000","last_clean":"2023-08-25T09:49:40.533055+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:49:40.533055+0000","last_undegraded":"2023-08-25T09:49:40.533055+0000","last_fullsized":"2023-08-25T09:49:40.533055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:48:29.319338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609374+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:49:40.609374+0000","last_peered":"2023-08-25T09:49:40.609374+0000","last_clean":"2023-08-25T09:49:40.609374+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:49:40.609374+0000","last_undegraded":"2023-08-25T09:49:40.609374+0000","last_fullsized":"2023-08-25T09:49:40.609374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:11:38.846168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"29'12","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.537617+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:50:05.537617+0000","last_peered":"2023-08-25T09:50:05.537617+0000","last_clean":"2023-08-25T09:50:05.537617+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:50:05.537617+0000","last_undegraded":"2023-08-25T09:50:05.537617+0000","last_fullsized":"2023-08-25T09:50:05.537617+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:19:49.118130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3841,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669680+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:49:39.669680+0000","last_peered":"2023-08-25T09:49:39.669680+0000","last_clean":"2023-08-25T09:49:39.669680+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:49:39.669680+0000","last_undegraded":"2023-08-25T09:49:39.669680+0000","last_fullsized":"2023-08-25T09:49:39.669680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:15:08.913052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532469+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:49:40.532469+0000","last_peered":"2023-08-25T09:49:40.532469+0000","last_clean":"2023-08-25T09:49:40.532469+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:49:40.532469+0000","last_undegraded":"2023-08-25T09:49:40.532469+0000","last_fullsized":"2023-08-25T09:49:40.532469+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:54:59.656364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669355+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:49:39.669355+0000","last_peered":"2023-08-25T09:49:39.669355+0000","last_clean":"2023-08-25T09:49:39.669355+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:49:39.669355+0000","last_undegraded":"2023-08-25T09:49:39.669355+0000","last_fullsized":"2023-08-25T09:49:39.669355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:48.989731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610694+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:49:40.610694+0000","last_peered":"2023-08-25T09:49:40.610694+0000","last_clean":"2023-08-25T09:49:40.610694+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:49:40.610694+0000","last_undegraded":"2023-08-25T09:49:40.610694+0000","last_fullsized":"2023-08-25T09:49:40.610694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:46.063058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670003+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:49:39.670003+0000","last_peered":"2023-08-25T09:49:39.670003+0000","last_clean":"2023-08-25T09:49:39.670003+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:49:39.670003+0000","last_undegraded":"2023-08-25T09:49:39.670003+0000","last_fullsized":"2023-08-25T09:49:39.670003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:13:11.987428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608483+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:49:40.608483+0000","last_peered":"2023-08-25T09:49:40.608483+0000","last_clean":"2023-08-25T09:49:40.608483+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:49:40.608483+0000","last_undegraded":"2023-08-25T09:49:40.608483+0000","last_fullsized":"2023-08-25T09:49:40.608483+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:28:33.537079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608779+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:49:40.608779+0000","last_peered":"2023-08-25T09:49:40.608779+0000","last_clean":"2023-08-25T09:49:40.608779+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:49:40.608779+0000","last_undegraded":"2023-08-25T09:49:40.608779+0000","last_fullsized":"2023-08-25T09:49:40.608779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:01:10.400897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.534056+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:49:40.534056+0000","last_peered":"2023-08-25T09:49:40.534056+0000","last_clean":"2023-08-25T09:49:40.534056+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:49:40.534056+0000","last_undegraded":"2023-08-25T09:49:40.534056+0000","last_fullsized":"2023-08-25T09:49:40.534056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:30.208912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.730451+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:50:05.730451+0000","last_peered":"2023-08-25T09:50:05.730451+0000","last_clean":"2023-08-25T09:50:05.730451+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:50:05.730451+0000","last_undegraded":"2023-08-25T09:50:05.730451+0000","last_fullsized":"2023-08-25T09:50:05.730451+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:46:45.992240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668225+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:49:39.668225+0000","last_peered":"2023-08-25T09:49:39.668225+0000","last_clean":"2023-08-25T09:49:39.668225+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:49:39.668225+0000","last_undegraded":"2023-08-25T09:49:39.668225+0000","last_fullsized":"2023-08-25T09:49:39.668225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:00:13.974928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669196+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:49:39.669196+0000","last_peered":"2023-08-25T09:49:39.669196+0000","last_clean":"2023-08-25T09:49:39.669196+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:49:39.669196+0000","last_undegraded":"2023-08-25T09:49:39.669196+0000","last_fullsized":"2023-08-25T09:49:39.669196+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:28:03.992015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671806+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:49:39.671806+0000","last_peered":"2023-08-25T09:49:39.671806+0000","last_clean":"2023-08-25T09:49:39.671806+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:49:39.671806+0000","last_undegraded":"2023-08-25T09:49:39.671806+0000","last_fullsized":"2023-08-25T09:49:39.671806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:14:52.844903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533580+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:49:40.533580+0000","last_peered":"2023-08-25T09:49:40.533580+0000","last_clean":"2023-08-25T09:49:40.533580+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:49:40.533580+0000","last_undegraded":"2023-08-25T09:49:40.533580+0000","last_fullsized":"2023-08-25T09:49:40.533580+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:00.063674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609540+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:49:40.609540+0000","last_peered":"2023-08-25T09:49:40.609540+0000","last_clean":"2023-08-25T09:49:40.609540+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:49:40.609540+0000","last_undegraded":"2023-08-25T09:49:40.609540+0000","last_fullsized":"2023-08-25T09:49:40.609540+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:59:18.199533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532919+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:49:40.532919+0000","last_peered":"2023-08-25T09:49:40.532919+0000","last_clean":"2023-08-25T09:49:40.532919+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:49:40.532919+0000","last_undegraded":"2023-08-25T09:49:40.532919+0000","last_fullsized":"2023-08-25T09:49:40.532919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:00:24.110176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533867+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:49:40.533867+0000","last_peered":"2023-08-25T09:49:40.533867+0000","last_clean":"2023-08-25T09:49:40.533867+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:49:40.533867+0000","last_undegraded":"2023-08-25T09:49:40.533867+0000","last_fullsized":"2023-08-25T09:49:40.533867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:25:26.896240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672608+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:49:39.672608+0000","last_peered":"2023-08-25T09:49:39.672608+0000","last_clean":"2023-08-25T09:49:39.672608+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:49:39.672608+0000","last_undegraded":"2023-08-25T09:49:39.672608+0000","last_fullsized":"2023-08-25T09:49:39.672608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:23.747043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668857+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:49:39.668857+0000","last_peered":"2023-08-25T09:49:39.668857+0000","last_clean":"2023-08-25T09:49:39.668857+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:49:39.668857+0000","last_undegraded":"2023-08-25T09:49:39.668857+0000","last_fullsized":"2023-08-25T09:49:39.668857+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:12:48.827644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533435+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:49:40.533435+0000","last_peered":"2023-08-25T09:49:40.533435+0000","last_clean":"2023-08-25T09:49:40.533435+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:49:40.533435+0000","last_undegraded":"2023-08-25T09:49:40.533435+0000","last_fullsized":"2023-08-25T09:49:40.533435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:07:36.521502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"29'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535468+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:50:05.535468+0000","last_peered":"2023-08-25T09:50:05.535468+0000","last_clean":"2023-08-25T09:50:05.535468+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:50:05.535468+0000","last_undegraded":"2023-08-25T09:50:05.535468+0000","last_fullsized":"2023-08-25T09:50:05.535468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:26:07.776398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668924+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:49:39.668924+0000","last_peered":"2023-08-25T09:49:39.668924+0000","last_clean":"2023-08-25T09:49:39.668924+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:49:39.668924+0000","last_undegraded":"2023-08-25T09:49:39.668924+0000","last_fullsized":"2023-08-25T09:49:39.668924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:22:13.379657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669047+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:49:39.669047+0000","last_peered":"2023-08-25T09:49:39.669047+0000","last_clean":"2023-08-25T09:49:39.669047+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:49:39.669047+0000","last_undegraded":"2023-08-25T09:49:39.669047+0000","last_fullsized":"2023-08-25T09:49:39.669047+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:36:02.088533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533994+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:49:40.533994+0000","last_peered":"2023-08-25T09:49:40.533994+0000","last_clean":"2023-08-25T09:49:40.533994+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:49:40.533994+0000","last_undegraded":"2023-08-25T09:49:40.533994+0000","last_fullsized":"2023-08-25T09:49:40.533994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:34:02.269298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532621+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:49:40.532621+0000","last_peered":"2023-08-25T09:49:40.532621+0000","last_clean":"2023-08-25T09:49:40.532621+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:49:40.532621+0000","last_undegraded":"2023-08-25T09:49:40.532621+0000","last_fullsized":"2023-08-25T09:49:40.532621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:52:34.012911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313601+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:49:40.313601+0000","last_peered":"2023-08-25T09:49:40.313601+0000","last_clean":"2023-08-25T09:49:40.313601+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:49:40.313601+0000","last_undegraded":"2023-08-25T09:49:40.313601+0000","last_fullsized":"2023-08-25T09:49:40.313601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:28.801938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314730+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:49:40.314730+0000","last_peered":"2023-08-25T09:49:40.314730+0000","last_clean":"2023-08-25T09:49:40.314730+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:49:40.314730+0000","last_undegraded":"2023-08-25T09:49:40.314730+0000","last_fullsized":"2023-08-25T09:49:40.314730+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:17:34.425122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669544+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:49:39.669544+0000","last_peered":"2023-08-25T09:49:39.669544+0000","last_clean":"2023-08-25T09:49:39.669544+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:49:39.669544+0000","last_undegraded":"2023-08-25T09:49:39.669544+0000","last_fullsized":"2023-08-25T09:49:39.669544+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:44.776136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"29'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668255+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:49:39.668255+0000","last_peered":"2023-08-25T09:49:39.668255+0000","last_clean":"2023-08-25T09:49:39.668255+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:49:39.668255+0000","last_undegraded":"2023-08-25T09:49:39.668255+0000","last_fullsized":"2023-08-25T09:49:39.668255+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:05:08.959803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670236+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:49:39.670236+0000","last_peered":"2023-08-25T09:49:39.670236+0000","last_clean":"2023-08-25T09:49:39.670236+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:49:39.670236+0000","last_undegraded":"2023-08-25T09:49:39.670236+0000","last_fullsized":"2023-08-25T09:49:39.670236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:04.664807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668818+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:49:39.668818+0000","last_peered":"2023-08-25T09:49:39.668818+0000","last_clean":"2023-08-25T09:49:39.668818+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:49:39.668818+0000","last_undegraded":"2023-08-25T09:49:39.668818+0000","last_fullsized":"2023-08-25T09:49:39.668818+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:45:13.832806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610040+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:49:40.610040+0000","last_peered":"2023-08-25T09:49:40.610040+0000","last_clean":"2023-08-25T09:49:40.610040+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:49:40.610040+0000","last_undegraded":"2023-08-25T09:49:40.610040+0000","last_fullsized":"2023-08-25T09:49:40.610040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:08.515395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609484+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:49:40.609484+0000","last_peered":"2023-08-25T09:49:40.609484+0000","last_clean":"2023-08-25T09:49:40.609484+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:49:40.609484+0000","last_undegraded":"2023-08-25T09:49:40.609484+0000","last_fullsized":"2023-08-25T09:49:40.609484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:02.892745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671904+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:49:39.671904+0000","last_peered":"2023-08-25T09:49:39.671904+0000","last_clean":"2023-08-25T09:49:39.671904+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:49:39.671904+0000","last_undegraded":"2023-08-25T09:49:39.671904+0000","last_fullsized":"2023-08-25T09:49:39.671904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:12:08.551735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":183,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670426+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:49:39.670426+0000","last_peered":"2023-08-25T09:49:39.670426+0000","last_clean":"2023-08-25T09:49:39.670426+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:49:39.670426+0000","last_undegraded":"2023-08-25T09:49:39.670426+0000","last_fullsized":"2023-08-25T09:49:39.670426+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:14:32.439794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670825+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:49:39.670825+0000","last_peered":"2023-08-25T09:49:39.670825+0000","last_clean":"2023-08-25T09:49:39.670825+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:49:39.670825+0000","last_undegraded":"2023-08-25T09:49:39.670825+0000","last_fullsized":"2023-08-25T09:49:39.670825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:14:42.235888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533862+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:49:40.533862+0000","last_peered":"2023-08-25T09:49:40.533862+0000","last_clean":"2023-08-25T09:49:40.533862+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:49:40.533862+0000","last_undegraded":"2023-08-25T09:49:40.533862+0000","last_fullsized":"2023-08-25T09:49:40.533862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:47:24.625104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"29'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.730045+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:50:05.730045+0000","last_peered":"2023-08-25T09:50:05.730045+0000","last_clean":"2023-08-25T09:50:05.730045+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:50:05.730045+0000","last_undegraded":"2023-08-25T09:50:05.730045+0000","last_fullsized":"2023-08-25T09:50:05.730045+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:34:01.834958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670693+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:49:39.670693+0000","last_peered":"2023-08-25T09:49:39.670693+0000","last_clean":"2023-08-25T09:49:39.670693+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:49:39.670693+0000","last_undegraded":"2023-08-25T09:49:39.670693+0000","last_fullsized":"2023-08-25T09:49:39.670693+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:53:52.223404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670888+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:49:39.670888+0000","last_peered":"2023-08-25T09:49:39.670888+0000","last_clean":"2023-08-25T09:49:39.670888+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:49:39.670888+0000","last_undegraded":"2023-08-25T09:49:39.670888+0000","last_fullsized":"2023-08-25T09:49:39.670888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:52.646017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"29'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.535826+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:50:05.535826+0000","last_peered":"2023-08-25T09:50:05.535826+0000","last_clean":"2023-08-25T09:50:05.535826+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:50:05.535826+0000","last_undegraded":"2023-08-25T09:50:05.535826+0000","last_fullsized":"2023-08-25T09:50:05.535826+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:07:22.260320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668612+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:49:39.668612+0000","last_peered":"2023-08-25T09:49:39.668612+0000","last_clean":"2023-08-25T09:49:39.668612+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:49:39.668612+0000","last_undegraded":"2023-08-25T09:49:39.668612+0000","last_fullsized":"2023-08-25T09:49:39.668612+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:48:39.386372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670019+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:49:39.670019+0000","last_peered":"2023-08-25T09:49:39.670019+0000","last_clean":"2023-08-25T09:49:39.670019+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:49:39.670019+0000","last_undegraded":"2023-08-25T09:49:39.670019+0000","last_fullsized":"2023-08-25T09:49:39.670019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:45:29.348922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610153+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:49:40.610153+0000","last_peered":"2023-08-25T09:49:40.610153+0000","last_clean":"2023-08-25T09:49:40.610153+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:49:40.610153+0000","last_undegraded":"2023-08-25T09:49:40.610153+0000","last_fullsized":"2023-08-25T09:49:40.610153+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:50:39.530928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.532880+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:49:40.532880+0000","last_peered":"2023-08-25T09:49:40.532880+0000","last_clean":"2023-08-25T09:49:40.532880+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:49:40.532880+0000","last_undegraded":"2023-08-25T09:49:40.532880+0000","last_fullsized":"2023-08-25T09:49:40.532880+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:08:17.159830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670546+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:49:39.670546+0000","last_peered":"2023-08-25T09:49:39.670546+0000","last_clean":"2023-08-25T09:49:39.670546+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:49:39.670546+0000","last_undegraded":"2023-08-25T09:49:39.670546+0000","last_fullsized":"2023-08-25T09:49:39.670546+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:10:36.871310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.729657+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:50:05.729657+0000","last_peered":"2023-08-25T09:50:05.729657+0000","last_clean":"2023-08-25T09:50:05.729657+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:50:05.729657+0000","last_undegraded":"2023-08-25T09:50:05.729657+0000","last_fullsized":"2023-08-25T09:50:05.729657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:13:22.282204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314057+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:49:40.314057+0000","last_peered":"2023-08-25T09:49:40.314057+0000","last_clean":"2023-08-25T09:49:40.314057+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:49:40.314057+0000","last_undegraded":"2023-08-25T09:49:40.314057+0000","last_fullsized":"2023-08-25T09:49:40.314057+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:32:23.040483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668795+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:49:39.668795+0000","last_peered":"2023-08-25T09:49:39.668795+0000","last_clean":"2023-08-25T09:49:39.668795+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:49:39.668795+0000","last_undegraded":"2023-08-25T09:49:39.668795+0000","last_fullsized":"2023-08-25T09:49:39.668795+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:21:47.452907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668565+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:49:39.668565+0000","last_peered":"2023-08-25T09:49:39.668565+0000","last_clean":"2023-08-25T09:49:39.668565+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:49:39.668565+0000","last_undegraded":"2023-08-25T09:49:39.668565+0000","last_fullsized":"2023-08-25T09:49:39.668565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:49:12.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313625+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:49:40.313625+0000","last_peered":"2023-08-25T09:49:40.313625+0000","last_clean":"2023-08-25T09:49:40.313625+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:49:40.313625+0000","last_undegraded":"2023-08-25T09:49:40.313625+0000","last_fullsized":"2023-08-25T09:49:40.313625+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:04:41.693477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609626+0000","last_change":"2023-08-25T09:49:34.628078+0000","last_active":"2023-08-25T09:49:40.609626+0000","last_peered":"2023-08-25T09:49:40.609626+0000","last_clean":"2023-08-25T09:49:40.609626+0000","last_became_active":"2023-08-25T09:49:34.627744+0000","last_became_peered":"2023-08-25T09:49:34.627744+0000","last_unstale":"2023-08-25T09:49:40.609626+0000","last_undegraded":"2023-08-25T09:49:40.609626+0000","last_fullsized":"2023-08-25T09:49:40.609626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:15.076203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671559+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:49:39.671559+0000","last_peered":"2023-08-25T09:49:39.671559+0000","last_clean":"2023-08-25T09:49:39.671559+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:49:39.671559+0000","last_undegraded":"2023-08-25T09:49:39.671559+0000","last_fullsized":"2023-08-25T09:49:39.671559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:50:31.566722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313898+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:49:40.313898+0000","last_peered":"2023-08-25T09:49:40.313898+0000","last_clean":"2023-08-25T09:49:40.313898+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:49:40.313898+0000","last_undegraded":"2023-08-25T09:49:40.313898+0000","last_fullsized":"2023-08-25T09:49:40.313898+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:29:27.667548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668920+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:49:39.668920+0000","last_peered":"2023-08-25T09:49:39.668920+0000","last_clean":"2023-08-25T09:49:39.668920+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:49:39.668920+0000","last_undegraded":"2023-08-25T09:49:39.668920+0000","last_fullsized":"2023-08-25T09:49:39.668920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:28:58.529027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"28'2","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671825+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:49:39.671825+0000","last_peered":"2023-08-25T09:49:39.671825+0000","last_clean":"2023-08-25T09:49:39.671825+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:49:39.671825+0000","last_undegraded":"2023-08-25T09:49:39.671825+0000","last_fullsized":"2023-08-25T09:49:39.671825+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:53:27.311764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313371+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:49:40.313371+0000","last_peered":"2023-08-25T09:49:40.313371+0000","last_clean":"2023-08-25T09:49:40.313371+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:49:40.313371+0000","last_undegraded":"2023-08-25T09:49:40.313371+0000","last_fullsized":"2023-08-25T09:49:40.313371+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:23:01.195200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668159+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:49:39.668159+0000","last_peered":"2023-08-25T09:49:39.668159+0000","last_clean":"2023-08-25T09:49:39.668159+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:49:39.668159+0000","last_undegraded":"2023-08-25T09:49:39.668159+0000","last_fullsized":"2023-08-25T09:49:39.668159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:32:15.403697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"28'1","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672436+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:49:39.672436+0000","last_peered":"2023-08-25T09:49:39.672436+0000","last_clean":"2023-08-25T09:49:39.672436+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:49:39.672436+0000","last_undegraded":"2023-08-25T09:49:39.672436+0000","last_fullsized":"2023-08-25T09:49:39.672436+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:53.368049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671121+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:49:39.671121+0000","last_peered":"2023-08-25T09:49:39.671121+0000","last_clean":"2023-08-25T09:49:39.671121+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:49:39.671121+0000","last_undegraded":"2023-08-25T09:49:39.671121+0000","last_fullsized":"2023-08-25T09:49:39.671121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:05:58.232199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671520+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:49:39.671520+0000","last_peered":"2023-08-25T09:49:39.671520+0000","last_clean":"2023-08-25T09:49:39.671520+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:49:39.671520+0000","last_undegraded":"2023-08-25T09:49:39.671520+0000","last_fullsized":"2023-08-25T09:49:39.671520+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:08:33.021697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"29'10","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.531847+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:50:05.531847+0000","last_peered":"2023-08-25T09:50:05.531847+0000","last_clean":"2023-08-25T09:50:05.531847+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:50:05.531847+0000","last_undegraded":"2023-08-25T09:50:05.531847+0000","last_fullsized":"2023-08-25T09:50:05.531847+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:16:55.451701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610899+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:49:40.610899+0000","last_peered":"2023-08-25T09:49:40.610899+0000","last_clean":"2023-08-25T09:49:40.610899+0000","last_became_active":"2023-08-25T09:49:34.630012+0000","last_became_peered":"2023-08-25T09:49:34.630012+0000","last_unstale":"2023-08-25T09:49:40.610899+0000","last_undegraded":"2023-08-25T09:49:40.610899+0000","last_fullsized":"2023-08-25T09:49:40.610899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:11:33.237689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610543+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:49:40.610543+0000","last_peered":"2023-08-25T09:49:40.610543+0000","last_clean":"2023-08-25T09:49:40.610543+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:49:40.610543+0000","last_undegraded":"2023-08-25T09:49:40.610543+0000","last_fullsized":"2023-08-25T09:49:40.610543+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:25.560317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671645+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:49:39.671645+0000","last_peered":"2023-08-25T09:49:39.671645+0000","last_clean":"2023-08-25T09:49:39.671645+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:49:39.671645+0000","last_undegraded":"2023-08-25T09:49:39.671645+0000","last_fullsized":"2023-08-25T09:49:39.671645+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:37:05.937993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669858+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:49:39.669858+0000","last_peered":"2023-08-25T09:49:39.669858+0000","last_clean":"2023-08-25T09:49:39.669858+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:49:39.669858+0000","last_undegraded":"2023-08-25T09:49:39.669858+0000","last_fullsized":"2023-08-25T09:49:39.669858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:20:55.146183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"29'3","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.537356+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:50:05.537356+0000","last_peered":"2023-08-25T09:50:05.537356+0000","last_clean":"2023-08-25T09:50:05.537356+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:50:05.537356+0000","last_undegraded":"2023-08-25T09:50:05.537356+0000","last_fullsized":"2023-08-25T09:50:05.537356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:51.399369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672435+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:49:39.672435+0000","last_peered":"2023-08-25T09:49:39.672435+0000","last_clean":"2023-08-25T09:49:39.672435+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:49:39.672435+0000","last_undegraded":"2023-08-25T09:49:39.672435+0000","last_fullsized":"2023-08-25T09:49:39.672435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:41:04.943976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609915+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:49:40.609915+0000","last_peered":"2023-08-25T09:49:40.609915+0000","last_clean":"2023-08-25T09:49:40.609915+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:49:40.609915+0000","last_undegraded":"2023-08-25T09:49:40.609915+0000","last_fullsized":"2023-08-25T09:49:40.609915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:30:03.672997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"29'4","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:50:05.538798+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:50:05.538798+0000","last_peered":"2023-08-25T09:50:05.538798+0000","last_clean":"2023-08-25T09:50:05.538798+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:50:05.538798+0000","last_undegraded":"2023-08-25T09:50:05.538798+0000","last_fullsized":"2023-08-25T09:50:05.538798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:11:58.806748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.610312+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:49:40.610312+0000","last_peered":"2023-08-25T09:49:40.610312+0000","last_clean":"2023-08-25T09:49:40.610312+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:49:40.610312+0000","last_undegraded":"2023-08-25T09:49:40.610312+0000","last_fullsized":"2023-08-25T09:49:40.610312+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:15:05.871192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672132+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:49:39.672132+0000","last_peered":"2023-08-25T09:49:39.672132+0000","last_clean":"2023-08-25T09:49:39.672132+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:49:39.672132+0000","last_undegraded":"2023-08-25T09:49:39.672132+0000","last_fullsized":"2023-08-25T09:49:39.672132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:51:00.162861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609435+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:49:40.609435+0000","last_peered":"2023-08-25T09:49:40.609435+0000","last_clean":"2023-08-25T09:49:40.609435+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:49:40.609435+0000","last_undegraded":"2023-08-25T09:49:40.609435+0000","last_fullsized":"2023-08-25T09:49:40.609435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:54:36.909125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609010+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:49:40.609010+0000","last_peered":"2023-08-25T09:49:40.609010+0000","last_clean":"2023-08-25T09:49:40.609010+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:49:40.609010+0000","last_undegraded":"2023-08-25T09:49:40.609010+0000","last_fullsized":"2023-08-25T09:49:40.609010+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:45.655937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671033+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:49:39.671033+0000","last_peered":"2023-08-25T09:49:39.671033+0000","last_clean":"2023-08-25T09:49:39.671033+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:49:39.671033+0000","last_undegraded":"2023-08-25T09:49:39.671033+0000","last_fullsized":"2023-08-25T09:49:39.671033+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:19:24.241659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609947+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:49:40.609947+0000","last_peered":"2023-08-25T09:49:40.609947+0000","last_clean":"2023-08-25T09:49:40.609947+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:49:40.609947+0000","last_undegraded":"2023-08-25T09:49:40.609947+0000","last_fullsized":"2023-08-25T09:49:40.609947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:27:06.196565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533764+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:49:40.533764+0000","last_peered":"2023-08-25T09:49:40.533764+0000","last_clean":"2023-08-25T09:49:40.533764+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:49:40.533764+0000","last_undegraded":"2023-08-25T09:49:40.533764+0000","last_fullsized":"2023-08-25T09:49:40.533764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:53:01.633281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.672281+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:49:39.672281+0000","last_peered":"2023-08-25T09:49:39.672281+0000","last_clean":"2023-08-25T09:49:39.672281+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:49:39.672281+0000","last_undegraded":"2023-08-25T09:49:39.672281+0000","last_fullsized":"2023-08-25T09:49:39.672281+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:38:19.645378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533734+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:49:40.533734+0000","last_peered":"2023-08-25T09:49:40.533734+0000","last_clean":"2023-08-25T09:49:40.533734+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:49:40.533734+0000","last_undegraded":"2023-08-25T09:49:40.533734+0000","last_fullsized":"2023-08-25T09:49:40.533734+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:27:54.228223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.533904+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:49:40.533904+0000","last_peered":"2023-08-25T09:49:40.533904+0000","last_clean":"2023-08-25T09:49:40.533904+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:49:40.533904+0000","last_undegraded":"2023-08-25T09:49:40.533904+0000","last_fullsized":"2023-08-25T09:49:40.533904+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:06:19.107446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670060+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:49:39.670060+0000","last_peered":"2023-08-25T09:49:39.670060+0000","last_clean":"2023-08-25T09:49:39.670060+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:49:39.670060+0000","last_undegraded":"2023-08-25T09:49:39.670060+0000","last_fullsized":"2023-08-25T09:49:39.670060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:08:51.960006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.608748+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:49:40.608748+0000","last_peered":"2023-08-25T09:49:40.608748+0000","last_clean":"2023-08-25T09:49:40.608748+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:49:40.608748+0000","last_undegraded":"2023-08-25T09:49:40.608748+0000","last_fullsized":"2023-08-25T09:49:40.608748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:35.777074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668458+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:49:39.668458+0000","last_peered":"2023-08-25T09:49:39.668458+0000","last_clean":"2023-08-25T09:49:39.668458+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:49:39.668458+0000","last_undegraded":"2023-08-25T09:49:39.668458+0000","last_fullsized":"2023-08-25T09:49:39.668458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:40:57.260235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669943+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:49:39.669943+0000","last_peered":"2023-08-25T09:49:39.669943+0000","last_clean":"2023-08-25T09:49:39.669943+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:49:39.669943+0000","last_undegraded":"2023-08-25T09:49:39.669943+0000","last_fullsized":"2023-08-25T09:49:39.669943+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:42:36.500862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670365+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:49:39.670365+0000","last_peered":"2023-08-25T09:49:39.670365+0000","last_clean":"2023-08-25T09:49:39.670365+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:49:39.670365+0000","last_undegraded":"2023-08-25T09:49:39.670365+0000","last_fullsized":"2023-08-25T09:49:39.670365+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:33:44.740556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314206+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:49:40.314206+0000","last_peered":"2023-08-25T09:49:40.314206+0000","last_clean":"2023-08-25T09:49:40.314206+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:49:40.314206+0000","last_undegraded":"2023-08-25T09:49:40.314206+0000","last_fullsized":"2023-08-25T09:49:40.314206+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:35.345143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.670439+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:49:39.670439+0000","last_peered":"2023-08-25T09:49:39.670439+0000","last_clean":"2023-08-25T09:49:39.670439+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:49:39.670439+0000","last_undegraded":"2023-08-25T09:49:39.670439+0000","last_fullsized":"2023-08-25T09:49:39.670439+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:38.142714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669800+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:49:39.669800+0000","last_peered":"2023-08-25T09:49:39.669800+0000","last_clean":"2023-08-25T09:49:39.669800+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:49:39.669800+0000","last_undegraded":"2023-08-25T09:49:39.669800+0000","last_fullsized":"2023-08-25T09:49:39.669800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:05:15.100928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668185+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:49:39.668185+0000","last_peered":"2023-08-25T09:49:39.668185+0000","last_clean":"2023-08-25T09:49:39.668185+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:49:39.668185+0000","last_undegraded":"2023-08-25T09:49:39.668185+0000","last_fullsized":"2023-08-25T09:49:39.668185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:43:52.824866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671435+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:49:39.671435+0000","last_peered":"2023-08-25T09:49:39.671435+0000","last_clean":"2023-08-25T09:49:39.671435+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:49:39.671435+0000","last_undegraded":"2023-08-25T09:49:39.671435+0000","last_fullsized":"2023-08-25T09:49:39.671435+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:22:34.193671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668773+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:49:39.668773+0000","last_peered":"2023-08-25T09:49:39.668773+0000","last_clean":"2023-08-25T09:49:39.668773+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:49:39.668773+0000","last_undegraded":"2023-08-25T09:49:39.668773+0000","last_fullsized":"2023-08-25T09:49:39.668773+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:38:09.260537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.313744+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:49:40.313744+0000","last_peered":"2023-08-25T09:49:40.313744+0000","last_clean":"2023-08-25T09:49:40.313744+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:49:40.313744+0000","last_undegraded":"2023-08-25T09:49:40.313744+0000","last_fullsized":"2023-08-25T09:49:40.313744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:47:49.414014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314502+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:49:40.314502+0000","last_peered":"2023-08-25T09:49:40.314502+0000","last_clean":"2023-08-25T09:49:40.314502+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:49:40.314502+0000","last_undegraded":"2023-08-25T09:49:40.314502+0000","last_fullsized":"2023-08-25T09:49:40.314502+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:52.308305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669862+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:49:39.669862+0000","last_peered":"2023-08-25T09:49:39.669862+0000","last_clean":"2023-08-25T09:49:39.669862+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:49:39.669862+0000","last_undegraded":"2023-08-25T09:49:39.669862+0000","last_fullsized":"2023-08-25T09:49:39.669862+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:33:20.778440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669203+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:49:39.669203+0000","last_peered":"2023-08-25T09:49:39.669203+0000","last_clean":"2023-08-25T09:49:39.669203+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:49:39.669203+0000","last_undegraded":"2023-08-25T09:49:39.669203+0000","last_fullsized":"2023-08-25T09:49:39.669203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:04:14.357542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669995+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:49:39.669995+0000","last_peered":"2023-08-25T09:49:39.669995+0000","last_clean":"2023-08-25T09:49:39.669995+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:49:39.669995+0000","last_undegraded":"2023-08-25T09:49:39.669995+0000","last_fullsized":"2023-08-25T09:49:39.669995+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:06.555517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668019+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:49:39.668019+0000","last_peered":"2023-08-25T09:49:39.668019+0000","last_clean":"2023-08-25T09:49:39.668019+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:49:39.668019+0000","last_undegraded":"2023-08-25T09:49:39.668019+0000","last_fullsized":"2023-08-25T09:49:39.668019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:35:29.581577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.671260+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:49:39.671260+0000","last_peered":"2023-08-25T09:49:39.671260+0000","last_clean":"2023-08-25T09:49:39.671260+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:49:39.671260+0000","last_undegraded":"2023-08-25T09:49:39.671260+0000","last_fullsized":"2023-08-25T09:49:39.671260+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:23:43.256956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669816+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:49:39.669816+0000","last_peered":"2023-08-25T09:49:39.669816+0000","last_clean":"2023-08-25T09:49:39.669816+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:49:39.669816+0000","last_undegraded":"2023-08-25T09:49:39.669816+0000","last_fullsized":"2023-08-25T09:49:39.669816+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:05:53.483156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669716+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:49:39.669716+0000","last_peered":"2023-08-25T09:49:39.669716+0000","last_clean":"2023-08-25T09:49:39.669716+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:49:39.669716+0000","last_undegraded":"2023-08-25T09:49:39.669716+0000","last_fullsized":"2023-08-25T09:49:39.669716+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:17.656387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.609913+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:49:40.609913+0000","last_peered":"2023-08-25T09:49:40.609913+0000","last_clean":"2023-08-25T09:49:40.609913+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:49:40.609913+0000","last_undegraded":"2023-08-25T09:49:40.609913+0000","last_fullsized":"2023-08-25T09:49:40.609913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:04:46.102346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.669727+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:49:39.669727+0000","last_peered":"2023-08-25T09:49:39.669727+0000","last_clean":"2023-08-25T09:49:39.669727+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:49:39.669727+0000","last_undegraded":"2023-08-25T09:49:39.669727+0000","last_fullsized":"2023-08-25T09:49:39.669727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:03:17.962039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:39.668925+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:49:39.668925+0000","last_peered":"2023-08-25T09:49:39.668925+0000","last_clean":"2023-08-25T09:49:39.668925+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:49:39.668925+0000","last_undegraded":"2023-08-25T09:49:39.668925+0000","last_fullsized":"2023-08-25T09:49:39.668925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:15:02.299452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-08-25T09:49:40.314453+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:49:40.314453+0000","last_peered":"2023-08-25T09:49:40.314453+0000","last_clean":"2023-08-25T09:49:40.314453+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:49:40.314453+0000","last_undegraded":"2023-08-25T09:49:40.314453+0000","last_fullsized":"2023-08-25T09:49:40.314453+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:31.990094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":6045,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509452,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":130840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.651}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.339}]}]},{"osd":7,"up_from":15,"seq":64424509452,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":405504,"data_stored":721238,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.423}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4299999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509453,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.175}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3129999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":126995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.637}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]}]},{"osd":3,"up_from":14,"seq":60129542155,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":126925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1020000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":126321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":126231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":389120,"data_stored":717893,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:08.241 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-08-25T09:50:08.242 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-08-25T09:50:08.242 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-08-25T09:50:08.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-08-25T09:50:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.483+0000 7fd1667a6640 1 Processor -- start 2023-08-25T09:50:08.485 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.483+0000 7fd1667a6640 1 -- start start 2023-08-25T09:50:08.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd1667a6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd1601594d0 0x7fd1600c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd1667a6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd1601598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd1667a6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16006a960 con 0x7fd16006cc70 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd1667a6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16006ac20 con 0x7fd1601594d0 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd1667a6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006af00 con 0x7fd1600c3940 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd1601594d0 0x7fd1600c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd1601594d0 0x7fd1600c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60336/0 (socket says 172.21.15.71:60336) 2023-08-25T09:50:08.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 -- 172.21.15.71:0/1451150284 learned_addr learned my addr 172.21.15.71:0/1451150284 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 -- 172.21.15.71:0/1451150284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 msgr2=0x7fd1601598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd164d43640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd1601598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd1601598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 -- 172.21.15.71:0/1451150284 >> v1:172.21.15.164:6789/0 conn(0x7fd16006cc70 legacy=0x7fd16014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.485+0000 7fd15ffff640 1 -- 172.21.15.71:0/1451150284 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600c2e60 con 0x7fd1601594d0 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.486+0000 7fd164d43640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd1601598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.487+0000 7fd15ffff640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1601594d0 0x7fd1600c4c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd150009870 tx=0x7fd15002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f677057c26445c6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.487+0000 7fd15f7fe640 1 -- 172.21.15.71:0/1451150284 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150031640 con 0x7fd1601594d0 2023-08-25T09:50:08.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.487+0000 7fd15f7fe640 1 -- 172.21.15.71:0/1451150284 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1500317a0 con 0x7fd1601594d0 2023-08-25T09:50:08.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.487+0000 7fd15f7fe640 1 -- 172.21.15.71:0/1451150284 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150031b30 con 0x7fd1601594d0 2023-08-25T09:50:08.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.487+0000 7fd1667a6640 1 -- 172.21.15.71:0/1451150284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1601594d0 msgr2=0x7fd1600c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.487+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1601594d0 0x7fd1600c4c90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd150009870 tx=0x7fd15002b8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.488+0000 7fd1667a6640 1 -- 172.21.15.71:0/1451150284 shutdown_connections 2023-08-25T09:50:08.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.488+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd1601598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.488+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1451150284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1601594d0 0x7fd1600c4c90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.488+0000 7fd1667a6640 1 -- 172.21.15.71:0/1451150284 >> 172.21.15.71:0/1451150284 conn(0x7fd1600654f0 msgr2=0x7fd160065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:08.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.488+0000 7fd1667a6640 1 -- 172.21.15.71:0/1451150284 shutdown_connections 2023-08-25T09:50:08.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.488+0000 7fd1667a6640 1 -- 172.21.15.71:0/1451150284 wait complete. 2023-08-25T09:50:08.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.489+0000 7fd1667a6640 1 Processor -- start 2023-08-25T09:50:08.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.489+0000 7fd1667a6640 1 -- start start 2023-08-25T09:50:08.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.490+0000 7fd1667a6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 0x7fd1600c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.490+0000 7fd1667a6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd160157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.490+0000 7fd1667a6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1601594d0 0x7fd160158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.490+0000 7fd1667a6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160157b50 con 0x7fd16006cc70 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.490+0000 7fd1667a6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601ee650 con 0x7fd1601594d0 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.490+0000 7fd1667a6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601ee930 con 0x7fd1600c3940 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 0x7fd1600c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 0x7fd1600c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60350/0 (socket says 172.21.15.71:60350) 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 -- 172.21.15.71:0/1205069348 learned_addr learned my addr 172.21.15.71:0/1205069348 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd157fff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd160157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd164d43640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1601594d0 0x7fd160158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 -- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 msgr2=0x7fd160157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd160157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 -- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1601594d0 msgr2=0x7fd160158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1601594d0 0x7fd160158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 -- 172.21.15.71:0/1205069348 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1500094d0 con 0x7fd16006cc70 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd164d43640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1601594d0 0x7fd160158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:08.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd157fff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd160157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:08.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.491+0000 7fd15ffff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 0x7fd1600c6080 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd150009870 tx=0x7fd1500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:08.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.492+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15003ac60 con 0x7fd16006cc70 2023-08-25T09:50:08.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.492+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15003ade0 con 0x7fd16006cc70 2023-08-25T09:50:08.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.492+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1601eec10 con 0x7fd16006cc70 2023-08-25T09:50:08.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.492+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150044630 con 0x7fd16006cc70 2023-08-25T09:50:08.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.492+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd160157600 con 0x7fd16006cc70 2023-08-25T09:50:08.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.493+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1600678d0 con 0x7fd16006cc70 2023-08-25T09:50:08.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.494+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd15002f050 con 0x7fd16006cc70 2023-08-25T09:50:08.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.495+0000 7fd15d7fa640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd14006a050 0x7fd14006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:08.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.495+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6813+0+0 (secure 0 0 0) 0x7fd1500aab80 con 0x7fd16006cc70 2023-08-25T09:50:08.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.495+0000 7fd157fff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd14006a050 0x7fd14006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:08.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.496+0000 7fd157fff640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd14006a050 0x7fd14006c510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd1600c9a10 tx=0x7fd148008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:08.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.499+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd150075f10 con 0x7fd16006cc70 2023-08-25T09:50:08.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.730+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd160069160 con 0x7fd16006cc70 2023-08-25T09:50:08.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.732+0000 7fd15d7fa640 1 -- 172.21.15.71:0/1205069348 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7fd1500350e0 con 0x7fd16006cc70 2023-08-25T09:50:08.734 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:08.734 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-08-25T09:50:08.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd14006a050 msgr2=0x7fd14006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd14006a050 0x7fd14006c510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd1600c9a10 tx=0x7fd148008040 comp rx=0 tx=0).stop 2023-08-25T09:50:08.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 msgr2=0x7fd1600c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:08.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 0x7fd1600c6080 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd150009870 tx=0x7fd1500046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 shutdown_connections 2023-08-25T09:50:08.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd1601594d0 0x7fd160158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd14006a050 0x7fd14006c510 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd1600c3940 0x7fd160157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 --2- 172.21.15.71:0/1205069348 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd16006cc70 0x7fd1600c6080 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:08.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 >> 172.21.15.71:0/1205069348 conn(0x7fd1600654f0 msgr2=0x7fd160069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:08.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 shutdown_connections 2023-08-25T09:50:08.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:08.735+0000 7fd1667a6640 1 -- 172.21.15.71:0/1205069348 wait complete. 2023-08-25T09:50:08.757 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T09:50:08.755+0000 7f5fc1b18640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-08-25T09:50:11.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-08-25T09:50:11.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.958+0000 7ff2ec05e640 1 Processor -- start 2023-08-25T09:50:11.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.958+0000 7ff2ec05e640 1 -- start start 2023-08-25T09:50:11.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2ec05e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff2e406a960 0x7ff2e415d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:11.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2ec05e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e415c110 0x7ff2e4071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:11.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2ec05e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e415cc30 con 0x7ff2e40bf9b0 2023-08-25T09:50:11.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2ec05e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e40c7390 con 0x7ff2e406a960 2023-08-25T09:50:11.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2ec05e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e406bf40 con 0x7ff2e415c110 2023-08-25T09:50:11.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2e9dfa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff2e406a960 0x7ff2e415d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:11.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2e9dfa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff2e406a960 0x7ff2e415d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56590/0 (socket says 172.21.15.71:56590) 2023-08-25T09:50:11.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2e9dfa640 1 -- 172.21.15.71:0/424509192 learned_addr learned my addr 172.21.15.71:0/424509192 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:11.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.960+0000 7ff2ea5fb640 1 --1- 172.21.15.71:0/424509192 >> v1:172.21.15.71:6789/0 conn(0x7ff2e40bf9b0 0x7ff2e4158f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41726/0 (socket says 172.21.15.71:41726) 2023-08-25T09:50:11.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.961+0000 7ff2e95f9640 1 --2- 172.21.15.71:0/424509192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e415c110 0x7ff2e4071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:11.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.961+0000 7ff2e9dfa640 1 -- 172.21.15.71:0/424509192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e415c110 msgr2=0x7ff2e4071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:11.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.961+0000 7ff2e9dfa640 1 --2- 172.21.15.71:0/424509192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e415c110 0x7ff2e4071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:11.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.961+0000 7ff2e9dfa640 1 -- 172.21.15.71:0/424509192 >> v1:172.21.15.71:6789/0 conn(0x7ff2e40bf9b0 legacy=0x7ff2e4158f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:11.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.961+0000 7ff2e8df8640 1 -- 172.21.15.71:0/424509192 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504253557 0 0) 0x7ff2e415cc30 con 0x7ff2e40bf9b0 2023-08-25T09:50:11.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.961+0000 7ff2e9dfa640 1 -- 172.21.15.71:0/424509192 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40c6dd0 con 0x7ff2e406a960 2023-08-25T09:50:11.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.962+0000 7ff2e9dfa640 1 --2- 172.21.15.71:0/424509192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e406a960 0x7ff2e415d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4009e80 tx=0x7ff2d402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1a9365571f63a6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.962+0000 7ff2e8df8640 1 -- 172.21.15.71:0/424509192 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d402cb70 con 0x7ff2e406a960 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.962+0000 7ff2e8df8640 1 -- 172.21.15.71:0/424509192 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d402ccd0 con 0x7ff2e406a960 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.962+0000 7ff2e8df8640 1 -- 172.21.15.71:0/424509192 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4034610 con 0x7ff2e406a960 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 -- 172.21.15.71:0/424509192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e406a960 msgr2=0x7ff2e415d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/424509192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e406a960 0x7ff2e415d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4009e80 tx=0x7ff2d402bfd0 comp rx=0 tx=0).stop 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 -- 172.21.15.71:0/424509192 shutdown_connections 2023-08-25T09:50:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/424509192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e415c110 0x7ff2e4071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:11.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/424509192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e406a960 0x7ff2e415d0c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:11.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 -- 172.21.15.71:0/424509192 >> 172.21.15.71:0/424509192 conn(0x7ff2e4069e80 msgr2=0x7ff2e40654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:11.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 -- 172.21.15.71:0/424509192 shutdown_connections 2023-08-25T09:50:11.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.963+0000 7ff2ec05e640 1 -- 172.21.15.71:0/424509192 wait complete. 2023-08-25T09:50:11.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.965+0000 7ff2ec05e640 1 Processor -- start 2023-08-25T09:50:11.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.965+0000 7ff2ec05e640 1 -- start start 2023-08-25T09:50:11.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ec05e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e406a960 0x7ff2e41f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:11.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ec05e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e40bf9b0 0x7ff2e40be890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ec05e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 0x7ff2e40bedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ec05e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e40be470 con 0x7ff2e415c110 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ec05e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e4156d90 con 0x7ff2e40bf9b0 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ec05e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e41f2880 con 0x7ff2e406a960 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ea5fb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 0x7ff2e40bedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ea5fb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 0x7ff2e40bedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56600/0 (socket says 172.21.15.71:56600) 2023-08-25T09:50:11.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.966+0000 7ff2ea5fb640 1 -- 172.21.15.71:0/604105844 learned_addr learned my addr 172.21.15.71:0/604105844 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2e9dfa640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e406a960 0x7ff2e41f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ea5fb640 1 -- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e406a960 msgr2=0x7ff2e41f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ea5fb640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e406a960 0x7ff2e41f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2e95f9640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e40bf9b0 0x7ff2e40be890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ea5fb640 1 -- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e40bf9b0 msgr2=0x7ff2e40be890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ea5fb640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e40bf9b0 0x7ff2e40be890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:11.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ea5fb640 1 -- 172.21.15.71:0/604105844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d4009b30 con 0x7ff2e415c110 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2e9dfa640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e406a960 0x7ff2e41f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ea5fb640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 0x7ff2e40bedd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c450 tx=0x7ff2dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc0074e0 con 0x7ff2e415c110 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc007660 con 0x7ff2e415c110 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc004a20 con 0x7ff2e415c110 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e4156fe0 con 0x7ff2e415c110 2023-08-25T09:50:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.967+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e41572c0 con 0x7ff2e415c110 2023-08-25T09:50:11.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.969+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e40be5b0 con 0x7ff2e415c110 2023-08-25T09:50:11.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.969+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff2dc004b80 con 0x7ff2e415c110 2023-08-25T09:50:11.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.971+0000 7ff2e27fc640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d006a050 0x7ff2d006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:11.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.971+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff2dc087b00 con 0x7ff2e415c110 2023-08-25T09:50:11.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.971+0000 7ff2e9dfa640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d006a050 0x7ff2d006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:11.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.972+0000 7ff2e9dfa640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d006a050 0x7ff2d006c510 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4035000 tx=0x7ff2d4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:11.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:11.976+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff2e40be5b0 con 0x7ff2e415c110 2023-08-25T09:50:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.206+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff2e40be5b0 con 0x7ff2e415c110 2023-08-25T09:50:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.207+0000 7ff2e27fc640 1 -- 172.21.15.71:0/604105844 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7ff2e40be5b0 con 0x7ff2e415c110 2023-08-25T09:50:12.208 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:12.209 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-08-25T09:50:12.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.209+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d006a050 msgr2=0x7ff2d006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:12.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.209+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d006a050 0x7ff2d006c510 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4035000 tx=0x7ff2d4030000 comp rx=0 tx=0).stop 2023-08-25T09:50:12.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.209+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 msgr2=0x7ff2e40bedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.209+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 0x7ff2e40bedd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c450 tx=0x7ff2dc00c910 comp rx=0 tx=0).stop 2023-08-25T09:50:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 shutdown_connections 2023-08-25T09:50:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e40bf9b0 0x7ff2e40be890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d006a050 0x7ff2d006c510 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e415c110 0x7ff2e40bedd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:12.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 --2- 172.21.15.71:0/604105844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e406a960 0x7ff2e41f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:12.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 >> 172.21.15.71:0/604105844 conn(0x7ff2e4069e80 msgr2=0x7ff2e40cc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:12.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 shutdown_connections 2023-08-25T09:50:12.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:12.210+0000 7ff2ec05e640 1 -- 172.21.15.71:0/604105844 wait complete. 2023-08-25T09:50:13.757 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T09:50:13.756+0000 7f5fc1b18640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-08-25T09:50:15.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-08-25T09:50:15.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.454+0000 7ff4f0f4e640 1 Processor -- start 2023-08-25T09:50:15.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.454+0000 7ff4f0f4e640 1 -- start start 2023-08-25T09:50:15.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.455+0000 7ff4f0f4e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4ec1510e0 0x7ff4ec0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.455+0000 7ff4f0f4e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec0c1dc0 con 0x7ff4ec0c3660 2023-08-25T09:50:15.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4f0f4e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec156e00 con 0x7ff4ec0c3a40 2023-08-25T09:50:15.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4f0f4e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4ec157a80 con 0x7ff4ec1510e0 2023-08-25T09:50:15.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4ea59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff4ec0c3660 0x7ff4ec06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41728/0 (socket says 172.21.15.71:41728) 2023-08-25T09:50:15.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4ea59c640 1 -- 172.21.15.71:0/3197899533 learned_addr learned my addr 172.21.15.71:0/3197899533 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:15.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4ead9d640 1 --2- 172.21.15.71:0/3197899533 >> v2:172.21.15.164:3300/0 conn(0x7ff4ec1510e0 0x7ff4ec0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:15.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134906796 0 0) 0x7ff4ec0c1dc0 con 0x7ff4ec0c3660 2023-08-25T09:50:15.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.456+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d8003520 con 0x7ff4ec0c3660 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428963171 0 0) 0x7ff4d8003520 con 0x7ff4ec0c3660 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 >> v2:172.21.15.164:3300/0 conn(0x7ff4ec1510e0 msgr2=0x7ff4ec0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 --2- 172.21.15.71:0/3197899533 >> v2:172.21.15.164:3300/0 conn(0x7ff4ec1510e0 0x7ff4ec0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 >> v1:172.21.15.164:6789/0 conn(0x7ff4ec0c3a40 legacy=0x7ff4ec1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec1570e0 con 0x7ff4ec0c3660 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff4dc001670 con 0x7ff4ec0c3660 2023-08-25T09:50:15.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 >> v1:172.21.15.71:6789/0 conn(0x7ff4ec0c3660 legacy=0x7ff4ec06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4d80039e0 0x7ff4d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4d80047c0 0x7ff4d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec157a80 con 0x7ff4d8005180 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d800c090 con 0x7ff4d80047c0 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d800c250 con 0x7ff4d80039e0 2023-08-25T09:50:15.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4ead9d640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.457+0000 7ff4e3fff640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4d80047c0 0x7ff4d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4ead9d640 1 -- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4d80039e0 msgr2=0x7ff4d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4ead9d640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4d80039e0 0x7ff4d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4ead9d640 1 -- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4d80047c0 msgr2=0x7ff4d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4ead9d640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4d80047c0 0x7ff4d8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4ead9d640 1 -- 172.21.15.71:0/3197899533 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8003520 con 0x7ff4d8005180 2023-08-25T09:50:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4e3fff640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4d80047c0 0x7ff4d8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.458+0000 7ff4ead9d640 1 -- 172.21.15.71:0/3197899533 reap_dead start 2023-08-25T09:50:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.459+0000 7ff4ead9d640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4d800b980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009fc0 tx=0x7ff4d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c23488f90913b2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.459+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e402cc60 con 0x7ff4d8005180 2023-08-25T09:50:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.459+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e400f040 con 0x7ff4d8005180 2023-08-25T09:50:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.459+0000 7ff4e9d9b640 1 -- 172.21.15.71:0/3197899533 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4003990 con 0x7ff4d8005180 2023-08-25T09:50:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.459+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/3197899533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 msgr2=0x7ff4d800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.459+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4d800b980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009fc0 tx=0x7ff4d80057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.460+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/3197899533 shutdown_connections 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.460+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/3197899533 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4d800b980 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.460+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/3197899533 >> 172.21.15.71:0/3197899533 conn(0x7ff4ec0654f0 msgr2=0x7ff4ec065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.460+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/3197899533 shutdown_connections 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.460+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/3197899533 wait complete. 2023-08-25T09:50:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.461+0000 7ff4f0f4e640 1 Processor -- start 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.461+0000 7ff4f0f4e640 1 -- start start 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.462+0000 7ff4f0f4e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4ec153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.462+0000 7ff4f0f4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4ec14e990 0x7ff4ec06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.462+0000 7ff4f0f4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4ec1ed600 0x7ff4ec144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.462+0000 7ff4f0f4e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec06f420 con 0x7ff4d8005180 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.462+0000 7ff4f0f4e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec151fa0 con 0x7ff4ec1ed600 2023-08-25T09:50:15.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.462+0000 7ff4f0f4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec151960 con 0x7ff4ec14e990 2023-08-25T09:50:15.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4ec153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:15.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4ec153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56636/0 (socket says 172.21.15.71:56636) 2023-08-25T09:50:15.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 -- 172.21.15.71:0/1386697759 learned_addr learned my addr 172.21.15.71:0/1386697759 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:15.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ead9d640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4ec1ed600 0x7ff4ec144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:15.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 -- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4ec14e990 msgr2=0x7ff4ec06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4ec14e990 0x7ff4ec06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 -- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4ec1ed600 msgr2=0x7ff4ec144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4ec1ed600 0x7ff4ec144100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 -- 172.21.15.71:0/1386697759 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec151ce0 con 0x7ff4d8005180 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.463+0000 7ff4ea59c640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4ec153350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00c700 tx=0x7ff4dc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.464+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0072d0 con 0x7ff4d8005180 2023-08-25T09:50:15.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.464+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc007450 con 0x7ff4d8005180 2023-08-25T09:50:15.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.464+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0047e0 con 0x7ff4d8005180 2023-08-25T09:50:15.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.464+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec1ee280 con 0x7ff4d8005180 2023-08-25T09:50:15.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.464+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec1515f0 con 0x7ff4d8005180 2023-08-25T09:50:15.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.465+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ec069160 con 0x7ff4d8005180 2023-08-25T09:50:15.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.466+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff4dc004960 con 0x7ff4d8005180 2023-08-25T09:50:15.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.467+0000 7ff4e2ffd640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff4cc06a050 0x7ff4cc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:15.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.467+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff4dc087c80 con 0x7ff4d8005180 2023-08-25T09:50:15.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.467+0000 7ff4e3fff640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff4cc06a050 0x7ff4cc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:15.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.468+0000 7ff4e3fff640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff4cc06a050 0x7ff4cc06c510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec070260 tx=0x7ff4d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:15.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.471+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff4dc053030 con 0x7ff4d8005180 2023-08-25T09:50:15.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.701+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff4ec154680 con 0x7ff4d8005180 2023-08-25T09:50:15.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.702+0000 7ff4e2ffd640 1 -- 172.21.15.71:0/1386697759 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+196 (secure 0 0 0) 0x7ff4dc018080 con 0x7ff4d8005180 2023-08-25T09:50:15.704 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:15.704 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2023-08-25T09:50:15.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.704+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff4cc06a050 msgr2=0x7ff4cc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.704+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff4cc06a050 0x7ff4cc06c510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec070260 tx=0x7ff4d4009040 comp rx=0 tx=0).stop 2023-08-25T09:50:15.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.704+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 msgr2=0x7ff4ec153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:15.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.704+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4ec153350 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00c700 tx=0x7ff4dc00cbc0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 shutdown_connections 2023-08-25T09:50:15.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4ec1ed600 0x7ff4ec144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff4cc06a050 0x7ff4cc06c510 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4ec14e990 0x7ff4ec06eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 --2- 172.21.15.71:0/1386697759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff4d8005180 0x7ff4ec153350 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 >> 172.21.15.71:0/1386697759 conn(0x7ff4ec0654f0 msgr2=0x7ff4ec0658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 shutdown_connections 2023-08-25T09:50:15.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:15.705+0000 7ff4f0f4e640 1 -- 172.21.15.71:0/1386697759 wait complete. 2023-08-25T09:50:18.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-08-25T09:50:18.757 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T09:50:18.756+0000 7f5fc1b18640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-08-25T09:50:18.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.946+0000 7f92e0638640 1 Processor -- start 2023-08-25T09:50:18.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.946+0000 7f92e0638640 1 -- start start 2023-08-25T09:50:18.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.948+0000 7f92e0638640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d8153200 0x7f92d815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:18.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.948+0000 7f92e0638640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f92d80c6de0 0x7f92d8163850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:18.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.948+0000 7f92e0638640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d815c8d0 con 0x7f92d8159950 2023-08-25T09:50:18.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.948+0000 7f92e0638640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d8159e20 con 0x7f92d80c6de0 2023-08-25T09:50:18.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.948+0000 7f92e0638640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81535e0 con 0x7f92d8153200 2023-08-25T09:50:18.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.949+0000 7f92debd5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f92d80c6de0 0x7f92d8163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:18.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.949+0000 7f92debd5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f92d80c6de0 0x7f92d8163850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:45736/0 (socket says 172.21.15.71:45736) 2023-08-25T09:50:18.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.949+0000 7f92debd5640 1 -- 172.21.15.71:0/2861153898 learned_addr learned my addr 172.21.15.71:0/2861153898 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:18.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.949+0000 7f92de3d4640 1 --2- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d8153200 0x7f92d815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:18.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.950+0000 7f92debd5640 1 -- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d8153200 msgr2=0x7f92d815cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:18.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.950+0000 7f92debd5640 1 --2- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d8153200 0x7f92d815cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:18.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.950+0000 7f92debd5640 1 -- 172.21.15.71:0/2861153898 >> v1:172.21.15.164:6789/0 conn(0x7f92d8159950 legacy=0x7f92d806e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:18.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.950+0000 7f92debd5640 1 -- 172.21.15.71:0/2861153898 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d806aa00 con 0x7f92d80c6de0 2023-08-25T09:50:18.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.951+0000 7f92debd5640 1 --2- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d80c6de0 0x7f92d8163850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009e50 tx=0x7f92d002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676fbc7413bd87e0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:18.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92dd3d2640 1 -- 172.21.15.71:0/2861153898 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d002cb90 con 0x7f92d80c6de0 2023-08-25T09:50:18.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92dd3d2640 1 -- 172.21.15.71:0/2861153898 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d002ccf0 con 0x7f92d80c6de0 2023-08-25T09:50:18.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92dd3d2640 1 -- 172.21.15.71:0/2861153898 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0034610 con 0x7f92d80c6de0 2023-08-25T09:50:18.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92e0638640 1 -- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d80c6de0 msgr2=0x7f92d8163850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:18.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92e0638640 1 --2- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d80c6de0 0x7f92d8163850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009e50 tx=0x7f92d002bff0 comp rx=0 tx=0).stop 2023-08-25T09:50:18.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92e0638640 1 -- 172.21.15.71:0/2861153898 shutdown_connections 2023-08-25T09:50:18.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92e0638640 1 --2- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d80c6de0 0x7f92d8163850 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:18.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92e0638640 1 --2- 172.21.15.71:0/2861153898 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d8153200 0x7f92d815cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:18.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.952+0000 7f92e0638640 1 -- 172.21.15.71:0/2861153898 >> 172.21.15.71:0/2861153898 conn(0x7f92d80654f0 msgr2=0x7f92d80658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:18.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.953+0000 7f92e0638640 1 -- 172.21.15.71:0/2861153898 shutdown_connections 2023-08-25T09:50:18.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.953+0000 7f92e0638640 1 -- 172.21.15.71:0/2861153898 wait complete. 2023-08-25T09:50:18.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.955+0000 7f92e0638640 1 Processor -- start 2023-08-25T09:50:18.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.955+0000 7f92e0638640 1 -- start start 2023-08-25T09:50:18.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92e0638640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d80c6de0 0x7f92d814e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:18.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92e0638640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8153200 0x7f92d80c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:18.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92e0638640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 0x7f92d80c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:18.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92e0638640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81540a0 con 0x7f92d8159950 2023-08-25T09:50:18.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92e0638640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80c3f80 con 0x7f92d8153200 2023-08-25T09:50:18.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92e0638640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80c3ca0 con 0x7f92d80c6de0 2023-08-25T09:50:18.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92debd5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 0x7f92d80c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92debd5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 0x7f92d80c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56640/0 (socket says 172.21.15.71:56640) 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.956+0000 7f92debd5640 1 -- 172.21.15.71:0/841409892 learned_addr learned my addr 172.21.15.71:0/841409892 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92de3d4640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d80c6de0 0x7f92d814e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92ddbd3640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8153200 0x7f92d80c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92debd5640 1 -- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d80c6de0 msgr2=0x7f92d814e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92debd5640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d80c6de0 0x7f92d814e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:18.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92debd5640 1 -- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8153200 msgr2=0x7f92d80c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92debd5640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8153200 0x7f92d80c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92debd5640 1 -- 172.21.15.71:0/841409892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d0009b30 con 0x7f92d8159950 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92ddbd3640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8153200 0x7f92d80c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92debd5640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 0x7f92d80c41d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009e50 tx=0x7f92d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0009510 con 0x7f92d8159950 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.957+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d003e450 con 0x7f92d8159950 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.958+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d81f1ae0 con 0x7f92d8159950 2023-08-25T09:50:18.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.958+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d003e800 con 0x7f92d8159950 2023-08-25T09:50:18.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.958+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d81f20a0 con 0x7f92d8159950 2023-08-25T09:50:18.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.959+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d8069160 con 0x7f92d8159950 2023-08-25T09:50:18.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.960+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f92d003e960 con 0x7f92d8159950 2023-08-25T09:50:18.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.961+0000 7f92ceffd640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92bc06a080 0x7f92bc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:18.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.961+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f92d00ab4a0 con 0x7f92d8159950 2023-08-25T09:50:18.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.961+0000 7f92de3d4640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92bc06a080 0x7f92bc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:18.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.961+0000 7f92de3d4640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92bc06a080 0x7f92bc06c540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009ad0 tx=0x7f92d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:18.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:18.965+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f92d00767c0 con 0x7f92d8159950 2023-08-25T09:50:19.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.193+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f92d806ed40 con 0x7f92d8159950 2023-08-25T09:50:19.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.194+0000 7f92ceffd640 1 -- 172.21.15.71:0/841409892 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f92d002ce60 con 0x7f92d8159950 2023-08-25T09:50:19.197 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:19.197 INFO:teuthology.orchestra.run.smithi071.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-08-25T09:50:19.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92bc06a080 msgr2=0x7f92bc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92bc06a080 0x7f92bc06c540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009ad0 tx=0x7f92d4008040 comp rx=0 tx=0).stop 2023-08-25T09:50:19.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 msgr2=0x7f92d80c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 0x7f92d80c41d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009e50 tx=0x7f92d0004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 shutdown_connections 2023-08-25T09:50:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8153200 0x7f92d80c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92bc06a080 0x7f92bc06c540 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d8159950 0x7f92d80c41d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.196+0000 7f92e0638640 1 --2- 172.21.15.71:0/841409892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d80c6de0 0x7f92d814e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.197+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 >> 172.21.15.71:0/841409892 conn(0x7f92d80654f0 msgr2=0x7f92d81e4390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:19.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.197+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 shutdown_connections 2023-08-25T09:50:19.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.197+0000 7f92e0638640 1 -- 172.21.15.71:0/841409892 wait complete. 2023-08-25T09:50:19.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:50:19.216 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-08-25T09:50:19.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:19.219 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:50:19.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.450+0000 7fd81dcfb640 1 Processor -- start 2023-08-25T09:50:19.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.451+0000 7fd81dcfb640 1 -- start start 2023-08-25T09:50:19.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.452+0000 7fd81dcfb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd8181542d0 0x7fd8181576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.452+0000 7fd81dcfb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd8180c4fb0 0x7fd8180c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.452+0000 7fd81dcfb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818157bf0 con 0x7fd818069850 2023-08-25T09:50:19.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.452+0000 7fd81dcfb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8180c4b60 con 0x7fd8181542d0 2023-08-25T09:50:19.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.452+0000 7fd81dcfb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81806a130 con 0x7fd8180c4fb0 2023-08-25T09:50:19.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd817fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd818069850 0x7fd8180c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41742/0 (socket says 172.21.15.71:41742) 2023-08-25T09:50:19.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd817fff640 1 -- 172.21.15.71:0/3434028753 learned_addr learned my addr 172.21.15.71:0/3434028753 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd81ccf9640 1 --2- 172.21.15.71:0/3434028753 >> v2:172.21.15.71:3300/0 conn(0x7fd8181542d0 0x7fd8181576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd81d4fa640 1 --2- 172.21.15.71:0/3434028753 >> v2:172.21.15.164:3300/0 conn(0x7fd8180c4fb0 0x7fd8180c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd81ccf9640 1 -- 172.21.15.71:0/3434028753 >> v2:172.21.15.164:3300/0 conn(0x7fd8180c4fb0 msgr2=0x7fd8180c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd81ccf9640 1 --2- 172.21.15.71:0/3434028753 >> v2:172.21.15.164:3300/0 conn(0x7fd8180c4fb0 0x7fd8180c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd81ccf9640 1 -- 172.21.15.71:0/3434028753 >> v1:172.21.15.71:6789/0 conn(0x7fd818069850 legacy=0x7fd8180c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd81ccf9640 1 -- 172.21.15.71:0/3434028753 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180c5980 con 0x7fd8181542d0 2023-08-25T09:50:19.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.453+0000 7fd8177fe640 1 -- 172.21.15.71:0/3434028753 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209781779 0 0) 0x7fd804001110 con 0x7fd818069850 2023-08-25T09:50:19.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.454+0000 7fd81ccf9640 1 --2- 172.21.15.71:0/3434028753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8181542d0 0x7fd8181576b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd8180c1800 tx=0x7fd80c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84efd19aa596fa10 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd8177fe640 1 -- 172.21.15.71:0/3434028753 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c031830 con 0x7fd8181542d0 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd8177fe640 1 -- 172.21.15.71:0/3434028753 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80c031990 con 0x7fd8181542d0 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd8177fe640 1 -- 172.21.15.71:0/3434028753 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c031d20 con 0x7fd8181542d0 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd81dcfb640 1 -- 172.21.15.71:0/3434028753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8181542d0 msgr2=0x7fd8181576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/3434028753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8181542d0 0x7fd8181576b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd8180c1800 tx=0x7fd80c02b8d0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd81dcfb640 1 -- 172.21.15.71:0/3434028753 shutdown_connections 2023-08-25T09:50:19.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/3434028753 >> v2:172.21.15.164:3300/0 conn(0x7fd8180c4fb0 0x7fd8180c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/3434028753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd8181542d0 0x7fd8181576b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.455+0000 7fd81dcfb640 1 -- 172.21.15.71:0/3434028753 >> 172.21.15.71:0/3434028753 conn(0x7fd81806c280 msgr2=0x7fd81806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:19.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.456+0000 7fd81dcfb640 1 -- 172.21.15.71:0/3434028753 shutdown_connections 2023-08-25T09:50:19.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.456+0000 7fd81dcfb640 1 -- 172.21.15.71:0/3434028753 wait complete. 2023-08-25T09:50:19.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.457+0000 7fd81dcfb640 1 Processor -- start 2023-08-25T09:50:19.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.457+0000 7fd81dcfb640 1 -- start start 2023-08-25T09:50:19.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.458+0000 7fd81dcfb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 0x7fd8181eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.458+0000 7fd81dcfb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8180c4fb0 0x7fd8180adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.458+0000 7fd81dcfb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8181542d0 0x7fd818151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.458+0000 7fd81dcfb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818152700 con 0x7fd818069850 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.458+0000 7fd81dcfb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818154b40 con 0x7fd8181542d0 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.458+0000 7fd81dcfb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd818154e20 con 0x7fd8180c4fb0 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 0x7fd8181eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 0x7fd8181eeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56660/0 (socket says 172.21.15.71:56660) 2023-08-25T09:50:19.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 -- 172.21.15.71:0/1855923966 learned_addr learned my addr 172.21.15.71:0/1855923966 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:19.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 -- 172.21.15.71:0/1855923966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8180c4fb0 msgr2=0x7fd8180adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8180c4fb0 0x7fd8180adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 -- 172.21.15.71:0/1855923966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8181542d0 msgr2=0x7fd818151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8181542d0 0x7fd818151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 -- 172.21.15.71:0/1855923966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c009400 con 0x7fd818069850 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.459+0000 7fd81ccf9640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 0x7fd8181eeb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd80c000c00 tx=0x7fd80c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.460+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c004230 con 0x7fd818069850 2023-08-25T09:50:19.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.460+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80c0043b0 con 0x7fd818069850 2023-08-25T09:50:19.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.460+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c004520 con 0x7fd818069850 2023-08-25T09:50:19.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.460+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd818155100 con 0x7fd818069850 2023-08-25T09:50:19.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.460+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8181556c0 con 0x7fd818069850 2023-08-25T09:50:19.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.461+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81804e420 con 0x7fd818069850 2023-08-25T09:50:19.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.462+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd80c03a8a0 con 0x7fd818069850 2023-08-25T09:50:19.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.463+0000 7fd8157fa640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd7fc06a050 0x7fd7fc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.463+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fd80c0aaed0 con 0x7fd818069850 2023-08-25T09:50:19.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.463+0000 7fd817fff640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd7fc06a050 0x7fd7fc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.464+0000 7fd817fff640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd7fc06a050 0x7fd7fc06c510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd804007fd0 tx=0x7fd804007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:19.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.467+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd80c076200 con 0x7fd818069850 2023-08-25T09:50:19.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.676+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd818069f10 con 0x7fd818069850 2023-08-25T09:50:19.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.679+0000 7fd8157fa640 1 -- 172.21.15.71:0/1855923966 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7fd80c07ae60 con 0x7fd818069850 2023-08-25T09:50:19.682 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:19.683 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd7fc06a050 msgr2=0x7fd7fc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd7fc06a050 0x7fd7fc06c510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd804007fd0 tx=0x7fd804007960 comp rx=0 tx=0).stop 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 msgr2=0x7fd8181eeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 0x7fd8181eeb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd80c000c00 tx=0x7fd80c0046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 shutdown_connections 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8181542d0 0x7fd818151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd7fc06a050 0x7fd7fc06c510 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd8180c4fb0 0x7fd8180adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 --2- 172.21.15.71:0/1855923966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd818069850 0x7fd8181eeb20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 >> 172.21.15.71:0/1855923966 conn(0x7fd81806c280 msgr2=0x7fd81814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:19.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 shutdown_connections 2023-08-25T09:50:19.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.682+0000 7fd81dcfb640 1 -- 172.21.15.71:0/1855923966 wait complete. 2023-08-25T09:50:19.702 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:50:19.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:50:19.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.909+0000 7fc359c0d640 1 Processor -- start 2023-08-25T09:50:19.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.910+0000 7fc359c0d640 1 -- start start 2023-08-25T09:50:19.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc359c0d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3540c4560 0x7fc35406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc359c0d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc354070cf0 0x7fc3541637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc359c0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35406eb60 con 0x7fc354152d80 2023-08-25T09:50:19.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc359c0d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35406ca30 con 0x7fc354070cf0 2023-08-25T09:50:19.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc359c0d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35406cf90 con 0x7fc3540c4560 2023-08-25T09:50:19.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc3537fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc354152d80 0x7fc3540c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41748/0 (socket says 172.21.15.71:41748) 2023-08-25T09:50:19.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.911+0000 7fc3537fe640 1 -- 172.21.15.71:0/1484668888 learned_addr learned my addr 172.21.15.71:0/1484668888 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:19.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc353fff640 1 --2- 172.21.15.71:0/1484668888 >> v2:172.21.15.71:3300/0 conn(0x7fc354070cf0 0x7fc3541637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc352ffd640 1 --2- 172.21.15.71:0/1484668888 >> v2:172.21.15.164:3300/0 conn(0x7fc3540c4560 0x7fc35406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc3527fc640 1 -- 172.21.15.71:0/1484668888 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629784440 0 0) 0x7fc35406eb60 con 0x7fc354152d80 2023-08-25T09:50:19.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc3527fc640 1 -- 172.21.15.71:0/1484668888 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc340003520 con 0x7fc354152d80 2023-08-25T09:50:19.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc353fff640 1 -- 172.21.15.71:0/1484668888 >> v2:172.21.15.164:3300/0 conn(0x7fc3540c4560 msgr2=0x7fc35406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:19.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc3527fc640 1 -- 172.21.15.71:0/1484668888 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677455993 0 0) 0x7fc340003520 con 0x7fc354152d80 2023-08-25T09:50:19.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc353fff640 1 --2- 172.21.15.71:0/1484668888 >> v2:172.21.15.164:3300/0 conn(0x7fc3540c4560 0x7fc35406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc353fff640 1 -- 172.21.15.71:0/1484668888 >> v1:172.21.15.71:6789/0 conn(0x7fc354152d80 legacy=0x7fc3540c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.912+0000 7fc353fff640 1 -- 172.21.15.71:0/1484668888 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354158fd0 con 0x7fc354070cf0 2023-08-25T09:50:19.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.914+0000 7fc353fff640 1 --2- 172.21.15.71:0/1484668888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc3541637b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc35406db40 tx=0x7fc34c02bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58058c7048798ae8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:19.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.914+0000 7fc3527fc640 1 -- 172.21.15.71:0/1484668888 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344001670 con 0x7fc354070cf0 2023-08-25T09:50:19.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.914+0000 7fc3527fc640 1 -- 172.21.15.71:0/1484668888 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c02c830 con 0x7fc354070cf0 2023-08-25T09:50:19.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.914+0000 7fc3527fc640 1 -- 172.21.15.71:0/1484668888 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c02cba0 con 0x7fc354070cf0 2023-08-25T09:50:19.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 -- 172.21.15.71:0/1484668888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 msgr2=0x7fc3541637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1484668888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc3541637b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc35406db40 tx=0x7fc34c02bd00 comp rx=0 tx=0).stop 2023-08-25T09:50:19.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 -- 172.21.15.71:0/1484668888 shutdown_connections 2023-08-25T09:50:19.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1484668888 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc3541637b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1484668888 >> v2:172.21.15.164:3300/0 conn(0x7fc3540c4560 0x7fc35406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 -- 172.21.15.71:0/1484668888 >> 172.21.15.71:0/1484668888 conn(0x7fc3540654f0 msgr2=0x7fc354065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:19.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 -- 172.21.15.71:0/1484668888 shutdown_connections 2023-08-25T09:50:19.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.915+0000 7fc359c0d640 1 -- 172.21.15.71:0/1484668888 wait complete. 2023-08-25T09:50:19.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.917+0000 7fc359c0d640 1 Processor -- start 2023-08-25T09:50:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.917+0000 7fc359c0d640 1 -- start start 2023-08-25T09:50:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc359c0d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc354155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc359c0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3540c4560 0x7fc3540c3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc359c0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc354152d80 0x7fc3540c2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc359c0d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc354155fc0 con 0x7fc354070cf0 2023-08-25T09:50:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc359c0d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3541542a0 con 0x7fc354152d80 2023-08-25T09:50:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc359c0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3540c20e0 con 0x7fc3540c4560 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc3537fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc354155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.918+0000 7fc3537fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc354155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56688/0 (socket says 172.21.15.71:56688) 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 -- 172.21.15.71:0/1390093035 learned_addr learned my addr 172.21.15.71:0/1390093035 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc353fff640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc354152d80 0x7fc3540c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 -- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3540c4560 msgr2=0x7fc3540c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3540c4560 0x7fc3540c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 -- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc354152d80 msgr2=0x7fc3540c2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:19.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc354152d80 0x7fc3540c2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:19.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 -- 172.21.15.71:0/1390093035 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc34c009a70 con 0x7fc354070cf0 2023-08-25T09:50:19.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc353fff640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc354152d80 0x7fc3540c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:19.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc3537fe640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc354155470 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c750 tx=0x7fc34400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344005ad0 con 0x7fc354070cf0 2023-08-25T09:50:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344005c30 con 0x7fc354070cf0 2023-08-25T09:50:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344007070 con 0x7fc354070cf0 2023-08-25T09:50:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.919+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc354155ce0 con 0x7fc354070cf0 2023-08-25T09:50:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.920+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3540c1a90 con 0x7fc354070cf0 2023-08-25T09:50:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.921+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc354069160 con 0x7fc354070cf0 2023-08-25T09:50:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.922+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc3440071d0 con 0x7fc354070cf0 2023-08-25T09:50:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.922+0000 7fc34bfff640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc33406a080 0x7fc33406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.923+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fc3440885a0 con 0x7fc354070cf0 2023-08-25T09:50:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.923+0000 7fc352ffd640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc33406a080 0x7fc33406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:19.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.924+0000 7fc352ffd640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc33406a080 0x7fc33406c540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc33c006a90 tx=0x7fc33c006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:19.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:19.927+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc3440538d0 con 0x7fc354070cf0 2023-08-25T09:50:20.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.115+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc35406c180 con 0x7fc354070cf0 2023-08-25T09:50:20.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.116+0000 7fc34bfff640 1 -- 172.21.15.71:0/1390093035 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc344056ef0 con 0x7fc354070cf0 2023-08-25T09:50:20.119 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:50:20.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc33406a080 msgr2=0x7fc33406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc33406a080 0x7fc33406c540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc33c006a90 tx=0x7fc33c006a20 comp rx=0 tx=0).stop 2023-08-25T09:50:20.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 msgr2=0x7fc354155470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc354155470 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc34400c750 tx=0x7fc34400cc10 comp rx=0 tx=0).stop 2023-08-25T09:50:20.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 shutdown_connections 2023-08-25T09:50:20.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc354152d80 0x7fc3540c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc33406a080 0x7fc33406c540 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3540c4560 0x7fc3540c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 --2- 172.21.15.71:0/1390093035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc354070cf0 0x7fc354155470 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 >> 172.21.15.71:0/1390093035 conn(0x7fc3540654f0 msgr2=0x7fc3540c81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:20.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 shutdown_connections 2023-08-25T09:50:20.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.119+0000 7fc359c0d640 1 -- 172.21.15.71:0/1390093035 wait complete. 2023-08-25T09:50:20.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:50:20.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.356+0000 7f2950d79640 1 Processor -- start 2023-08-25T09:50:20.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.357+0000 7f2950d79640 1 -- start start 2023-08-25T09:50:20.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.358+0000 7f2950d79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.358+0000 7f2950d79640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f294c1509d0 0x7f294c150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.358+0000 7f2950d79640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c0c1dc0 con 0x7f294c0c3a40 2023-08-25T09:50:20.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.358+0000 7f2950d79640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294c157a80 con 0x7f294c1509d0 2023-08-25T09:50:20.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.358+0000 7f2950d79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c1513a0 con 0x7f294c0c3660 2023-08-25T09:50:20.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.359+0000 7f294a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.359+0000 7f294ad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f294c1509d0 0x7f294c150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.359+0000 7f294a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c0c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:49148/0 (socket says 172.21.15.71:49148) 2023-08-25T09:50:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.359+0000 7f294a59c640 1 -- 172.21.15.71:0/1749115797 learned_addr learned my addr 172.21.15.71:0/1749115797 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.360+0000 7f294ad9d640 1 -- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 msgr2=0x7f294c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.360+0000 7f294ad9d640 1 --2- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.360+0000 7f294ad9d640 1 -- 172.21.15.71:0/1749115797 >> v1:172.21.15.164:6789/0 conn(0x7f294c0c3a40 legacy=0x7f294c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.360+0000 7f294ad9d640 1 -- 172.21.15.71:0/1749115797 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c156e00 con 0x7f294c1509d0 2023-08-25T09:50:20.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.361+0000 7f294ad9d640 1 --2- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c1509d0 0x7f294c150e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f294402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e94878df04d0cee3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:20.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.362+0000 7f2949d9b640 1 -- 172.21.15.71:0/1749115797 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294402cce0 con 0x7f294c1509d0 2023-08-25T09:50:20.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.362+0000 7f2949d9b640 1 -- 172.21.15.71:0/1749115797 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294402ce40 con 0x7f294c1509d0 2023-08-25T09:50:20.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.362+0000 7f2949d9b640 1 -- 172.21.15.71:0/1749115797 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944034660 con 0x7f294c1509d0 2023-08-25T09:50:20.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.362+0000 7f2950d79640 1 -- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c1509d0 msgr2=0x7f294c150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.362+0000 7f2950d79640 1 --2- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c1509d0 0x7f294c150e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f294402bf30 comp rx=0 tx=0).stop 2023-08-25T09:50:20.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.363+0000 7f2950d79640 1 -- 172.21.15.71:0/1749115797 shutdown_connections 2023-08-25T09:50:20.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.363+0000 7f2950d79640 1 --2- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c1509d0 0x7f294c150e60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.363+0000 7f2950d79640 1 --2- 172.21.15.71:0/1749115797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.363+0000 7f2950d79640 1 -- 172.21.15.71:0/1749115797 >> 172.21.15.71:0/1749115797 conn(0x7f294c0654f0 msgr2=0x7f294c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:20.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.363+0000 7f2950d79640 1 -- 172.21.15.71:0/1749115797 shutdown_connections 2023-08-25T09:50:20.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.363+0000 7f2950d79640 1 -- 172.21.15.71:0/1749115797 wait complete. 2023-08-25T09:50:20.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.364+0000 7f2950d79640 1 Processor -- start 2023-08-25T09:50:20.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.365+0000 7f2950d79640 1 -- start start 2023-08-25T09:50:20.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.365+0000 7f2950d79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.365+0000 7f2950d79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c0c3a40 0x7f294c14d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.365+0000 7f2950d79640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 0x7f294c14dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.365+0000 7f2950d79640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c14d0e0 con 0x7f294c1509d0 2023-08-25T09:50:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.365+0000 7f2950d79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c1ef960 con 0x7f294c0c3a40 2023-08-25T09:50:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f2950d79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c06e960 con 0x7f294c0c3660 2023-08-25T09:50:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 0x7f294c14dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 0x7f294c14dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50764/0 (socket says 172.21.15.71:50764) 2023-08-25T09:50:20.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 -- 172.21.15.71:0/2367154752 learned_addr learned my addr 172.21.15.71:0/2367154752 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:20.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f2943fff640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c0c3a40 0x7f294c14d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294a59c640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 -- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 msgr2=0x7f294c06f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c06f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 -- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c0c3a40 msgr2=0x7f294c14d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c0c3a40 0x7f294c14d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.366+0000 7f294ad9d640 1 -- 172.21.15.71:0/2367154752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2944009a70 con 0x7f294c1509d0 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.367+0000 7f294ad9d640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 0x7f294c14dab0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f2944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.367+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294403d600 con 0x7f294c1509d0 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.367+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294403d780 con 0x7f294c1509d0 2023-08-25T09:50:20.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.367+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c06ec00 con 0x7f294c1509d0 2023-08-25T09:50:20.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.367+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294403db30 con 0x7f294c1509d0 2023-08-25T09:50:20.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.367+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c06fb10 con 0x7f294c1509d0 2023-08-25T09:50:20.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.368+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c0c29c0 con 0x7f294c1509d0 2023-08-25T09:50:20.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.369+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f294403b040 con 0x7f294c1509d0 2023-08-25T09:50:20.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.370+0000 7f2942ffd640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f292c06a080 0x7f292c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.370+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f29440aad50 con 0x7f294c1509d0 2023-08-25T09:50:20.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.370+0000 7f294a59c640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f292c06a080 0x7f292c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.371+0000 7f294a59c640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f292c06a080 0x7f292c06c540 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f293c009ac0 tx=0x7f293c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:20.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.374+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f29440ae030 con 0x7f294c1509d0 2023-08-25T09:50:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.563+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f294c14cbd0 con 0x7f294c1509d0 2023-08-25T09:50:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.564+0000 7f2942ffd640 1 -- 172.21.15.71:0/2367154752 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2944039020 con 0x7f294c1509d0 2023-08-25T09:50:20.567 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:50:20.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f292c06a080 msgr2=0x7f292c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f292c06a080 0x7f292c06c540 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f293c009ac0 tx=0x7f293c008040 comp rx=0 tx=0).stop 2023-08-25T09:50:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 msgr2=0x7f294c14dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 0x7f294c14dab0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f2944004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 shutdown_connections 2023-08-25T09:50:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f294c0c3a40 0x7f294c14d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f292c06a080 0x7f292c06c540 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f294c1509d0 0x7f294c14dab0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 --2- 172.21.15.71:0/2367154752 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f294c0c3660 0x7f294c06f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 >> 172.21.15.71:0/2367154752 conn(0x7f294c0654f0 msgr2=0x7f294c0c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:20.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 shutdown_connections 2023-08-25T09:50:20.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.567+0000 7f2950d79640 1 -- 172.21.15.71:0/2367154752 wait complete. 2023-08-25T09:50:20.590 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:50:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.813+0000 7fdd933e1640 1 Processor -- start 2023-08-25T09:50:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.813+0000 7fdd933e1640 1 -- start start 2023-08-25T09:50:20.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.814+0000 7fdd933e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd933e1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdd8c070cf0 0x7fdd8c1637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd933e1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd8c06eb60 con 0x7fdd8c0c4560 2023-08-25T09:50:20.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd933e1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8c06ca30 con 0x7fdd8c070cf0 2023-08-25T09:50:20.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd933e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c06cf90 con 0x7fdd8c152d80 2023-08-25T09:50:20.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd9197e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdd8c070cf0 0x7fdd8c1637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd9197e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdd8c070cf0 0x7fdd8c1637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50772/0 (socket says 172.21.15.71:50772) 2023-08-25T09:50:20.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd9197e640 1 -- 172.21.15.71:0/2436563762 learned_addr learned my addr 172.21.15.71:0/2436563762 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:20.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.815+0000 7fdd9117d640 1 --2- 172.21.15.71:0/2436563762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.816+0000 7fdd9197e640 1 -- 172.21.15.71:0/2436563762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 msgr2=0x7fdd8c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:20.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.816+0000 7fdd9197e640 1 --2- 172.21.15.71:0/2436563762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c06e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.816+0000 7fdd9197e640 1 -- 172.21.15.71:0/2436563762 >> v1:172.21.15.164:6789/0 conn(0x7fdd8c0c4560 legacy=0x7fdd8c0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.816+0000 7fdd9197e640 1 -- 172.21.15.71:0/2436563762 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c158fd0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.817+0000 7fdd9197e640 1 --2- 172.21.15.71:0/2436563762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c1637b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009e50 tx=0x7fdd8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a744dfee988c7f8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:20.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.817+0000 7fdd83fff640 1 -- 172.21.15.71:0/2436563762 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8402ca70 con 0x7fdd8c070cf0 2023-08-25T09:50:20.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.817+0000 7fdd83fff640 1 -- 172.21.15.71:0/2436563762 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8402cbd0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.817+0000 7fdd83fff640 1 -- 172.21.15.71:0/2436563762 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd840343f0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 -- 172.21.15.71:0/2436563762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 msgr2=0x7fdd8c1637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 --2- 172.21.15.71:0/2436563762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c1637b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdd84009e50 tx=0x7fdd8402bed0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 -- 172.21.15.71:0/2436563762 shutdown_connections 2023-08-25T09:50:20.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 --2- 172.21.15.71:0/2436563762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c1637b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 --2- 172.21.15.71:0/2436563762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c06e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 -- 172.21.15.71:0/2436563762 >> 172.21.15.71:0/2436563762 conn(0x7fdd8c0654f0 msgr2=0x7fdd8c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:20.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 -- 172.21.15.71:0/2436563762 shutdown_connections 2023-08-25T09:50:20.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.818+0000 7fdd933e1640 1 -- 172.21.15.71:0/2436563762 wait complete. 2023-08-25T09:50:20.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.820+0000 7fdd933e1640 1 Processor -- start 2023-08-25T09:50:20.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.820+0000 7fdd933e1640 1 -- start start 2023-08-25T09:50:20.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.821+0000 7fdd933e1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.821+0000 7fdd933e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd8c0c4560 0x7fdd8c14ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.821+0000 7fdd933e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c14b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.821+0000 7fdd933e1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c0bc920 con 0x7fdd8c070cf0 2023-08-25T09:50:20.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.821+0000 7fdd933e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c15b7c0 con 0x7fdd8c0c4560 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.821+0000 7fdd933e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c1eebc0 con 0x7fdd8c152d80 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c15af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50784/0 (socket says 172.21.15.71:50784) 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 -- 172.21.15.71:0/1983290518 learned_addr learned my addr 172.21.15.71:0/1983290518 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 -- 172.21.15.71:0/1983290518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 msgr2=0x7fdd8c14b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c14b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 -- 172.21.15.71:0/1983290518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd8c0c4560 msgr2=0x7fdd8c14ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd8c0c4560 0x7fdd8c14ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 -- 172.21.15.71:0/1983290518 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84009b30 con 0x7fdd8c070cf0 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.822+0000 7fdd9117d640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c15af00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800e870 tx=0x7fdd8800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.823+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880094f0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.823+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8800c940 con 0x7fdd8c070cf0 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.823+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8800ccf0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.823+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c15ba60 con 0x7fdd8c070cf0 2023-08-25T09:50:20.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.823+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c1eeea0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.824+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c069160 con 0x7fdd8c070cf0 2023-08-25T09:50:20.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.825+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fdd88013470 con 0x7fdd8c070cf0 2023-08-25T09:50:20.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.826+0000 7fdd81ffb640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdd7006a050 0x7fdd7006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:20.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.826+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fdd88088aa0 con 0x7fdd8c070cf0 2023-08-25T09:50:20.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.826+0000 7fdd9097c640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdd7006a050 0x7fdd7006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:20.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.827+0000 7fdd9097c640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdd7006a050 0x7fdd7006c510 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c000f60 tx=0x7fdd7c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:20.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:20.831+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fdd88053d30 con 0x7fdd8c070cf0 2023-08-25T09:50:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.018+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd8c15d2c0 con 0x7fdd8c070cf0 2023-08-25T09:50:21.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.019+0000 7fdd81ffb640 1 -- 172.21.15.71:0/1983290518 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v31) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd88021020 con 0x7fdd8c070cf0 2023-08-25T09:50:21.022 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:21.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdd7006a050 msgr2=0x7fdd7006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdd7006a050 0x7fdd7006c510 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c000f60 tx=0x7fdd7c0079b0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 msgr2=0x7fdd8c15af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c15af00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800e870 tx=0x7fdd8800ed30 comp rx=0 tx=0).stop 2023-08-25T09:50:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 shutdown_connections 2023-08-25T09:50:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd8c0c4560 0x7fdd8c14ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdd7006a050 0x7fdd7006c510 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd8c152d80 0x7fdd8c14b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 --2- 172.21.15.71:0/1983290518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdd8c070cf0 0x7fdd8c15af00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 >> 172.21.15.71:0/1983290518 conn(0x7fdd8c0654f0 msgr2=0x7fdd8c150b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.021+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 shutdown_connections 2023-08-25T09:50:21.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.022+0000 7fdd933e1640 1 -- 172.21.15.71:0/1983290518 wait complete. 2023-08-25T09:50:21.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:50:21.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.255+0000 7fa14ee46640 1 Processor -- start 2023-08-25T09:50:21.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.255+0000 7fa14ee46640 1 -- start start 2023-08-25T09:50:21.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.257+0000 7fa14ee46640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa148153310 0x7fa14806a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.257+0000 7fa14ee46640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa1480c4510 0x7fa1480cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14ee46640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1481590c0 con 0x7fa1480cec00 2023-08-25T09:50:21.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14ee46640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa148159380 con 0x7fa1480c4510 2023-08-25T09:50:21.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14ee46640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148159660 con 0x7fa148153310 2023-08-25T09:50:21.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa1480c4510 0x7fa1480cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa1480c4510 0x7fa1480cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50798/0 (socket says 172.21.15.71:50798) 2023-08-25T09:50:21.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 -- 172.21.15.71:0/1936702023 learned_addr learned my addr 172.21.15.71:0/1936702023 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:21.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14cbe2640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa148153310 0x7fa14806a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 -- 172.21.15.71:0/1936702023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa148153310 msgr2=0x7fa14806a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:21.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa148153310 0x7fa14806a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 -- 172.21.15.71:0/1936702023 >> v1:172.21.15.164:6789/0 conn(0x7fa1480cec00 legacy=0x7fa1480c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:21.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14d3e3640 1 -- 172.21.15.71:0/1936702023 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148159940 con 0x7fa1480c4510 2023-08-25T09:50:21.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.258+0000 7fa14cbe2640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa148153310 0x7fa14806a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:21.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.259+0000 7fa14d3e3640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa1480cf020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa140009e50 tx=0x7fa14002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2673c75c5cb54d5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.260+0000 7fa1477fe640 1 -- 172.21.15.71:0/1936702023 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14002cb00 con 0x7fa1480c4510 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.260+0000 7fa1477fe640 1 -- 172.21.15.71:0/1936702023 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14002cc60 con 0x7fa1480c4510 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.260+0000 7fa1477fe640 1 -- 172.21.15.71:0/1936702023 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1400343f0 con 0x7fa1480c4510 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.260+0000 7fa14ee46640 1 -- 172.21.15.71:0/1936702023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 msgr2=0x7fa1480cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.260+0000 7fa14ee46640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa1480cf020 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa140009e50 tx=0x7fa14002bf60 comp rx=0 tx=0).stop 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.261+0000 7fa14ee46640 1 -- 172.21.15.71:0/1936702023 shutdown_connections 2023-08-25T09:50:21.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.261+0000 7fa14ee46640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa1480cf020 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.261+0000 7fa14ee46640 1 --2- 172.21.15.71:0/1936702023 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa148153310 0x7fa14806a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.261+0000 7fa14ee46640 1 -- 172.21.15.71:0/1936702023 >> 172.21.15.71:0/1936702023 conn(0x7fa148065910 msgr2=0x7fa148065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:21.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.261+0000 7fa14ee46640 1 -- 172.21.15.71:0/1936702023 shutdown_connections 2023-08-25T09:50:21.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.261+0000 7fa14ee46640 1 -- 172.21.15.71:0/1936702023 wait complete. 2023-08-25T09:50:21.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.262+0000 7fa14ee46640 1 Processor -- start 2023-08-25T09:50:21.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.263+0000 7fa14ee46640 1 -- start start 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.263+0000 7fa14ee46640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa148154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.263+0000 7fa14ee46640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1480cec00 0x7fa1480cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.263+0000 7fa14ee46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa148153310 0x7fa1481552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14ee46640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1481f2520 con 0x7fa1480c4510 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14ee46640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148154e20 con 0x7fa148153310 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14ee46640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1481f1fa0 con 0x7fa1480cec00 2023-08-25T09:50:21.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa148154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa148154560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50808/0 (socket says 172.21.15.71:50808) 2023-08-25T09:50:21.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 -- 172.21.15.71:0/3046710098 learned_addr learned my addr 172.21.15.71:0/3046710098 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:21.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14d3e3640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa148153310 0x7fa1481552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa147fff640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1480cec00 0x7fa1480cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 -- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1480cec00 msgr2=0x7fa1480cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1480cec00 0x7fa1480cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 -- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa148153310 msgr2=0x7fa1481552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa148153310 0x7fa1481552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14cbe2640 1 -- 172.21.15.71:0/3046710098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa140009b30 con 0x7fa1480c4510 2023-08-25T09:50:21.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa14d3e3640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa148153310 0x7fa1481552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:21.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.264+0000 7fa147fff640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1480cec00 0x7fa1480cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:21.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.265+0000 7fa14cbe2640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa148154560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00c7f0 tx=0x7fa13c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:21.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.265+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c007a30 con 0x7fa1480c4510 2023-08-25T09:50:21.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.265+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c007b90 con 0x7fa1480c4510 2023-08-25T09:50:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.265+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00ce40 con 0x7fa1480c4510 2023-08-25T09:50:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.265+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1481f2240 con 0x7fa1480c4510 2023-08-25T09:50:21.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.265+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1480ce330 con 0x7fa1480c4510 2023-08-25T09:50:21.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.266+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1480cdeb0 con 0x7fa1480c4510 2023-08-25T09:50:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.267+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa13c0113f0 con 0x7fa1480c4510 2023-08-25T09:50:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.268+0000 7fa1457fa640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa12c06a080 0x7fa12c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.268+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa13c088a70 con 0x7fa1480c4510 2023-08-25T09:50:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.268+0000 7fa147fff640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa12c06a080 0x7fa12c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.269+0000 7fa147fff640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa12c06a080 0x7fa12c06c540 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa148149820 tx=0x7fa134009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:21.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.273+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa1480cdeb0 con 0x7fa1480c4510 2023-08-25T09:50:21.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.464+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1480cdeb0 con 0x7fa1480c4510 2023-08-25T09:50:21.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.465+0000 7fa1457fa640 1 -- 172.21.15.71:0/3046710098 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v31) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1480cdeb0 con 0x7fa1480c4510 2023-08-25T09:50:21.468 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:50:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa12c06a080 msgr2=0x7fa12c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa12c06a080 0x7fa12c06c540 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa148149820 tx=0x7fa134009040 comp rx=0 tx=0).stop 2023-08-25T09:50:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 msgr2=0x7fa148154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa148154560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00c7f0 tx=0x7fa13c00cbf0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 shutdown_connections 2023-08-25T09:50:21.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa148153310 0x7fa1481552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa12c06a080 0x7fa12c06c540 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1480cec00 0x7fa1480cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 --2- 172.21.15.71:0/3046710098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1480c4510 0x7fa148154560 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 >> 172.21.15.71:0/3046710098 conn(0x7fa148065910 msgr2=0x7fa14806a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:21.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.468+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 shutdown_connections 2023-08-25T09:50:21.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.469+0000 7fa14ee46640 1 -- 172.21.15.71:0/3046710098 wait complete. 2023-08-25T09:50:21.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:50:21.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.699+0000 7f5292f5d640 1 Processor -- start 2023-08-25T09:50:21.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.699+0000 7f5292f5d640 1 -- start start 2023-08-25T09:50:21.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f5292f5d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f528c159fc0 0x7f528c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f5292f5d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c06a960 con 0x7f528c1594d0 2023-08-25T09:50:21.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f5292f5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c06ac20 con 0x7f528c06cc70 2023-08-25T09:50:21.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f5292f5d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528c06af00 con 0x7f528c159fc0 2023-08-25T09:50:21.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f5290cf9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f528c1594d0 0x7f528c14f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36788/0 (socket says 172.21.15.71:36788) 2023-08-25T09:50:21.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f5290cf9640 1 -- 172.21.15.71:0/2180608247 learned_addr learned my addr 172.21.15.71:0/2180608247 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:21.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.701+0000 7f52914fa640 1 --2- 172.21.15.71:0/2180608247 >> v2:172.21.15.71:3300/0 conn(0x7f528c159fc0 0x7f528c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.702+0000 7f528b7fe640 1 -- 172.21.15.71:0/2180608247 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437250719 0 0) 0x7f528c06a960 con 0x7f528c1594d0 2023-08-25T09:50:21.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.702+0000 7f528b7fe640 1 -- 172.21.15.71:0/2180608247 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527c003520 con 0x7f528c1594d0 2023-08-25T09:50:21.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.702+0000 7f52914fa640 1 -- 172.21.15.71:0/2180608247 >> v1:172.21.15.164:6789/0 conn(0x7f528c06cc70 legacy=0x7f528c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.702+0000 7f52914fa640 1 -- 172.21.15.71:0/2180608247 >> v1:172.21.15.71:6789/0 conn(0x7f528c1594d0 legacy=0x7f528c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.702+0000 7f52914fa640 1 -- 172.21.15.71:0/2180608247 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c0c2e60 con 0x7f528c159fc0 2023-08-25T09:50:21.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.703+0000 7f52914fa640 1 --2- 172.21.15.71:0/2180608247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c159fc0 0x7f528c0c4c90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5284009d90 tx=0x7f528402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec782e73daa17b5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:21.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.703+0000 7f528b7fe640 1 -- 172.21.15.71:0/2180608247 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528402cd80 con 0x7f528c159fc0 2023-08-25T09:50:21.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.703+0000 7f528b7fe640 1 -- 172.21.15.71:0/2180608247 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5284034440 con 0x7f528c159fc0 2023-08-25T09:50:21.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.703+0000 7f528b7fe640 1 -- 172.21.15.71:0/2180608247 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52840347d0 con 0x7f528c159fc0 2023-08-25T09:50:21.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 -- 172.21.15.71:0/2180608247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c159fc0 msgr2=0x7f528c0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 --2- 172.21.15.71:0/2180608247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c159fc0 0x7f528c0c4c90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5284009d90 tx=0x7f528402bfd0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 -- 172.21.15.71:0/2180608247 shutdown_connections 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 --2- 172.21.15.71:0/2180608247 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c159fc0 0x7f528c0c4c90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 -- 172.21.15.71:0/2180608247 >> 172.21.15.71:0/2180608247 conn(0x7f528c0654f0 msgr2=0x7f528c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 -- 172.21.15.71:0/2180608247 shutdown_connections 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.704+0000 7f5292f5d640 1 -- 172.21.15.71:0/2180608247 wait complete. 2023-08-25T09:50:21.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.706+0000 7f5292f5d640 1 Processor -- start 2023-08-25T09:50:21.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.706+0000 7f5292f5d640 1 -- start start 2023-08-25T09:50:21.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5292f5d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 0x7f528c14aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5292f5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f528c1594d0 0x7f528c14afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5292f5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f528c158630 0x7f528c1f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5292f5d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c158ba0 con 0x7f528c06cc70 2023-08-25T09:50:21.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5292f5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c158e60 con 0x7f528c158630 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5292f5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c149f90 con 0x7f528c1594d0 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 0x7f528c14aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 0x7f528c14aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50834/0 (socket says 172.21.15.71:50834) 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 -- 172.21.15.71:0/3220212668 learned_addr learned my addr 172.21.15.71:0/3220212668 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 -- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f528c1594d0 msgr2=0x7f528c14afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f52914fa640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f528c158630 0x7f528c1f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f528c1594d0 0x7f528c14afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 -- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f528c158630 msgr2=0x7f528c1f18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f528c158630 0x7f528c1f18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.707+0000 7f5290cf9640 1 -- 172.21.15.71:0/3220212668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5284009a70 con 0x7f528c06cc70 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.708+0000 7f5290cf9640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 0x7f528c14aa70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f528000c500 tx=0x7f528000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.708+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280005ce0 con 0x7f528c06cc70 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.708+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280005e40 con 0x7f528c06cc70 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.708+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280007330 con 0x7f528c06cc70 2023-08-25T09:50:21.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.708+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c14a270 con 0x7f528c06cc70 2023-08-25T09:50:21.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.708+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c14d540 con 0x7f528c06cc70 2023-08-25T09:50:21.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.709+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528c14a790 con 0x7f528c06cc70 2023-08-25T09:50:21.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.710+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5280007550 con 0x7f528c06cc70 2023-08-25T09:50:21.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.712+0000 7f52897fa640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f527006a050 0x7f527006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:21.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.712+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5280087ff0 con 0x7f528c06cc70 2023-08-25T09:50:21.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.712+0000 7f528bfff640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f527006a050 0x7f527006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:21.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.713+0000 7f528bfff640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f527006a050 0x7f527006c510 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5278000f60 tx=0x7f52780079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:21.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.716+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f528c14a790 con 0x7f528c06cc70 2023-08-25T09:50:21.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.906+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f528c14a790 con 0x7f528c06cc70 2023-08-25T09:50:21.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.907+0000 7f52897fa640 1 -- 172.21.15.71:0/3220212668 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) v1 ==== 103+0+11 (secure 0 0 0) 0x7f528c14a790 con 0x7f528c06cc70 2023-08-25T09:50:21.910 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:21.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f527006a050 msgr2=0x7f527006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f527006a050 0x7f527006c510 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5278000f60 tx=0x7f52780079b0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 msgr2=0x7f528c14aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:21.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 0x7f528c14aa70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f528000c500 tx=0x7f528000c9c0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 shutdown_connections 2023-08-25T09:50:21.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f528c158630 0x7f528c1f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f527006a050 0x7f527006c510 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f528c1594d0 0x7f528c14afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 --2- 172.21.15.71:0/3220212668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f528c06cc70 0x7f528c14aa70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:21.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.909+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 >> 172.21.15.71:0/3220212668 conn(0x7f528c0654f0 msgr2=0x7f528c148260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:21.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.910+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 shutdown_connections 2023-08-25T09:50:21.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:21.910+0000 7f5292f5d640 1 -- 172.21.15.71:0/3220212668 wait complete. 2023-08-25T09:50:21.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:50:22.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.145+0000 7f22ced07640 1 Processor -- start 2023-08-25T09:50:22.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.145+0000 7f22ced07640 1 -- start start 2023-08-25T09:50:22.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ced07640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f22c8066010 0x7f22c80c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ced07640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f22c8070cf0 0x7f22c80c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ced07640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c815db40 con 0x7f22c8157450 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ced07640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c815d230 con 0x7f22c8066010 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ced07640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c806cb40 con 0x7f22c8070cf0 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ccaa3640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f22c8157450 0x7f22c806e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36794/0 (socket says 172.21.15.71:36794) 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.147+0000 7f22ccaa3640 1 -- 172.21.15.71:0/877979670 learned_addr learned my addr 172.21.15.71:0/877979670 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c7fff640 1 --2- 172.21.15.71:0/877979670 >> v2:172.21.15.71:3300/0 conn(0x7f22c8066010 0x7f22c80c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22cd2a4640 1 --2- 172.21.15.71:0/877979670 >> v2:172.21.15.164:3300/0 conn(0x7f22c8070cf0 0x7f22c80c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c77fe640 1 -- 172.21.15.71:0/877979670 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780763938 0 0) 0x7f22c815db40 con 0x7f22c8157450 2023-08-25T09:50:22.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c77fe640 1 -- 172.21.15.71:0/877979670 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22b8003520 con 0x7f22c8157450 2023-08-25T09:50:22.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c7fff640 1 -- 172.21.15.71:0/877979670 >> v2:172.21.15.164:3300/0 conn(0x7f22c8070cf0 msgr2=0x7f22c80c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:22.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c7fff640 1 --2- 172.21.15.71:0/877979670 >> v2:172.21.15.164:3300/0 conn(0x7f22c8070cf0 0x7f22c80c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c7fff640 1 -- 172.21.15.71:0/877979670 >> v1:172.21.15.71:6789/0 conn(0x7f22c8157450 legacy=0x7f22c806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.148+0000 7f22c7fff640 1 -- 172.21.15.71:0/877979670 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c806ce20 con 0x7f22c8066010 2023-08-25T09:50:22.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.149+0000 7f22c7fff640 1 --2- 172.21.15.71:0/877979670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8066010 0x7f22c80c56a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f22c806dbe0 tx=0x7f22b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e778980f9eafa28c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:22.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.149+0000 7f22c77fe640 1 -- 172.21.15.71:0/877979670 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4030610 con 0x7f22c8066010 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.149+0000 7f22c77fe640 1 -- 172.21.15.71:0/877979670 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b4030770 con 0x7f22c8066010 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.149+0000 7f22c77fe640 1 -- 172.21.15.71:0/877979670 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4030ae0 con 0x7f22c8066010 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 -- 172.21.15.71:0/877979670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8066010 msgr2=0x7f22c80c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 --2- 172.21.15.71:0/877979670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8066010 0x7f22c80c56a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f22c806dbe0 tx=0x7f22b402b960 comp rx=0 tx=0).stop 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 -- 172.21.15.71:0/877979670 shutdown_connections 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 --2- 172.21.15.71:0/877979670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8066010 0x7f22c80c56a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 --2- 172.21.15.71:0/877979670 >> v2:172.21.15.164:3300/0 conn(0x7f22c8070cf0 0x7f22c80c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 -- 172.21.15.71:0/877979670 >> 172.21.15.71:0/877979670 conn(0x7f22c80654f0 msgr2=0x7f22c8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:22.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 -- 172.21.15.71:0/877979670 shutdown_connections 2023-08-25T09:50:22.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.150+0000 7f22ced07640 1 -- 172.21.15.71:0/877979670 wait complete. 2023-08-25T09:50:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.152+0000 7f22ced07640 1 Processor -- start 2023-08-25T09:50:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.152+0000 7f22ced07640 1 -- start start 2023-08-25T09:50:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.153+0000 7f22ced07640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22c8066010 0x7f22c80c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.153+0000 7f22ced07640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c8070cf0 0x7f22c8150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.153+0000 7f22ced07640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 0x7f22c8150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.153+0000 7f22ced07640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80c9e40 con 0x7f22c8157450 2023-08-25T09:50:22.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.153+0000 7f22ced07640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c81500b0 con 0x7f22c8066010 2023-08-25T09:50:22.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.153+0000 7f22ced07640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c8151fe0 con 0x7f22c8070cf0 2023-08-25T09:50:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 0x7f22c8150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 0x7f22c8150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50852/0 (socket says 172.21.15.71:50852) 2023-08-25T09:50:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 -- 172.21.15.71:0/3978351837 learned_addr learned my addr 172.21.15.71:0/3978351837 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22c7fff640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c8070cf0 0x7f22c8150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22ccaa3640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22c8066010 0x7f22c80c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 -- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c8070cf0 msgr2=0x7f22c8150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c8070cf0 0x7f22c8150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 -- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22c8066010 msgr2=0x7f22c80c28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22c8066010 0x7f22c80c28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 -- 172.21.15.71:0/3978351837 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b4009400 con 0x7f22c8157450 2023-08-25T09:50:22.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.154+0000 7f22cd2a4640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 0x7f22c8150a80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f22c0007e70 tx=0x7f22c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.155+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0014620 con 0x7f22c8157450 2023-08-25T09:50:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.155+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c00147a0 con 0x7f22c8157450 2023-08-25T09:50:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.155+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c80c9b60 con 0x7f22c8157450 2023-08-25T09:50:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.155+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0014b50 con 0x7f22c8157450 2023-08-25T09:50:22.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.155+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c80c8ef0 con 0x7f22c8157450 2023-08-25T09:50:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.156+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c815dfb0 con 0x7f22c8157450 2023-08-25T09:50:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.157+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f22c0014cb0 con 0x7f22c8157450 2023-08-25T09:50:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.158+0000 7f22c57fa640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f22ac06a080 0x7f22ac06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.158+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f22c0089330 con 0x7f22c8157450 2023-08-25T09:50:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.158+0000 7f22ccaa3640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f22ac06a080 0x7f22ac06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.159+0000 7f22ccaa3640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f22ac06a080 0x7f22ac06c540 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f22bc005880 tx=0x7f22bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:22.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.162+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f22c815dfb0 con 0x7f22c8157450 2023-08-25T09:50:22.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.386+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22c8150260 con 0x7f22c8157450 2023-08-25T09:50:22.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.387+0000 7f22c57fa640 1 -- 172.21.15.71:0/3978351837 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f22c8150260 con 0x7f22c8157450 2023-08-25T09:50:22.390 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:22.390 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:50:17.728382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:50:22.391 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 16 2023-08-25T09:50:22.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f22ac06a080 msgr2=0x7f22ac06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f22ac06a080 0x7f22ac06c540 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f22bc005880 tx=0x7f22bc009040 comp rx=0 tx=0).stop 2023-08-25T09:50:22.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 msgr2=0x7f22c8150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 0x7f22c8150a80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f22c0007e70 tx=0x7f22c000e4b0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 shutdown_connections 2023-08-25T09:50:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f22ac06a080 0x7f22ac06c540 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22c8157450 0x7f22c8150a80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22c8070cf0 0x7f22c8150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 --2- 172.21.15.71:0/3978351837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22c8066010 0x7f22c80c28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.390+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 >> 172.21.15.71:0/3978351837 conn(0x7f22c80654f0 msgr2=0x7f22c81568b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.391+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 shutdown_connections 2023-08-25T09:50:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.391+0000 7f22ced07640 1 -- 172.21.15.71:0/3978351837 wait complete. 2023-08-25T09:50:22.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:22.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.621+0000 7efcefc06640 1 Processor -- start 2023-08-25T09:50:22.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.621+0000 7efcefc06640 1 -- start start 2023-08-25T09:50:22.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efcefc06640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce80c3660 0x7efce80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efcefc06640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efce80c3a40 0x7efce806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efcefc06640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efce81509d0 0x7efce8150e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efcefc06640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce8157a80 con 0x7efce80c3a40 2023-08-25T09:50:22.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efcefc06640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce80c1dc0 con 0x7efce81509d0 2023-08-25T09:50:22.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efcefc06640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce8151380 con 0x7efce80c3660 2023-08-25T09:50:22.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efced1a1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efce80c3a40 0x7efce806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efced1a1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efce80c3a40 0x7efce806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50860/0 (socket says 172.21.15.71:50860) 2023-08-25T09:50:22.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.623+0000 7efced1a1640 1 -- 172.21.15.71:0/3822283426 learned_addr learned my addr 172.21.15.71:0/3822283426 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:22.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced9a2640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce80c3660 0x7efce80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efcee1a3640 1 --2- 172.21.15.71:0/3822283426 >> v2:172.21.15.164:3300/0 conn(0x7efce81509d0 0x7efce8150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced1a1640 1 -- 172.21.15.71:0/3822283426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce80c3660 msgr2=0x7efce80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:22.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced1a1640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce80c3660 0x7efce80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced1a1640 1 -- 172.21.15.71:0/3822283426 >> v2:172.21.15.164:3300/0 conn(0x7efce81509d0 msgr2=0x7efce8150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced1a1640 1 --2- 172.21.15.71:0/3822283426 >> v2:172.21.15.164:3300/0 conn(0x7efce81509d0 0x7efce8150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced1a1640 1 -- 172.21.15.71:0/3822283426 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce806cf70 con 0x7efce80c3a40 2023-08-25T09:50:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efced9a2640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce80c3660 0x7efce80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.624+0000 7efcee1a3640 1 --2- 172.21.15.71:0/3822283426 >> v2:172.21.15.164:3300/0 conn(0x7efce81509d0 0x7efce8150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:22.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.625+0000 7efced1a1640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce806ca30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efce80c4090 tx=0x7efcd802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e238df047eb765d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:22.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.626+0000 7efcec9a0640 1 -- 172.21.15.71:0/3822283426 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8030850 con 0x7efce80c3a40 2023-08-25T09:50:22.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.626+0000 7efcec9a0640 1 -- 172.21.15.71:0/3822283426 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd80309b0 con 0x7efce80c3a40 2023-08-25T09:50:22.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.626+0000 7efcec9a0640 1 -- 172.21.15.71:0/3822283426 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8030d40 con 0x7efce80c3a40 2023-08-25T09:50:22.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.626+0000 7efcefc06640 1 -- 172.21.15.71:0/3822283426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 msgr2=0x7efce806ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.626+0000 7efcefc06640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce806ca30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efce80c4090 tx=0x7efcd802b990 comp rx=0 tx=0).stop 2023-08-25T09:50:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 -- 172.21.15.71:0/3822283426 shutdown_connections 2023-08-25T09:50:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 --2- 172.21.15.71:0/3822283426 >> v2:172.21.15.164:3300/0 conn(0x7efce81509d0 0x7efce8150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce806ca30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 --2- 172.21.15.71:0/3822283426 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce80c3660 0x7efce80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 -- 172.21.15.71:0/3822283426 >> 172.21.15.71:0/3822283426 conn(0x7efce80654f0 msgr2=0x7efce8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:22.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 -- 172.21.15.71:0/3822283426 shutdown_connections 2023-08-25T09:50:22.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.627+0000 7efcefc06640 1 -- 172.21.15.71:0/3822283426 wait complete. 2023-08-25T09:50:22.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.628+0000 7efcefc06640 1 Processor -- start 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.629+0000 7efcefc06640 1 -- start start 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.629+0000 7efcefc06640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efce80c3660 0x7efce80c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.629+0000 7efcefc06640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce8152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.629+0000 7efcefc06640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce81509d0 0x7efce8158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.629+0000 7efcefc06640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce80c5ce0 con 0x7efce80c3a40 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.629+0000 7efcefc06640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce8156a70 con 0x7efce80c3660 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efcefc06640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce81f04b0 con 0x7efce81509d0 2023-08-25T09:50:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce8152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce8152730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50870/0 (socket says 172.21.15.71:50870) 2023-08-25T09:50:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 -- 172.21.15.71:0/717749575 learned_addr learned my addr 172.21.15.71:0/717749575 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced9a2640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efce80c3660 0x7efce80c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 -- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce81509d0 msgr2=0x7efce8158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce81509d0 0x7efce8158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 -- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efce80c3660 msgr2=0x7efce80c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efce80c3660 0x7efce80c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.630+0000 7efced1a1640 1 -- 172.21.15.71:0/717749575 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd80094d0 con 0x7efce80c3a40 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.631+0000 7efced1a1640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce8152730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efcd802b9d0 tx=0x7efcd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.631+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8030840 con 0x7efce80c3a40 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.631+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd8039d90 con 0x7efce80c3a40 2023-08-25T09:50:22.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.631+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce8156790 con 0x7efce80c3a40 2023-08-25T09:50:22.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.631+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8043650 con 0x7efce80c3a40 2023-08-25T09:50:22.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.631+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce8152210 con 0x7efce80c3a40 2023-08-25T09:50:22.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.633+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce80c29c0 con 0x7efce80c3a40 2023-08-25T09:50:22.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.634+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efcd802e070 con 0x7efce80c3a40 2023-08-25T09:50:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.635+0000 7efcde7fc640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efccc06a050 0x7efccc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.635+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7efcd80aaa80 con 0x7efce80c3a40 2023-08-25T09:50:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.635+0000 7efced9a2640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efccc06a050 0x7efccc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.636+0000 7efced9a2640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efccc06a050 0x7efccc06c510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efce400a950 tx=0x7efce4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:22.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.638+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efcd8075e30 con 0x7efce80c3a40 2023-08-25T09:50:22.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.842+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efce8069160 con 0x7efce80c3a40 2023-08-25T09:50:22.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.845+0000 7efcde7fc640 1 -- 172.21.15.71:0/717749575 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7efcd8079450 con 0x7efce80c3a40 2023-08-25T09:50:22.849 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:22.850 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:22.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efccc06a050 msgr2=0x7efccc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efccc06a050 0x7efccc06c510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efce400a950 tx=0x7efce4008040 comp rx=0 tx=0).stop 2023-08-25T09:50:22.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 msgr2=0x7efce8152730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:22.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce8152730 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efcd802b9d0 tx=0x7efcd8004730 comp rx=0 tx=0).stop 2023-08-25T09:50:22.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 shutdown_connections 2023-08-25T09:50:22.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efccc06a050 0x7efccc06c510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efce81509d0 0x7efce8158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efce80c3a40 0x7efce8152730 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 --2- 172.21.15.71:0/717749575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efce80c3660 0x7efce80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:22.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.848+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 >> 172.21.15.71:0/717749575 conn(0x7efce80654f0 msgr2=0x7efce80c3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:22.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.849+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 shutdown_connections 2023-08-25T09:50:22.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:22.849+0000 7efcefc06640 1 -- 172.21.15.71:0/717749575 wait complete. 2023-08-25T09:50:22.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:50:22.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:50:23.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.081+0000 7f1354942640 1 Processor -- start 2023-08-25T09:50:23.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.081+0000 7f1354942640 1 -- start start 2023-08-25T09:50:23.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.083+0000 7f1354942640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1350153260 0x7f13500c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.083+0000 7f1354942640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 0x7f13501637a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.083+0000 7f1354942640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13500c0d90 con 0x7f13500c0050 2023-08-25T09:50:23.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.083+0000 7f1354942640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f135006ae80 con 0x7f1350153260 2023-08-25T09:50:23.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.083+0000 7f1354942640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13500c07f0 con 0x7f135015cb20 2023-08-25T09:50:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.084+0000 7f134ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 0x7f13501637a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.084+0000 7f134e59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1350153260 0x7f13500c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.084+0000 7f134ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 0x7f13501637a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:49272/0 (socket says 172.21.15.71:49272) 2023-08-25T09:50:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.084+0000 7f134ed9d640 1 -- 172.21.15.71:0/1282138075 learned_addr learned my addr 172.21.15.71:0/1282138075 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.084+0000 7f134ed9d640 1 -- 172.21.15.71:0/1282138075 >> v2:172.21.15.164:3300/0 conn(0x7f1350153260 msgr2=0x7f13500c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:23.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.085+0000 7f134ed9d640 1 --2- 172.21.15.71:0/1282138075 >> v2:172.21.15.164:3300/0 conn(0x7f1350153260 0x7f13500c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.085+0000 7f134ed9d640 1 -- 172.21.15.71:0/1282138075 >> v1:172.21.15.164:6789/0 conn(0x7f13500c0050 legacy=0x7f135015c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.085+0000 7f134ed9d640 1 -- 172.21.15.71:0/1282138075 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135015cfb0 con 0x7f135015cb20 2023-08-25T09:50:23.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.085+0000 7f134d59a640 1 -- 172.21.15.71:0/1282138075 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221637356 0 0) 0x7f1344001110 con 0x7f13500c0050 2023-08-25T09:50:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.086+0000 7f134ed9d640 1 --2- 172.21.15.71:0/1282138075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 0x7f13501637a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1348009e50 tx=0x7f134802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac660e060eb9192 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.086+0000 7f134d59a640 1 -- 172.21.15.71:0/1282138075 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134802cb00 con 0x7f135015cb20 2023-08-25T09:50:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.086+0000 7f134d59a640 1 -- 172.21.15.71:0/1282138075 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134802cc60 con 0x7f135015cb20 2023-08-25T09:50:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.086+0000 7f134d59a640 1 -- 172.21.15.71:0/1282138075 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13480343f0 con 0x7f135015cb20 2023-08-25T09:50:23.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 -- 172.21.15.71:0/1282138075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 msgr2=0x7f13501637a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 --2- 172.21.15.71:0/1282138075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 0x7f13501637a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1348009e50 tx=0x7f134802bf60 comp rx=0 tx=0).stop 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 -- 172.21.15.71:0/1282138075 shutdown_connections 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 --2- 172.21.15.71:0/1282138075 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f135015cb20 0x7f13501637a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 --2- 172.21.15.71:0/1282138075 >> v2:172.21.15.164:3300/0 conn(0x7f1350153260 0x7f13500c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 -- 172.21.15.71:0/1282138075 >> 172.21.15.71:0/1282138075 conn(0x7f135006d350 msgr2=0x7f13500c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 -- 172.21.15.71:0/1282138075 shutdown_connections 2023-08-25T09:50:23.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.087+0000 7f1354942640 1 -- 172.21.15.71:0/1282138075 wait complete. 2023-08-25T09:50:23.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.089+0000 7f1354942640 1 Processor -- start 2023-08-25T09:50:23.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.089+0000 7f1354942640 1 -- start start 2023-08-25T09:50:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f1354942640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13500c0050 0x7f13501e4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f1354942640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1350153260 0x7f135006f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f1354942640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 0x7f1350071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f1354942640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1350070340 con 0x7f1350071090 2023-08-25T09:50:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f1354942640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13500705d0 con 0x7f1350153260 2023-08-25T09:50:23.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f1354942640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13500708b0 con 0x7f13500c0050 2023-08-25T09:50:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f134ed9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 0x7f1350071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f134ed9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 0x7f1350071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50894/0 (socket says 172.21.15.71:50894) 2023-08-25T09:50:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.090+0000 7f134ed9d640 1 -- 172.21.15.71:0/1605639242 learned_addr learned my addr 172.21.15.71:0/1605639242 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134e59c640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13500c0050 0x7f13501e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134dd9b640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1350153260 0x7f135006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134ed9d640 1 -- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13500c0050 msgr2=0x7f13501e4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134ed9d640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13500c0050 0x7f13501e4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134ed9d640 1 -- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1350153260 msgr2=0x7f135006f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134ed9d640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1350153260 0x7f135006f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134ed9d640 1 -- 172.21.15.71:0/1605639242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348009b30 con 0x7f1350071090 2023-08-25T09:50:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.091+0000 7f134ed9d640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 0x7f1350071520 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1348009e50 tx=0x7f1348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.092+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1348009510 con 0x7f1350071090 2023-08-25T09:50:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.092+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134803e450 con 0x7f1350071090 2023-08-25T09:50:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.092+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134803e800 con 0x7f1350071090 2023-08-25T09:50:23.101 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.092+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13501e4050 con 0x7f1350071090 2023-08-25T09:50:23.101 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.092+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13500bfd60 con 0x7f1350071090 2023-08-25T09:50:23.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.093+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13501e4510 con 0x7f1350071090 2023-08-25T09:50:23.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.094+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f134803e960 con 0x7f1350071090 2023-08-25T09:50:23.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.095+0000 7f133effd640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f133006a050 0x7f133006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.095+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f13480ab680 con 0x7f1350071090 2023-08-25T09:50:23.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.095+0000 7f134e59c640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f133006a050 0x7f133006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.096+0000 7f134e59c640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f133006a050 0x7f133006c510 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1340009ad0 tx=0x7f1340008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:23.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.099+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f13480769a0 con 0x7f1350071090 2023-08-25T09:50:23.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.290+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1350069160 con 0x7f1350071090 2023-08-25T09:50:23.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.291+0000 7f133effd640 1 -- 172.21.15.71:0/1605639242 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1348079fc0 con 0x7f1350071090 2023-08-25T09:50:23.294 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:50:23.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f133006a050 msgr2=0x7f133006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f133006a050 0x7f133006c510 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1340009ad0 tx=0x7f1340008040 comp rx=0 tx=0).stop 2023-08-25T09:50:23.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 msgr2=0x7f1350071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 0x7f1350071520 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1348009e50 tx=0x7f1348004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:23.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 shutdown_connections 2023-08-25T09:50:23.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1350153260 0x7f135006f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f133006a050 0x7f133006c510 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1350071090 0x7f1350071520 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 --2- 172.21.15.71:0/1605639242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13500c0050 0x7f13501e4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 >> 172.21.15.71:0/1605639242 conn(0x7f135006d350 msgr2=0x7f135015c200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:23.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 shutdown_connections 2023-08-25T09:50:23.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.294+0000 7f1354942640 1 -- 172.21.15.71:0/1605639242 wait complete. 2023-08-25T09:50:23.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:50:23.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.591+0000 7f522310c640 1 Processor -- start 2023-08-25T09:50:23.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.591+0000 7f522310c640 1 -- start start 2023-08-25T09:50:23.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f522310c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f521c070cf0 0x7f521c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f522310c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c06eb60 con 0x7f521c152d80 2023-08-25T09:50:23.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f522310c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c06ca30 con 0x7f521c0c4560 2023-08-25T09:50:23.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f522310c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c06cf90 con 0x7f521c070cf0 2023-08-25T09:50:23.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f52216a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f521c070cf0 0x7f521c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f5220ea8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f521c152d80 0x7f521c0c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36810/0 (socket says 172.21.15.71:36810) 2023-08-25T09:50:23.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.593+0000 7f5220ea8640 1 -- 172.21.15.71:0/3041998295 learned_addr learned my addr 172.21.15.71:0/3041998295 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:23.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.594+0000 7f52216a9640 1 -- 172.21.15.71:0/3041998295 >> v1:172.21.15.164:6789/0 conn(0x7f521c0c4560 legacy=0x7f521c1617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:23.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.594+0000 7f52216a9640 1 -- 172.21.15.71:0/3041998295 >> v1:172.21.15.71:6789/0 conn(0x7f521c152d80 legacy=0x7f521c0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.594+0000 7f52216a9640 1 -- 172.21.15.71:0/3041998295 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c158fd0 con 0x7f521c070cf0 2023-08-25T09:50:23.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.594+0000 7f521b7fe640 1 -- 172.21.15.71:0/3041998295 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885021053 0 0) 0x7f521c06eb60 con 0x7f521c152d80 2023-08-25T09:50:23.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.595+0000 7f52216a9640 1 --2- 172.21.15.71:0/3041998295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c070cf0 0x7f521c06e620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5214009d90 tx=0x7f521402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0446a0d364a335 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:23.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.595+0000 7f521b7fe640 1 -- 172.21.15.71:0/3041998295 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521402cbd0 con 0x7f521c070cf0 2023-08-25T09:50:23.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.595+0000 7f521b7fe640 1 -- 172.21.15.71:0/3041998295 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521402cd30 con 0x7f521c070cf0 2023-08-25T09:50:23.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.595+0000 7f521b7fe640 1 -- 172.21.15.71:0/3041998295 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521402cea0 con 0x7f521c070cf0 2023-08-25T09:50:23.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 -- 172.21.15.71:0/3041998295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c070cf0 msgr2=0x7f521c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 --2- 172.21.15.71:0/3041998295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c070cf0 0x7f521c06e620 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5214009d90 tx=0x7f521402be20 comp rx=0 tx=0).stop 2023-08-25T09:50:23.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 -- 172.21.15.71:0/3041998295 shutdown_connections 2023-08-25T09:50:23.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 --2- 172.21.15.71:0/3041998295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c070cf0 0x7f521c06e620 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 -- 172.21.15.71:0/3041998295 >> 172.21.15.71:0/3041998295 conn(0x7f521c0654f0 msgr2=0x7f521c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:23.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 -- 172.21.15.71:0/3041998295 shutdown_connections 2023-08-25T09:50:23.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.596+0000 7f522310c640 1 -- 172.21.15.71:0/3041998295 wait complete. 2023-08-25T09:50:23.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.598+0000 7f522310c640 1 Processor -- start 2023-08-25T09:50:23.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.598+0000 7f522310c640 1 -- start start 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f522310c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f521c070cf0 0x7f521c14bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f522310c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 0x7f521c0c33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f522310c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521c152d80 0x7f521c1f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f522310c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c14d9c0 con 0x7f521c0c4560 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f522310c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c1f1b50 con 0x7f521c070cf0 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f522310c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c14b640 con 0x7f521c152d80 2023-08-25T09:50:23.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f521bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 0x7f521c0c33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f521bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 0x7f521c0c33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50918/0 (socket says 172.21.15.71:50918) 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.599+0000 7f521bfff640 1 -- 172.21.15.71:0/1813448160 learned_addr learned my addr 172.21.15.71:0/1813448160 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f52216a9640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521c152d80 0x7f521c1f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f5220ea8640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f521c070cf0 0x7f521c14bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f521bfff640 1 -- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521c152d80 msgr2=0x7f521c1f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f521bfff640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521c152d80 0x7f521c1f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f521bfff640 1 -- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f521c070cf0 msgr2=0x7f521c14bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f521bfff640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f521c070cf0 0x7f521c14bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f521bfff640 1 -- 172.21.15.71:0/1813448160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5214009a70 con 0x7f521c0c4560 2023-08-25T09:50:23.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f52216a9640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521c152d80 0x7f521c1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:23.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f521bfff640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 0x7f521c0c33f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f520800c430 tx=0x7f520800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:23.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208011790 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.600+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52080118f0 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.601+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c14b920 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.601+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208011ce0 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.601+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c1508f0 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.602+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c14d300 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.603+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f520801c460 con 0x7f521c0c4560 2023-08-25T09:50:23.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.603+0000 7f52197fa640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f520006a080 0x7f520006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:23.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.604+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5208088970 con 0x7f521c0c4560 2023-08-25T09:50:23.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.604+0000 7f5220ea8640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f520006a080 0x7f520006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:23.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.604+0000 7f5220ea8640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f520006a080 0x7f520006c540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5210007fd0 tx=0x7f5210009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:23.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.608+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f521c14d300 con 0x7f521c0c4560 2023-08-25T09:50:23.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.800+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f521c14d6e0 con 0x7f521c0c4560 2023-08-25T09:50:23.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.801+0000 7f52197fa640 1 -- 172.21.15.71:0/1813448160 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f521c14d6e0 con 0x7f521c0c4560 2023-08-25T09:50:23.804 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:50:23.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.803+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f520006a080 msgr2=0x7f520006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.803+0000 7f522310c640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f520006a080 0x7f520006c540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5210007fd0 tx=0x7f5210009040 comp rx=0 tx=0).stop 2023-08-25T09:50:23.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.803+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 msgr2=0x7f521c0c33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:23.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.803+0000 7f522310c640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 0x7f521c0c33f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f520800c430 tx=0x7f520800c8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 shutdown_connections 2023-08-25T09:50:23.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f520006a080 0x7f520006c540 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f521c152d80 0x7f521c1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f521c0c4560 0x7f521c0c33f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 --2- 172.21.15.71:0/1813448160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f521c070cf0 0x7f521c14bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:23.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 >> 172.21.15.71:0/1813448160 conn(0x7f521c0654f0 msgr2=0x7f521c0c8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:23.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 shutdown_connections 2023-08-25T09:50:23.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:23.804+0000 7f522310c640 1 -- 172.21.15.71:0/1813448160 wait complete. 2023-08-25T09:50:23.826 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:50:24.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.045+0000 7fa4e655d640 1 Processor -- start 2023-08-25T09:50:24.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.045+0000 7fa4e655d640 1 -- start start 2023-08-25T09:50:24.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4e655d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 0x7fa4e00c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4e655d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa4e00c3940 0x7fa4e01598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4e655d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e006a960 con 0x7fa4e01594d0 2023-08-25T09:50:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4e655d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e006ac20 con 0x7fa4e00c3940 2023-08-25T09:50:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4e655d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e006af00 con 0x7fa4e006cc70 2023-08-25T09:50:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4e4afa640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa4e00c3940 0x7fa4e01598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4df7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 0x7fa4e00c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.047+0000 7fa4df7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 0x7fa4e00c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:49304/0 (socket says 172.21.15.71:49304) 2023-08-25T09:50:24.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.048+0000 7fa4df7fe640 1 -- 172.21.15.71:0/2132986755 learned_addr learned my addr 172.21.15.71:0/2132986755 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:24.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.048+0000 7fa4df7fe640 1 -- 172.21.15.71:0/2132986755 >> v2:172.21.15.164:3300/0 conn(0x7fa4e00c3940 msgr2=0x7fa4e01598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:24.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.048+0000 7fa4df7fe640 1 --2- 172.21.15.71:0/2132986755 >> v2:172.21.15.164:3300/0 conn(0x7fa4e00c3940 0x7fa4e01598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.048+0000 7fa4df7fe640 1 -- 172.21.15.71:0/2132986755 >> v1:172.21.15.164:6789/0 conn(0x7fa4e01594d0 legacy=0x7fa4e014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.048+0000 7fa4df7fe640 1 -- 172.21.15.71:0/2132986755 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00c2e60 con 0x7fa4e006cc70 2023-08-25T09:50:24.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.050+0000 7fa4df7fe640 1 --2- 172.21.15.71:0/2132986755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 0x7fa4e00c4c90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009870 tx=0x7fa4d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2c54b5e829cdf90 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.051+0000 7fa4deffd640 1 -- 172.21.15.71:0/2132986755 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4030840 con 0x7fa4e006cc70 2023-08-25T09:50:24.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.051+0000 7fa4deffd640 1 -- 172.21.15.71:0/2132986755 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d40309a0 con 0x7fa4e006cc70 2023-08-25T09:50:24.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.051+0000 7fa4deffd640 1 -- 172.21.15.71:0/2132986755 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4030d30 con 0x7fa4e006cc70 2023-08-25T09:50:24.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 -- 172.21.15.71:0/2132986755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 msgr2=0x7fa4e00c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 --2- 172.21.15.71:0/2132986755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 0x7fa4e00c4c90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009870 tx=0x7fa4d402b980 comp rx=0 tx=0).stop 2023-08-25T09:50:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 -- 172.21.15.71:0/2132986755 shutdown_connections 2023-08-25T09:50:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 --2- 172.21.15.71:0/2132986755 >> v2:172.21.15.164:3300/0 conn(0x7fa4e00c3940 0x7fa4e01598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 --2- 172.21.15.71:0/2132986755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e006cc70 0x7fa4e00c4c90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 -- 172.21.15.71:0/2132986755 >> 172.21.15.71:0/2132986755 conn(0x7fa4e00654f0 msgr2=0x7fa4e0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 -- 172.21.15.71:0/2132986755 shutdown_connections 2023-08-25T09:50:24.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.052+0000 7fa4e655d640 1 -- 172.21.15.71:0/2132986755 wait complete. 2023-08-25T09:50:24.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.054+0000 7fa4e655d640 1 Processor -- start 2023-08-25T09:50:24.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.054+0000 7fa4e655d640 1 -- start start 2023-08-25T09:50:24.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.055+0000 7fa4e655d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4e006cc70 0x7fa4e0144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.055+0000 7fa4e655d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 0x7fa4e0145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.055+0000 7fa4e655d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e01594d0 0x7fa4e01ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.055+0000 7fa4e655d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01ee1f0 con 0x7fa4e00c3940 2023-08-25T09:50:24.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.055+0000 7fa4e655d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01edb50 con 0x7fa4e006cc70 2023-08-25T09:50:24.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.055+0000 7fa4e655d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00c9da0 con 0x7fa4e01594d0 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 0x7fa4e0145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 0x7fa4e0145890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50930/0 (socket says 172.21.15.71:50930) 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 -- 172.21.15.71:0/1537343746 learned_addr learned my addr 172.21.15.71:0/1537343746 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4dffff640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4e006cc70 0x7fa4e0144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4e4afa640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e01594d0 0x7fa4e01ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 -- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e01594d0 msgr2=0x7fa4e01ee610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e01594d0 0x7fa4e01ee610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 -- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4e006cc70 msgr2=0x7fa4e0144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4e006cc70 0x7fa4e0144e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.056+0000 7fa4df7fe640 1 -- 172.21.15.71:0/1537343746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40094d0 con 0x7fa4e00c3940 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.057+0000 7fa4df7fe640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 0x7fa4e0145890 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009870 tx=0x7fa4d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.057+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4039a90 con 0x7fa4e00c3940 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.057+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d4039c10 con 0x7fa4e00c3940 2023-08-25T09:50:24.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.057+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40433f0 con 0x7fa4e00c3940 2023-08-25T09:50:24.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.057+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e01f1260 con 0x7fa4e00c3940 2023-08-25T09:50:24.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.057+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e0147820 con 0x7fa4e00c3940 2023-08-25T09:50:24.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.059+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e01ee330 con 0x7fa4e00c3940 2023-08-25T09:50:24.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.059+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa4d4042390 con 0x7fa4e00c3940 2023-08-25T09:50:24.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.060+0000 7fa4dcff9640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa4cc06a050 0x7fa4cc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.060+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa4d40aa9d0 con 0x7fa4e00c3940 2023-08-25T09:50:24.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.061+0000 7fa4dffff640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa4cc06a050 0x7fa4cc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.061+0000 7fa4dffff640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa4cc06a050 0x7fa4cc06c510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0000f60 tx=0x7fa4d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.065+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa4e01ee330 con 0x7fa4e00c3940 2023-08-25T09:50:24.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.252+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa4e01ee330 con 0x7fa4e00c3940 2023-08-25T09:50:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.253+0000 7fa4dcff9640 1 -- 172.21.15.71:0/1537343746 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v31) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa4e01ee330 con 0x7fa4e00c3940 2023-08-25T09:50:24.256 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:24.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa4cc06a050 msgr2=0x7fa4cc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa4cc06a050 0x7fa4cc06c510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0000f60 tx=0x7fa4d0009040 comp rx=0 tx=0).stop 2023-08-25T09:50:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 msgr2=0x7fa4e0145890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 0x7fa4e0145890 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009870 tx=0x7fa4d4004730 comp rx=0 tx=0).stop 2023-08-25T09:50:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 shutdown_connections 2023-08-25T09:50:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa4cc06a050 0x7fa4cc06c510 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4e01594d0 0x7fa4e01ee610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa4e00c3940 0x7fa4e0145890 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 --2- 172.21.15.71:0/1537343746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4e006cc70 0x7fa4e0144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 >> 172.21.15.71:0/1537343746 conn(0x7fa4e00654f0 msgr2=0x7fa4e0069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 shutdown_connections 2023-08-25T09:50:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.256+0000 7fa4e655d640 1 -- 172.21.15.71:0/1537343746 wait complete. 2023-08-25T09:50:24.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:50:24.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.484+0000 7f5dded4f640 1 Processor -- start 2023-08-25T09:50:24.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.484+0000 7f5dded4f640 1 -- start start 2023-08-25T09:50:24.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.486+0000 7f5dded4f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dd8159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.486+0000 7f5dded4f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd8158e40 con 0x7f5dd81530c0 2023-08-25T09:50:24.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.486+0000 7f5dded4f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd81534a0 con 0x7f5dd80c7740 2023-08-25T09:50:24.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.486+0000 7f5dded4f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd806bc80 con 0x7f5dd81592d0 2023-08-25T09:50:24.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.486+0000 7f5ddcaeb640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5dd81530c0 0x7f5dd80c6e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36824/0 (socket says 172.21.15.71:36824) 2023-08-25T09:50:24.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.486+0000 7f5ddcaeb640 1 -- 172.21.15.71:0/214326721 learned_addr learned my addr 172.21.15.71:0/214326721 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:24.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551776329 0 0) 0x7f5dd8158e40 con 0x7f5dd81530c0 2023-08-25T09:50:24.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dd8159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8003520 con 0x7f5dd81530c0 2023-08-25T09:50:24.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4067189059 0 0) 0x7f5dc8003520 con 0x7f5dd81530c0 2023-08-25T09:50:24.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 msgr2=0x7f5dd8159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dd8159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 >> v1:172.21.15.164:6789/0 conn(0x7f5dd80c7740 legacy=0x7f5dd806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd8159ca0 con 0x7f5dd81530c0 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f5dcc001670 con 0x7f5dd81530c0 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 >> v1:172.21.15.71:6789/0 conn(0x7f5dd81530c0 legacy=0x7f5dd80c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dc8003a10 0x7f5dc8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dc80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dc8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc800bb20 con 0x7f5dc8004d00 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc800bc90 con 0x7f5dc8003a10 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.487+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc800be50 con 0x7f5dd81592d0 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dc8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddcaeb640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dc8003a10 0x7f5dc8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5dd7fff640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dc80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddd2ec640 1 -- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 msgr2=0x7f5dc80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dc80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddd2ec640 1 -- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dc8003a10 msgr2=0x7f5dc8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dc8003a10 0x7f5dc8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5ddd2ec640 1 -- 172.21.15.71:0/214326721 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8003520 con 0x7f5dc8004d00 2023-08-25T09:50:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.488+0000 7f5dd7fff640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81592d0 0x7f5dc80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:24.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.489+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dc8005180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009f90 tx=0x7f5dc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbf44ce85835325 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.489+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd002cbd0 con 0x7f5dc8004d00 2023-08-25T09:50:24.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd000f040 con 0x7f5dc8004d00 2023-08-25T09:50:24.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dd77fe640 1 -- 172.21.15.71:0/214326721 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0034440 con 0x7f5dc8004d00 2023-08-25T09:50:24.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dded4f640 1 -- 172.21.15.71:0/214326721 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 msgr2=0x7f5dc8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dded4f640 1 --2- 172.21.15.71:0/214326721 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dc8005180 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009f90 tx=0x7f5dc80057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5ddd2ec640 1 -- 172.21.15.71:0/214326721 reap_dead start 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dded4f640 1 -- 172.21.15.71:0/214326721 shutdown_connections 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dded4f640 1 -- 172.21.15.71:0/214326721 >> 172.21.15.71:0/214326721 conn(0x7f5dd80654f0 msgr2=0x7f5dd8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dded4f640 1 -- 172.21.15.71:0/214326721 shutdown_connections 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.490+0000 7f5dded4f640 1 -- 172.21.15.71:0/214326721 wait complete. 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.492+0000 7f5dded4f640 1 Processor -- start 2023-08-25T09:50:24.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.492+0000 7f5dded4f640 1 -- start start 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5dded4f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dd8151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5dded4f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81e2770 0x7f5dd81efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5dded4f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dd80cc510 0x7f5dd81f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5dded4f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81f2730 con 0x7f5dc8004d00 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5dded4f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81f1bb0 con 0x7f5dd80cc510 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5dded4f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd81f1e90 con 0x7f5dd81e2770 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5ddcaeb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dd8151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5ddcaeb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dd8151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50960/0 (socket says 172.21.15.71:50960) 2023-08-25T09:50:24.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.493+0000 7f5ddcaeb640 1 -- 172.21.15.71:0/2739678241 learned_addr learned my addr 172.21.15.71:0/2739678241 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:24.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5dd7fff640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81e2770 0x7f5dd81efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddcaeb640 1 -- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81e2770 msgr2=0x7f5dd81efe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddcaeb640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81e2770 0x7f5dd81efe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddcaeb640 1 -- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dd80cc510 msgr2=0x7f5dd81f0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dd80cc510 0x7f5dd81f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddcaeb640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dd80cc510 0x7f5dd81f0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddcaeb640 1 -- 172.21.15.71:0/2739678241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd81f2170 con 0x7f5dc8004d00 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddd2ec640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dd80cc510 0x7f5dd81f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5ddcaeb640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dd8151740 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00e730 tx=0x7f5dcc00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00c6c0 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc00c840 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00cbf0 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.494+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd81f2450 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.495+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd81f1850 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.496+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd8069160 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.497+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5dcc01d460 con 0x7f5dc8004d00 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.497+0000 7f5dd57fa640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5dbc06a080 0x7f5dbc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.498+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f5dcc0889a0 con 0x7f5dc8004d00 2023-08-25T09:50:24.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.498+0000 7f5dd7fff640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5dbc06a080 0x7f5dbc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.498+0000 7f5dd7fff640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5dbc06a080 0x7f5dbc06c540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4007fd0 tx=0x7f5dc4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.502+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5dcc053d70 con 0x7f5dc8004d00 2023-08-25T09:50:24.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.690+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5dd80c6450 con 0x7f5dc8004d00 2023-08-25T09:50:24.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.691+0000 7f5dd57fa640 1 -- 172.21.15.71:0/2739678241 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v31) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5dcc086e60 con 0x7f5dc8004d00 2023-08-25T09:50:24.694 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:50:24.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5dbc06a080 msgr2=0x7f5dbc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5dbc06a080 0x7f5dbc06c540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4007fd0 tx=0x7f5dc4009040 comp rx=0 tx=0).stop 2023-08-25T09:50:24.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 msgr2=0x7f5dd8151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dd8151740 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00e730 tx=0x7f5dcc00ebf0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 shutdown_connections 2023-08-25T09:50:24.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dd80cc510 0x7f5dd81f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5dbc06a080 0x7f5dbc06c540 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dd81e2770 0x7f5dd81efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 --2- 172.21.15.71:0/2739678241 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dc8004d00 0x7f5dd8151740 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 >> 172.21.15.71:0/2739678241 conn(0x7f5dd80654f0 msgr2=0x7f5dd8152340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:24.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 shutdown_connections 2023-08-25T09:50:24.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.694+0000 7f5dded4f640 1 -- 172.21.15.71:0/2739678241 wait complete. 2023-08-25T09:50:24.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:50:24.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.918+0000 7f96b228e640 1 Processor -- start 2023-08-25T09:50:24.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.918+0000 7f96b228e640 1 -- start start 2023-08-25T09:50:24.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.919+0000 7f96b228e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f96ac152d80 0x7f96ac06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.919+0000 7f96b228e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac070cf0 0x7f96ac1637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.919+0000 7f96b228e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac06eb60 con 0x7f96ac0c4560 2023-08-25T09:50:24.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.919+0000 7f96b228e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac06ca30 con 0x7f96ac152d80 2023-08-25T09:50:24.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.919+0000 7f96b228e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac06cf90 con 0x7f96ac070cf0 2023-08-25T09:50:24.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.920+0000 7f96b082b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac070cf0 0x7f96ac1637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.920+0000 7f96abfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f96ac152d80 0x7f96ac06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.920+0000 7f96abfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f96ac152d80 0x7f96ac06e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:40982/0 (socket says 172.21.15.71:40982) 2023-08-25T09:50:24.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.920+0000 7f96b082b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac070cf0 0x7f96ac1637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:49354/0 (socket says 172.21.15.71:49354) 2023-08-25T09:50:24.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.920+0000 7f96abfff640 1 -- 172.21.15.71:0/1545098472 learned_addr learned my addr 172.21.15.71:0/1545098472 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:24.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.921+0000 7f96abfff640 1 -- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac070cf0 msgr2=0x7f96ac1637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:24.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.921+0000 7f96abfff640 1 --2- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac070cf0 0x7f96ac1637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.921+0000 7f96abfff640 1 -- 172.21.15.71:0/1545098472 >> v1:172.21.15.164:6789/0 conn(0x7f96ac0c4560 legacy=0x7f96ac0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.921+0000 7f96abfff640 1 -- 172.21.15.71:0/1545098472 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac158fd0 con 0x7f96ac152d80 2023-08-25T09:50:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.922+0000 7f96abfff640 1 --2- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac152d80 0x7f96ac06e620 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f969c009870 tx=0x7f969c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e58fecf704376c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.923+0000 7f96ab7fe640 1 -- 172.21.15.71:0/1545098472 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c031640 con 0x7f96ac152d80 2023-08-25T09:50:24.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.923+0000 7f96ab7fe640 1 -- 172.21.15.71:0/1545098472 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c0317a0 con 0x7f96ac152d80 2023-08-25T09:50:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.923+0000 7f96ab7fe640 1 -- 172.21.15.71:0/1545098472 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c031b30 con 0x7f96ac152d80 2023-08-25T09:50:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.923+0000 7f96b228e640 1 -- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac152d80 msgr2=0x7f96ac06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.923+0000 7f96b228e640 1 --2- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac152d80 0x7f96ac06e620 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f969c009870 tx=0x7f969c02b8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.924+0000 7f96b228e640 1 -- 172.21.15.71:0/1545098472 shutdown_connections 2023-08-25T09:50:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.924+0000 7f96b228e640 1 --2- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac070cf0 0x7f96ac1637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.924+0000 7f96b228e640 1 --2- 172.21.15.71:0/1545098472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac152d80 0x7f96ac06e620 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.924+0000 7f96b228e640 1 -- 172.21.15.71:0/1545098472 >> 172.21.15.71:0/1545098472 conn(0x7f96ac0654f0 msgr2=0x7f96ac065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:24.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.924+0000 7f96b228e640 1 -- 172.21.15.71:0/1545098472 shutdown_connections 2023-08-25T09:50:24.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.924+0000 7f96b228e640 1 -- 172.21.15.71:0/1545098472 wait complete. 2023-08-25T09:50:24.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.926+0000 7f96b228e640 1 Processor -- start 2023-08-25T09:50:24.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.926+0000 7f96b228e640 1 -- start start 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96b228e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 0x7f96ac153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96b228e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac0c4560 0x7f96ac1f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96b228e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac152d80 0x7f96ac154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96b228e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac1f25d0 con 0x7f96ac070cf0 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96b228e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac1f22f0 con 0x7f96ac0c4560 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96b228e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac14c230 con 0x7f96ac152d80 2023-08-25T09:50:24.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 0x7f96ac153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 0x7f96ac153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50970/0 (socket says 172.21.15.71:50970) 2023-08-25T09:50:24.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.927+0000 7f96abfff640 1 -- 172.21.15.71:0/3991962047 learned_addr learned my addr 172.21.15.71:0/3991962047 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:24.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96b082b640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac152d80 0x7f96ac154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96a3fff640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac0c4560 0x7f96ac1f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96abfff640 1 -- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac152d80 msgr2=0x7f96ac154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96abfff640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac152d80 0x7f96ac154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96abfff640 1 -- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac0c4560 msgr2=0x7f96ac1f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:24.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96abfff640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac0c4560 0x7f96ac1f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96abfff640 1 -- 172.21.15.71:0/3991962047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0094d0 con 0x7f96ac070cf0 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.928+0000 7f96abfff640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 0x7f96ac153ba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.929+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c03a930 con 0x7f96ac070cf0 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.929+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c03aa90 con 0x7f96ac070cf0 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.929+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c03ae40 con 0x7f96ac070cf0 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.929+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac14c510 con 0x7f96ac070cf0 2023-08-25T09:50:24.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.929+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac1540e0 con 0x7f96ac070cf0 2023-08-25T09:50:24.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.930+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac069160 con 0x7f96ac070cf0 2023-08-25T09:50:24.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.931+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f969c02f050 con 0x7f96ac070cf0 2023-08-25T09:50:24.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.932+0000 7f96a97fa640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f968c06a080 0x7f968c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:24.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.932+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f969c0aa700 con 0x7f96ac070cf0 2023-08-25T09:50:24.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.932+0000 7f96a3fff640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f968c06a080 0x7f968c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:24.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.933+0000 7f96a3fff640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f968c06a080 0x7f968c06c540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0c3df0 tx=0x7f9694009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:24.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:24.936+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f969c075a30 con 0x7f96ac070cf0 2023-08-25T09:50:25.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.126+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f96ac06c4a0 con 0x7f96ac070cf0 2023-08-25T09:50:25.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.127+0000 7f96a97fa640 1 -- 172.21.15.71:0/3991962047 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) v1 ==== 103+0+11 (secure 0 0 0) 0x7f969c079050 con 0x7f96ac070cf0 2023-08-25T09:50:25.130 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:25.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f968c06a080 msgr2=0x7f968c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f968c06a080 0x7f968c06c540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0c3df0 tx=0x7f9694009040 comp rx=0 tx=0).stop 2023-08-25T09:50:25.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 msgr2=0x7f96ac153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 0x7f96ac153ba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f969c000c00 tx=0x7f969c0046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 shutdown_connections 2023-08-25T09:50:25.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ac0c4560 0x7f96ac1f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f968c06a080 0x7f968c06c540 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ac152d80 0x7f96ac154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 --2- 172.21.15.71:0/3991962047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f96ac070cf0 0x7f96ac153ba0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 >> 172.21.15.71:0/3991962047 conn(0x7f96ac0654f0 msgr2=0x7f96ac0c8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:25.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 shutdown_connections 2023-08-25T09:50:25.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.130+0000 7f96b228e640 1 -- 172.21.15.71:0/3991962047 wait complete. 2023-08-25T09:50:25.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:50:25.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.365+0000 7ff73da97640 1 Processor -- start 2023-08-25T09:50:25.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.365+0000 7ff73da97640 1 -- start start 2023-08-25T09:50:25.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.366+0000 7ff73da97640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff738069850 0x7ff7381576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.366+0000 7ff73da97640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7380c4fb0 0x7ff7380c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff73da97640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff738157bf0 con 0x7ff7381542d0 2023-08-25T09:50:25.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff73da97640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7380c4b60 con 0x7ff738069850 2023-08-25T09:50:25.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff73da97640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73806a130 con 0x7ff7380c4fb0 2023-08-25T09:50:25.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff737fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff738069850 0x7ff7381576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff737fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff738069850 0x7ff7381576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50974/0 (socket says 172.21.15.71:50974) 2023-08-25T09:50:25.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff737fff640 1 -- 172.21.15.71:0/2353738224 learned_addr learned my addr 172.21.15.71:0/2353738224 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:25.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff73d296640 1 --2- 172.21.15.71:0/2353738224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7380c4fb0 0x7ff7380c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff737fff640 1 -- 172.21.15.71:0/2353738224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7380c4fb0 msgr2=0x7ff7380c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:25.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff737fff640 1 --2- 172.21.15.71:0/2353738224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7380c4fb0 0x7ff7380c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.367+0000 7ff737fff640 1 -- 172.21.15.71:0/2353738224 >> v1:172.21.15.164:6789/0 conn(0x7ff7381542d0 legacy=0x7ff7380c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.368+0000 7ff737fff640 1 -- 172.21.15.71:0/2353738224 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380c5980 con 0x7ff738069850 2023-08-25T09:50:25.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.369+0000 7ff737fff640 1 --2- 172.21.15.71:0/2353738224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff738069850 0x7ff7381576b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff724009870 tx=0x7ff72402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f223f32e63daeeb server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:25.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.369+0000 7ff7377fe640 1 -- 172.21.15.71:0/2353738224 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724030840 con 0x7ff738069850 2023-08-25T09:50:25.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.369+0000 7ff7377fe640 1 -- 172.21.15.71:0/2353738224 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7240309a0 con 0x7ff738069850 2023-08-25T09:50:25.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.369+0000 7ff7377fe640 1 -- 172.21.15.71:0/2353738224 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724030d30 con 0x7ff738069850 2023-08-25T09:50:25.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.369+0000 7ff73da97640 1 -- 172.21.15.71:0/2353738224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff738069850 msgr2=0x7ff7381576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.369+0000 7ff73da97640 1 --2- 172.21.15.71:0/2353738224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff738069850 0x7ff7381576b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff724009870 tx=0x7ff72402b980 comp rx=0 tx=0).stop 2023-08-25T09:50:25.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.370+0000 7ff73da97640 1 -- 172.21.15.71:0/2353738224 shutdown_connections 2023-08-25T09:50:25.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.370+0000 7ff73da97640 1 --2- 172.21.15.71:0/2353738224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7380c4fb0 0x7ff7380c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.370+0000 7ff73da97640 1 --2- 172.21.15.71:0/2353738224 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff738069850 0x7ff7381576b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.370+0000 7ff73da97640 1 -- 172.21.15.71:0/2353738224 >> 172.21.15.71:0/2353738224 conn(0x7ff73806c280 msgr2=0x7ff73806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:25.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.370+0000 7ff73da97640 1 -- 172.21.15.71:0/2353738224 shutdown_connections 2023-08-25T09:50:25.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.370+0000 7ff73da97640 1 -- 172.21.15.71:0/2353738224 wait complete. 2023-08-25T09:50:25.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.371+0000 7ff73da97640 1 Processor -- start 2023-08-25T09:50:25.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 -- start start 2023-08-25T09:50:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff738069850 0x7ff7381450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 0x7ff7381ef480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7381542d0 0x7ff7381ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff738159ee0 con 0x7ff7380c4fb0 2023-08-25T09:50:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7381f02e0 con 0x7ff738069850 2023-08-25T09:50:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.372+0000 7ff73da97640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73815a9e0 con 0x7ff7381542d0 2023-08-25T09:50:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 0x7ff7381ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 0x7ff7381ef480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50988/0 (socket says 172.21.15.71:50988) 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 -- 172.21.15.71:0/2577832878 learned_addr learned my addr 172.21.15.71:0/2577832878 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff73ca95640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff738069850 0x7ff7381450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 -- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7381542d0 msgr2=0x7ff7381ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7381542d0 0x7ff7381ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 -- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff738069850 msgr2=0x7ff7381450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff738069850 0x7ff7381450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 -- 172.21.15.71:0/2577832878 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7240094d0 con 0x7ff7380c4fb0 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff73ca95640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff738069850 0x7ff7381450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.373+0000 7ff737fff640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 0x7ff7381ef480 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff724000c00 tx=0x7ff724004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.374+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724039c90 con 0x7ff7380c4fb0 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.374+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724039df0 con 0x7ff7380c4fb0 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.374+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738152230 con 0x7ff7380c4fb0 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.374+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724043640 con 0x7ff7380c4fb0 2023-08-25T09:50:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.374+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff738157970 con 0x7ff7380c4fb0 2023-08-25T09:50:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.375+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73804e420 con 0x7ff7380c4fb0 2023-08-25T09:50:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.376+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff7240437a0 con 0x7ff7380c4fb0 2023-08-25T09:50:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.377+0000 7ff7357fa640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff71c06a050 0x7ff71c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.377+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff72403e070 con 0x7ff7380c4fb0 2023-08-25T09:50:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.377+0000 7ff73ca95640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff71c06a050 0x7ff71c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.378+0000 7ff73ca95640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff71c06a050 0x7ff71c06c510 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff72c000f60 tx=0x7ff72c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.381+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff724076da0 con 0x7ff7380c4fb0 2023-08-25T09:50:25.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.609+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff738069f10 con 0x7ff7380c4fb0 2023-08-25T09:50:25.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.610+0000 7ff7357fa640 1 -- 172.21.15.71:0/2577832878 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6523 (secure 0 0 0) 0x7ff72407a3c0 con 0x7ff7380c4fb0 2023-08-25T09:50:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:25.615 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:50:23.760465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:50:25.616 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 17 2023-08-25T09:50:25.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff71c06a050 msgr2=0x7ff71c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff71c06a050 0x7ff71c06c510 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff72c000f60 tx=0x7ff72c009040 comp rx=0 tx=0).stop 2023-08-25T09:50:25.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 msgr2=0x7ff7381ef480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 0x7ff7381ef480 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff724000c00 tx=0x7ff724004730 comp rx=0 tx=0).stop 2023-08-25T09:50:25.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 shutdown_connections 2023-08-25T09:50:25.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff71c06a050 0x7ff71c06c510 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7381542d0 0x7ff7381ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7380c4fb0 0x7ff7381ef480 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 --2- 172.21.15.71:0/2577832878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff738069850 0x7ff7381450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 >> 172.21.15.71:0/2577832878 conn(0x7ff73806c280 msgr2=0x7ff7380c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:25.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 shutdown_connections 2023-08-25T09:50:25.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.613+0000 7ff73da97640 1 -- 172.21.15.71:0/2577832878 wait complete. 2023-08-25T09:50:25.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:25.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.839+0000 7f1860542640 1 Processor -- start 2023-08-25T09:50:25.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.839+0000 7f1860542640 1 -- start start 2023-08-25T09:50:25.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.840+0000 7f1860542640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f18581594d0 0x7f18580c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.840+0000 7f1860542640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f185806cc70 0x7f185814f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.840+0000 7f1860542640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18580c3940 0x7f185814fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.840+0000 7f1860542640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185806a960 con 0x7f185806cc70 2023-08-25T09:50:25.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.840+0000 7f1860542640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185806ac20 con 0x7f18581594d0 2023-08-25T09:50:25.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.840+0000 7f1860542640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185806af00 con 0x7f18580c3940 2023-08-25T09:50:25.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f185806cc70 0x7f185814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f185806cc70 0x7f185814f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50994/0 (socket says 172.21.15.71:50994) 2023-08-25T09:50:25.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 -- 172.21.15.71:0/3378859991 learned_addr learned my addr 172.21.15.71:0/3378859991 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:25.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185eadf640 1 --2- 172.21.15.71:0/3378859991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18580c3940 0x7f185814fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185e2de640 1 --2- 172.21.15.71:0/3378859991 >> v2:172.21.15.164:3300/0 conn(0x7f18581594d0 0x7f18580c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 -- 172.21.15.71:0/3378859991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18580c3940 msgr2=0x7f185814fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:25.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 --2- 172.21.15.71:0/3378859991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18580c3940 0x7f185814fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 -- 172.21.15.71:0/3378859991 >> v2:172.21.15.164:3300/0 conn(0x7f18581594d0 msgr2=0x7f18580c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:25.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 --2- 172.21.15.71:0/3378859991 >> v2:172.21.15.164:3300/0 conn(0x7f18581594d0 0x7f18580c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.841+0000 7f185dadd640 1 -- 172.21.15.71:0/3378859991 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580c2e60 con 0x7f185806cc70 2023-08-25T09:50:25.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.842+0000 7f185dadd640 1 --2- 172.21.15.71:0/3378859991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185806cc70 0x7f185814f850 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1858071d10 tx=0x7f184802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33c031664f0e194 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:25.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f185d2dc640 1 -- 172.21.15.71:0/3378859991 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848031860 con 0x7f185806cc70 2023-08-25T09:50:25.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f185d2dc640 1 -- 172.21.15.71:0/3378859991 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18480319c0 con 0x7f185806cc70 2023-08-25T09:50:25.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f185d2dc640 1 -- 172.21.15.71:0/3378859991 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848031d50 con 0x7f185806cc70 2023-08-25T09:50:25.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 -- 172.21.15.71:0/3378859991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185806cc70 msgr2=0x7f185814f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 --2- 172.21.15.71:0/3378859991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185806cc70 0x7f185814f850 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1858071d10 tx=0x7f184802b900 comp rx=0 tx=0).stop 2023-08-25T09:50:25.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 -- 172.21.15.71:0/3378859991 shutdown_connections 2023-08-25T09:50:25.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 --2- 172.21.15.71:0/3378859991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18580c3940 0x7f185814fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 --2- 172.21.15.71:0/3378859991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185806cc70 0x7f185814f850 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 --2- 172.21.15.71:0/3378859991 >> v2:172.21.15.164:3300/0 conn(0x7f18581594d0 0x7f18580c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 -- 172.21.15.71:0/3378859991 >> 172.21.15.71:0/3378859991 conn(0x7f18580654f0 msgr2=0x7f1858065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:25.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 -- 172.21.15.71:0/3378859991 shutdown_connections 2023-08-25T09:50:25.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.843+0000 7f1860542640 1 -- 172.21.15.71:0/3378859991 wait complete. 2023-08-25T09:50:25.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.845+0000 7f1860542640 1 Processor -- start 2023-08-25T09:50:25.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.845+0000 7f1860542640 1 -- start start 2023-08-25T09:50:25.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f1860542640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f185806cc70 0x7f185814b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f1860542640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 0x7f18581eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f1860542640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18581594d0 0x7f1858146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f1860542640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185814e240 con 0x7f18580c3940 2023-08-25T09:50:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f1860542640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1858070140 con 0x7f185806cc70 2023-08-25T09:50:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f1860542640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185814e450 con 0x7f18581594d0 2023-08-25T09:50:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.846+0000 7f185dadd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 0x7f18581eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 0x7f18581eef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51008/0 (socket says 172.21.15.71:51008) 2023-08-25T09:50:25.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 -- 172.21.15.71:0/1178615554 learned_addr learned my addr 172.21.15.71:0/1178615554 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185e2de640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f185806cc70 0x7f185814b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185eadf640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18581594d0 0x7f1858146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 -- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18581594d0 msgr2=0x7f1858146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18581594d0 0x7f1858146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 -- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f185806cc70 msgr2=0x7f185814b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:25.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f185806cc70 0x7f185814b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 -- 172.21.15.71:0/1178615554 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18480094d0 con 0x7f18580c3940 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f185dadd640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 0x7f18581eef80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f184802b940 tx=0x7f18480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848031850 con 0x7f18580c3940 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18480443f0 con 0x7f18580c3940 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.847+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18480447a0 con 0x7f18580c3940 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.848+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185814e730 con 0x7f18580c3940 2023-08-25T09:50:25.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.848+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1858146960 con 0x7f18580c3940 2023-08-25T09:50:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.849+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185814ec50 con 0x7f18580c3940 2023-08-25T09:50:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.849+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f184804d450 con 0x7f18580c3940 2023-08-25T09:50:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.850+0000 7f184effd640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f183c06e460 0x7f183c070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.850+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f18480ab010 con 0x7f18580c3940 2023-08-25T09:50:25.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.851+0000 7f185e2de640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f183c06e460 0x7f183c070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:25.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.851+0000 7f185e2de640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f183c06e460 0x7f183c070920 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f18580c1ba0 tx=0x7f1854008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:25.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:25.855+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f185814ec50 con 0x7f18580c3940 2023-08-25T09:50:26.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.045+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f185814ec50 con 0x7f18580c3940 2023-08-25T09:50:26.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.046+0000 7f184effd640 1 -- 172.21.15.71:0/1178615554 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7f185814ec50 con 0x7f18580c3940 2023-08-25T09:50:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:26.050 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:26.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f183c06e460 msgr2=0x7f183c070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f183c06e460 0x7f183c070920 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f18580c1ba0 tx=0x7f1854008040 comp rx=0 tx=0).stop 2023-08-25T09:50:26.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 msgr2=0x7f18581eef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 0x7f18581eef80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f184802b940 tx=0x7f18480046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 shutdown_connections 2023-08-25T09:50:26.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f183c06e460 0x7f183c070920 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18581594d0 0x7f1858146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18580c3940 0x7f18581eef80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 --2- 172.21.15.71:0/1178615554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f185806cc70 0x7f185814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 >> 172.21.15.71:0/1178615554 conn(0x7f18580654f0 msgr2=0x7f18580c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:26.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 shutdown_connections 2023-08-25T09:50:26.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.049+0000 7f1860542640 1 -- 172.21.15.71:0/1178615554 wait complete. 2023-08-25T09:50:26.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:50:26.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.284+0000 7f0df816b640 1 Processor -- start 2023-08-25T09:50:26.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.284+0000 7f0df816b640 1 -- start start 2023-08-25T09:50:26.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.286+0000 7f0df816b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0df00c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.286+0000 7f0df816b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df015db40 con 0x7f0df0066010 2023-08-25T09:50:26.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.286+0000 7f0df816b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df015d230 con 0x7f0df0070cf0 2023-08-25T09:50:26.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.286+0000 7f0df816b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df006cb40 con 0x7f0df0157450 2023-08-25T09:50:26.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.286+0000 7f0df5706640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0df0066010 0x7f0df006e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36828/0 (socket says 172.21.15.71:36828) 2023-08-25T09:50:26.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.286+0000 7f0df5706640 1 -- 172.21.15.71:0/958062362 learned_addr learned my addr 172.21.15.71:0/958062362 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:26.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df5f07640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0df00c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714828486 0 0) 0x7f0df015db40 con 0x7f0df0066010 2023-08-25T09:50:26.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc003520 con 0x7f0df0066010 2023-08-25T09:50:26.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3549881624 0 0) 0x7f0ddc003520 con 0x7f0df0066010 2023-08-25T09:50:26.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 msgr2=0x7f0df00c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:26.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0df00c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 >> v1:172.21.15.164:6789/0 conn(0x7f0df0070cf0 legacy=0x7f0df00c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df006ce20 con 0x7f0df0066010 2023-08-25T09:50:26.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0de0001670 con 0x7f0df0066010 2023-08-25T09:50:26.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df5f07640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0df00c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:26.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 >> v1:172.21.15.71:6789/0 conn(0x7f0df0066010 legacy=0x7f0df006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0ddc003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ddc004000 0x7f0ddc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004d00 0x7f0ddc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df006cb40 con 0x7f0ddc004000 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc00bc00 con 0x7f0ddc004d00 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.287+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc00bdc0 con 0x7f0df0157450 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5706640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ddc004000 0x7f0ddc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5f07640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0ddc003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df6708640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004d00 0x7f0ddc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5706640 1 -- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 msgr2=0x7f0ddc003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:26.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5706640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df0157450 0x7f0ddc003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5706640 1 -- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004d00 msgr2=0x7f0ddc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:26.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5706640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004d00 0x7f0ddc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.288+0000 7f0df5706640 1 -- 172.21.15.71:0/958062362 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc003520 con 0x7f0ddc004000 2023-08-25T09:50:26.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.289+0000 7f0df5706640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ddc004000 0x7f0ddc0047c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0de000c240 tx=0x7f0ddc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7272089f6fffd2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:26.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.289+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0005ad0 con 0x7f0ddc004000 2023-08-25T09:50:26.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.289+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de0005c30 con 0x7f0ddc004000 2023-08-25T09:50:26.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.289+0000 7f0df4f05640 1 -- 172.21.15.71:0/958062362 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de002caf0 con 0x7f0ddc004000 2023-08-25T09:50:26.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df816b640 1 -- 172.21.15.71:0/958062362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ddc004000 msgr2=0x7f0ddc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df816b640 1 --2- 172.21.15.71:0/958062362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ddc004000 0x7f0ddc0047c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0de000c240 tx=0x7f0ddc0057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df6708640 1 -- 172.21.15.71:0/958062362 reap_dead start 2023-08-25T09:50:26.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df816b640 1 -- 172.21.15.71:0/958062362 shutdown_connections 2023-08-25T09:50:26.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df816b640 1 -- 172.21.15.71:0/958062362 >> 172.21.15.71:0/958062362 conn(0x7f0df00654f0 msgr2=0x7f0df0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:26.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df816b640 1 -- 172.21.15.71:0/958062362 shutdown_connections 2023-08-25T09:50:26.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.290+0000 7f0df816b640 1 -- 172.21.15.71:0/958062362 wait complete. 2023-08-25T09:50:26.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.292+0000 7f0df816b640 1 Processor -- start 2023-08-25T09:50:26.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.292+0000 7f0df816b640 1 -- start start 2023-08-25T09:50:26.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df816b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004000 0x7f0df00c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df816b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 0x7f0df014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df816b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df00c1d60 0x7f0df014f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df816b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df015e610 con 0x7f0df00c3590 2023-08-25T09:50:26.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df816b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df01f29e0 con 0x7f0ddc004000 2023-08-25T09:50:26.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df816b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df01f2f10 con 0x7f0df00c1d60 2023-08-25T09:50:26.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df5706640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 0x7f0df014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df5706640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 0x7f0df014f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51030/0 (socket says 172.21.15.71:51030) 2023-08-25T09:50:26.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df5706640 1 -- 172.21.15.71:0/404398290 learned_addr learned my addr 172.21.15.71:0/404398290 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:26.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.293+0000 7f0df5f07640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004000 0x7f0df00c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df6708640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df00c1d60 0x7f0df014f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df5706640 1 -- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df00c1d60 msgr2=0x7f0df014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df5706640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df00c1d60 0x7f0df014f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df5706640 1 -- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004000 msgr2=0x7f0df00c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df5706640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004000 0x7f0df00c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df5706640 1 -- 172.21.15.71:0/404398290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df014fd90 con 0x7f0df00c3590 2023-08-25T09:50:26.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df6708640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df00c1d60 0x7f0df014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:26.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df5706640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 0x7f0df014f310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0de000c240 tx=0x7f0de0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:26.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de003c040 con 0x7f0df00c3590 2023-08-25T09:50:26.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de002ccc0 con 0x7f0df00c3590 2023-08-25T09:50:26.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0007270 con 0x7f0df00c3590 2023-08-25T09:50:26.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df0150270 con 0x7f0df00c3590 2023-08-25T09:50:26.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.294+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df00c2570 con 0x7f0df00c3590 2023-08-25T09:50:26.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.295+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df0071e30 con 0x7f0df00c3590 2023-08-25T09:50:26.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.296+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0de0032070 con 0x7f0df00c3590 2023-08-25T09:50:26.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.297+0000 7f0de67fc640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0dd406a050 0x7f0dd406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:26.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.297+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0de00a9b40 con 0x7f0df00c3590 2023-08-25T09:50:26.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.298+0000 7f0df5f07640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0dd406a050 0x7f0dd406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:26.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.298+0000 7f0df5f07640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0dd406a050 0x7f0dd406c510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0dec006fd0 tx=0x7f0dec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:26.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.302+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0de0074e60 con 0x7f0df00c3590 2023-08-25T09:50:26.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.524+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0df0069160 con 0x7f0df00c3590 2023-08-25T09:50:26.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.525+0000 7f0de67fc640 1 -- 172.21.15.71:0/404398290 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f0de0046090 con 0x7f0df00c3590 2023-08-25T09:50:26.528 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:26.529 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:50:23.760465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:50:26.529 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 17 2023-08-25T09:50:26.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.528+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0dd406a050 msgr2=0x7f0dd406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.528+0000 7f0df816b640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0dd406a050 0x7f0dd406c510 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0dec006fd0 tx=0x7f0dec008040 comp rx=0 tx=0).stop 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.528+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 msgr2=0x7f0df014f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.528+0000 7f0df816b640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 0x7f0df014f310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0de000c240 tx=0x7f0de0001a00 comp rx=0 tx=0).stop 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 shutdown_connections 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0dd406a050 0x7f0dd406c510 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0df00c1d60 0x7f0df014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0df00c3590 0x7f0df014f310 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 --2- 172.21.15.71:0/404398290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ddc004000 0x7f0df00c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:26.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 >> 172.21.15.71:0/404398290 conn(0x7f0df00654f0 msgr2=0x7f0df0156990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:26.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 shutdown_connections 2023-08-25T09:50:26.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:26.529+0000 7f0df816b640 1 -- 172.21.15.71:0/404398290 wait complete. 2023-08-25T09:50:27.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-08-25T09:49:34.063814+0000', 'modified': '2023-08-25T09:50:23.760465+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4306, 'mds_1': 4293, 'mds_2': 4290, 'mds_3': 4287, 'mds_4': 4238}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4238': {'gid': 4238, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.164:6835/2707317155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 2707317155}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 2707317155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4287': {'gid': 4287, 'name': 'a', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.71:6836/1556118065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6834', 'nonce': 1556118065}, {'type': 'v1', 'addr': '172.21.15.71:6836', 'nonce': 1556118065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4290': {'gid': 4290, 'name': 'c', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.71:6837/3404832976', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6835', 'nonce': 3404832976}, {'type': 'v1', 'addr': '172.21.15.71:6837', 'nonce': 3404832976}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4293': {'gid': 4293, 'name': 'e', 'rank': 1, 'incarnation': 8, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.71:6839/3369232025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6838', 'nonce': 3369232025}, {'type': 'v1', 'addr': '172.21.15.71:6839', 'nonce': 3369232025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4306': {'gid': 4306, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.164:6837/881271092', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 881271092}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 881271092}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-08-25T09:50:27.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-08-25T09:50:27.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-08-25T09:50:27.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.380+0000 7fa99474e640 1 Processor -- start 2023-08-25T09:50:27.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.380+0000 7fa99474e640 1 -- start start 2023-08-25T09:50:27.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.382+0000 7fa99474e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa98c150ce0 0x7fa98c0c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.382+0000 7fa99474e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa98c0c58c0 0x7fa98c0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.382+0000 7fa99474e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c0c2ba0 con 0x7fa98c0c76b0 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.382+0000 7fa99474e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0c1770 con 0x7fa98c0c58c0 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.382+0000 7fa99474e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0c1f30 con 0x7fa98c150ce0 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.382+0000 7fa991ce9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa98c0c58c0 0x7fa98c0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa991ce9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa98c0c58c0 0x7fa98c0c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51038/0 (socket says 172.21.15.71:51038) 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa991ce9640 1 -- 172.21.15.71:0/1126199670 learned_addr learned my addr 172.21.15.71:0/1126199670 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa9924ea640 1 --2- 172.21.15.71:0/1126199670 >> v2:172.21.15.164:3300/0 conn(0x7fa98c150ce0 0x7fa98c0c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa991ce9640 1 -- 172.21.15.71:0/1126199670 >> v2:172.21.15.164:3300/0 conn(0x7fa98c150ce0 msgr2=0x7fa98c0c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:27.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa991ce9640 1 --2- 172.21.15.71:0/1126199670 >> v2:172.21.15.164:3300/0 conn(0x7fa98c150ce0 0x7fa98c0c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa991ce9640 1 -- 172.21.15.71:0/1126199670 >> v1:172.21.15.164:6789/0 conn(0x7fa98c0c76b0 legacy=0x7fa98c15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:27.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.383+0000 7fa991ce9640 1 -- 172.21.15.71:0/1126199670 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0c6400 con 0x7fa98c0c58c0 2023-08-25T09:50:27.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.384+0000 7fa991ce9640 1 --2- 172.21.15.71:0/1126199670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c58c0 0x7fa98c0c34b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa98c074090 tx=0x7fa97c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dd68e200acaecd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:27.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.384+0000 7fa9914e8640 1 -- 172.21.15.71:0/1126199670 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c030860 con 0x7fa98c0c58c0 2023-08-25T09:50:27.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.384+0000 7fa9914e8640 1 -- 172.21.15.71:0/1126199670 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c0309c0 con 0x7fa98c0c58c0 2023-08-25T09:50:27.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.384+0000 7fa9914e8640 1 -- 172.21.15.71:0/1126199670 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c030d50 con 0x7fa98c0c58c0 2023-08-25T09:50:27.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 -- 172.21.15.71:0/1126199670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c58c0 msgr2=0x7fa98c0c34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:27.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 --2- 172.21.15.71:0/1126199670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c58c0 0x7fa98c0c34b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa98c074090 tx=0x7fa97c02b9a0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 -- 172.21.15.71:0/1126199670 shutdown_connections 2023-08-25T09:50:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 --2- 172.21.15.71:0/1126199670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c58c0 0x7fa98c0c34b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 --2- 172.21.15.71:0/1126199670 >> v2:172.21.15.164:3300/0 conn(0x7fa98c150ce0 0x7fa98c0c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 -- 172.21.15.71:0/1126199670 >> 172.21.15.71:0/1126199670 conn(0x7fa98c06bff0 msgr2=0x7fa98c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 -- 172.21.15.71:0/1126199670 shutdown_connections 2023-08-25T09:50:27.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.385+0000 7fa99474e640 1 -- 172.21.15.71:0/1126199670 wait complete. 2023-08-25T09:50:27.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.387+0000 7fa99474e640 1 Processor -- start 2023-08-25T09:50:27.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.387+0000 7fa99474e640 1 -- start start 2023-08-25T09:50:27.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa99474e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c0c58c0 0x7fa98c0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa99474e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 0x7fa98c1f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa99474e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c150ce0 0x7fa98c1f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa99474e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0c8770 con 0x7fa98c0c76b0 2023-08-25T09:50:27.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa99474e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c06fec0 con 0x7fa98c150ce0 2023-08-25T09:50:27.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa99474e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c070440 con 0x7fa98c0c58c0 2023-08-25T09:50:27.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa991ce9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 0x7fa98c1f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa991ce9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 0x7fa98c1f1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51048/0 (socket says 172.21.15.71:51048) 2023-08-25T09:50:27.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.388+0000 7fa991ce9640 1 -- 172.21.15.71:0/2040618150 learned_addr learned my addr 172.21.15.71:0/2040618150 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:27.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa991ce9640 1 -- 172.21.15.71:0/2040618150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c0c58c0 msgr2=0x7fa98c0c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:27.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa991ce9640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c0c58c0 0x7fa98c0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa991ce9640 1 -- 172.21.15.71:0/2040618150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c150ce0 msgr2=0x7fa98c1f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:27.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa991ce9640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c150ce0 0x7fa98c1f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa991ce9640 1 -- 172.21.15.71:0/2040618150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c0094d0 con 0x7fa98c0c76b0 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa991ce9640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 0x7fa98c1f1710 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa97c02b9e0 tx=0x7fa97c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa982ffd640 1 -- 172.21.15.71:0/2040618150 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c039c90 con 0x7fa98c0c76b0 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa982ffd640 1 -- 172.21.15.71:0/2040618150 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c039df0 con 0x7fa98c0c76b0 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa982ffd640 1 -- 172.21.15.71:0/2040618150 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c043620 con 0x7fa98c0c76b0 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c0c8d70 con 0x7fa98c0c76b0 2023-08-25T09:50:27.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.389+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c0c8490 con 0x7fa98c0c76b0 2023-08-25T09:50:27.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.392+0000 7fa982ffd640 1 -- 172.21.15.71:0/2040618150 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa97c043780 con 0x7fa98c0c76b0 2023-08-25T09:50:27.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.393+0000 7fa982ffd640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa97006a050 0x7fa97006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.393+0000 7fa982ffd640 1 -- 172.21.15.71:0/2040618150 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa97c03e070 con 0x7fa98c0c76b0 2023-08-25T09:50:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.393+0000 7fa9924ea640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa97006a050 0x7fa97006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.394+0000 7fa99739f740 1 Processor -- start 2023-08-25T09:50:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.394+0000 7fa99739f740 1 -- start start 2023-08-25T09:50:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.394+0000 7fa9924ea640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa97006a050 0x7fa97006c510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa988006e20 tx=0x7fa988008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:27.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa99739f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0bd84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd6630 0x55bcb0c17550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa99739f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0b45190 con 0x55bcb0c21a50 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0b42880 con 0x55bcb0bd7bd0 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0b3e170 con 0x55bcb0bd6630 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa991ce9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0bd84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa991ce9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0bd84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51058/0 (socket says 172.21.15.71:51058) 2023-08-25T09:50:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.395+0000 7fa991ce9640 1 -- 172.21.15.71:0/4152357175 learned_addr learned my addr 172.21.15.71:0/4152357175 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa991ce9640 1 -- 172.21.15.71:0/4152357175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd6630 msgr2=0x55bcb0c17550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa991ce9640 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd6630 0x55bcb0c17550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa991ce9640 1 -- 172.21.15.71:0/4152357175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 msgr2=0x55bcb0bd7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:27.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa991ce9640 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa991ce9640 1 -- 172.21.15.71:0/4152357175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcb0b3c7f0 con 0x55bcb0c21a50 2023-08-25T09:50:27.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa991ce9640 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0bd84f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa97c07a7c0 tx=0x7fa97c07af80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c0cb991dedf164 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa96ffff640 1 -- 172.21.15.71:0/4152357175 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c078970 con 0x55bcb0c21a50 2023-08-25T09:50:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa96ffff640 1 -- 172.21.15.71:0/4152357175 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c078bd0 con 0x55bcb0c21a50 2023-08-25T09:50:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.396+0000 7fa96ffff640 1 -- 172.21.15.71:0/4152357175 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c0820b0 con 0x55bcb0c21a50 2023-08-25T09:50:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 -- 172.21.15.71:0/4152357175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 msgr2=0x55bcb0bd84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0bd84f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa97c07a7c0 tx=0x7fa97c07af80 comp rx=0 tx=0).stop 2023-08-25T09:50:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 -- 172.21.15.71:0/4152357175 shutdown_connections 2023-08-25T09:50:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd6630 0x55bcb0c17550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0bd84f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 --2- 172.21.15.71:0/4152357175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 -- 172.21.15.71:0/4152357175 >> 172.21.15.71:0/4152357175 conn(0x55bcb0bd4340 msgr2=0x55bcb0bd6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 -- 172.21.15.71:0/4152357175 shutdown_connections 2023-08-25T09:50:27.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.397+0000 7fa99739f740 1 -- 172.21.15.71:0/4152357175 wait complete. 2023-08-25T09:50:27.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.398+0000 7fa99739f740 1 -- start start 2023-08-25T09:50:27.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.398+0000 7fa99739f740 1 Processor -- start 2023-08-25T09:50:27.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0c00350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0c21a50 0x55bcb0c2be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0bfda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0addd20 con 0x55bcb0c194d0 2023-08-25T09:50:27.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0add9c0 con 0x55bcb0bd7bd0 2023-08-25T09:50:27.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcb08a3f20 con 0x55bcb0c21a50 2023-08-25T09:50:27.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa99739f740 10 client.0 _ll_register_callbacks cb 0x55bcb0af5630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:27.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa992ceb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0bfda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa992ceb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0bfda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51064/0 (socket says 172.21.15.71:51064) 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.399+0000 7fa992ceb640 1 -- 172.21.15.71:0/85663972 learned_addr learned my addr 172.21.15.71:0/85663972 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa9924ea640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0c00350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa991ce9640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0c21a50 0x55bcb0c2be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa992ceb640 1 -- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0c21a50 msgr2=0x55bcb0c2be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa992ceb640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0c21a50 0x55bcb0c2be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa992ceb640 1 -- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 msgr2=0x55bcb0c00350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa992ceb640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0c00350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:27.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa992ceb640 1 -- 172.21.15.71:0/85663972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c0c8490 con 0x55bcb0c194d0 2023-08-25T09:50:27.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa9924ea640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0c00350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa991ce9640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0c21a50 0x55bcb0c2be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa992ceb640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0bfda30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa98400c190 tx=0x7fa98400c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68413a7283c52fc8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa96d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa96d7fa640 1 -- 172.21.15.71:0/85663972 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984003d80 con 0x55bcb0c194d0 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa96d7fa640 1 -- 172.21.15.71:0/85663972 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa98400ce20 con 0x55bcb0c194d0 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcb0b12540 con 0x55bcb0c194d0 2023-08-25T09:50:27.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa96d7fa640 1 -- 172.21.15.71:0/85663972 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa984001630 con 0x55bcb0c194d0 2023-08-25T09:50:27.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa99739f740 10 client.4899 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:27.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.400+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55bcb0b35f30 con 0x55bcb0c194d0 2023-08-25T09:50:27.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa96d7fa640 1 -- 172.21.15.71:0/85663972 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bcb0b12540 con 0x55bcb0c194d0 2023-08-25T09:50:27.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa96d7fa640 1 -- 172.21.15.71:0/85663972 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7fa9840050d0 con 0x55bcb0c194d0 2023-08-25T09:50:27.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa99739f740 10 client.4899 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:27.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa99739f740 10 client.4899 resolve_mds: resolved name 'd' to daemon mds.d 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa99739f740 20 client.4899 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa99739f740 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0bfafd0 0x55bcb0c131e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa99739f740 4 client.4899 mds_command: new command op to 4238 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.401+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 --> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcb0b37150 con 0x55bcb0bfafd0 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.402+0000 7fa9924ea640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0bfafd0 0x55bcb0c131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.403+0000 7fa9924ea640 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0bfafd0 0x55bcb0c131e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ad6e3395864328cd server_cookie=a8c32fdaa9eb5ef5 in_seq=0 out_seq=0 2023-08-25T09:50:27.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:27.403+0000 7fa96d7fa640 10 client.4899 ms_handle_connect on v2:172.21.15.164:6834/2707317155 2023-08-25T09:50:28.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.775+0000 7fa96d7fa640 1 -- 172.21.15.71:0/85663972 <== mds.4 v2:172.21.15.164:6834/2707317155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bcb0b37150 con 0x55bcb0bfafd0 2023-08-25T09:50:28.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.775+0000 7fa96d7fa640 10 client.4899 handle_command_reply: tid=0 2023-08-25T09:50:28.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.776+0000 7fa99739f740 1 client.4899 shutdown 2023-08-25T09:50:28.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.776+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 msgr2=0x55bcb0bfda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:28.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.776+0000 7fa99739f740 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0bfda30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa98400c190 tx=0x7fa98400c650 comp rx=0 tx=0).stop 2023-08-25T09:50:28.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 shutdown_connections 2023-08-25T09:50:28.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0bfafd0 0x55bcb0c131e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0bfda30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0c21a50 0x55bcb0c2be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 --2- 172.21.15.71:0/85663972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0bd7bd0 0x55bcb0c00350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 >> 172.21.15.71:0/85663972 conn(0x55bcb0bd6630 msgr2=0x55bcb0bd5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:28.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa96d7fa640 0 client.4899 ms_handle_reset on v2:172.21.15.164:6834/2707317155 2023-08-25T09:50:28.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 shutdown_connections 2023-08-25T09:50:28.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.777+0000 7fa99739f740 1 -- 172.21.15.71:0/85663972 wait complete. 2023-08-25T09:50:28.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.778+0000 7fa99739f740 20 client.4899 trim_cache size 0 max 16384 2023-08-25T09:50:28.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.802+0000 7fa99739f740 1 Processor -- start 2023-08-25T09:50:28.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.802+0000 7fa99739f740 1 -- start start 2023-08-25T09:50:28.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.803+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0c009d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.803+0000 7fa99739f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0c15d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.803+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c21a50 0x55bcb0c16280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.803+0000 7fa99739f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0b40050 con 0x55bcb0c194d0 2023-08-25T09:50:28.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.803+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0bfb750 con 0x55bcb0c21a50 2023-08-25T09:50:28.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.803+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0b71a60 con 0x55bcb0bd7bd0 2023-08-25T09:50:28.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa991ce9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0c15d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa992ceb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c21a50 0x55bcb0c16280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa991ce9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0c15d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51078/0 (socket says 172.21.15.71:51078) 2023-08-25T09:50:28.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa991ce9640 1 -- 172.21.15.71:0/1352773767 learned_addr learned my addr 172.21.15.71:0/1352773767 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:28.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa9924ea640 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0c009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa992ceb640 1 -- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 msgr2=0x55bcb0c009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:28.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.804+0000 7fa992ceb640 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0c009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa992ceb640 1 -- 172.21.15.71:0/1352773767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 msgr2=0x55bcb0c15d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:28.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa992ceb640 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0c15d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa992ceb640 1 -- 172.21.15.71:0/1352773767 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcb0adca10 con 0x55bcb0c21a50 2023-08-25T09:50:28.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa991ce9640 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0c15d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:50:28.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa992ceb640 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c21a50 0x55bcb0c16280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa98401b040 tx=0x7fa98400a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404f27701810fef0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa980ff9640 1 -- 172.21.15.71:0/1352773767 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98400cb50 con 0x55bcb0c21a50 2023-08-25T09:50:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa980ff9640 1 -- 172.21.15.71:0/1352773767 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9840050d0 con 0x55bcb0c21a50 2023-08-25T09:50:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.805+0000 7fa980ff9640 1 -- 172.21.15.71:0/1352773767 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9840057b0 con 0x55bcb0c21a50 2023-08-25T09:50:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 -- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c21a50 msgr2=0x55bcb0c16280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:28.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c21a50 0x55bcb0c16280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa98401b040 tx=0x7fa98400a000 comp rx=0 tx=0).stop 2023-08-25T09:50:28.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 -- 172.21.15.71:0/1352773767 shutdown_connections 2023-08-25T09:50:28.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c21a50 0x55bcb0c16280 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c194d0 0x55bcb0c15d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 --2- 172.21.15.71:0/1352773767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0c009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 -- 172.21.15.71:0/1352773767 >> 172.21.15.71:0/1352773767 conn(0x55bcb0bd6630 msgr2=0x55bcb0bcd730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:28.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 -- 172.21.15.71:0/1352773767 shutdown_connections 2023-08-25T09:50:28.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.807+0000 7fa99739f740 1 -- 172.21.15.71:0/1352773767 wait complete. 2023-08-25T09:50:28.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.809+0000 7fa99739f740 1 -- start start 2023-08-25T09:50:28.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.809+0000 7fa99739f740 1 Processor -- start 2023-08-25T09:50:28.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c194d0 0x55bcb0c7b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0c81570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0c69f00 con 0x55bcb0c21a50 2023-08-25T09:50:28.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0c303f0 con 0x55bcb0c194d0 2023-08-25T09:50:28.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcb0c81f80 con 0x55bcb0bd7bd0 2023-08-25T09:50:28.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.811+0000 7fa99739f740 10 client.0 _ll_register_callbacks cb 0x7fa97c076bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:28.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0c81570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0c81570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51080/0 (socket says 172.21.15.71:51080) 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 -- 172.21.15.71:0/2776690350 learned_addr learned my addr 172.21.15.71:0/2776690350 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa991ce9640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c194d0 0x55bcb0c7b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa9924ea640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 -- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 msgr2=0x55bcb0bd7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 -- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c194d0 msgr2=0x55bcb0c7b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:28.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c194d0 0x55bcb0c7b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.812+0000 7fa992ceb640 1 -- 172.21.15.71:0/2776690350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcb0b3c7f0 con 0x55bcb0c21a50 2023-08-25T09:50:28.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.813+0000 7fa992ceb640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0c81570 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa98401b040 tx=0x7fa984001410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978e6639ad7ba1c5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:28.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.813+0000 7fa96d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:28.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.813+0000 7fa96d7fa640 1 -- 172.21.15.71:0/2776690350 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa984003d80 con 0x55bcb0c21a50 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.813+0000 7fa96d7fa640 1 -- 172.21.15.71:0/2776690350 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fa984002090 con 0x55bcb0c21a50 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.813+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa97c078bd0 con 0x55bcb0c21a50 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa96d7fa640 1 -- 172.21.15.71:0/2776690350 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa984018250 con 0x55bcb0c21a50 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 10 client.4905 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55bcb0b35f30 con 0x55bcb0c21a50 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa96d7fa640 1 -- 172.21.15.71:0/2776690350 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c078bd0 con 0x55bcb0c21a50 2023-08-25T09:50:28.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa96d7fa640 1 -- 172.21.15.71:0/2776690350 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7fa9840183b0 con 0x55bcb0c21a50 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 10 client.4905 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 10 client.4905 resolve_mds: resolved name 'd' to daemon mds.d 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 20 client.4905 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0c17060 0x55bcb0c17440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 4 client.4905 mds_command: new command op to 4238 tid=0[{"prefix": "status"}] 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.814+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 --> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bcb07ed5e0 con 0x55bcb0c17060 2023-08-25T09:50:28.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.815+0000 7fa9924ea640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0c17060 0x55bcb0c17440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:28.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.816+0000 7fa9924ea640 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0c17060 0x55bcb0c17440 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bb9b6bc943181260 server_cookie=696b9ae4095f2649 in_seq=0 out_seq=0 2023-08-25T09:50:28.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.816+0000 7fa96d7fa640 10 client.4905 ms_handle_connect on v2:172.21.15.164:6834/2707317155 2023-08-25T09:50:28.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa96d7fa640 1 -- 172.21.15.71:0/2776690350 <== mds.4 v2:172.21.15.164:6834/2707317155 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55bcb07ed5e0 con 0x55bcb0c17060 2023-08-25T09:50:28.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa96d7fa640 10 client.4905 handle_command_reply: tid=0 2023-08-25T09:50:28.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 client.4905 shutdown 2023-08-25T09:50:28.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 msgr2=0x55bcb0c81570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:28.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0c81570 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa98401b040 tx=0x7fa984001410 comp rx=0 tx=0).stop 2023-08-25T09:50:28.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 shutdown_connections 2023-08-25T09:50:28.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:6834/2707317155,v1:172.21.15.164:6835/2707317155] conn(0x55bcb0c17060 0x55bcb0c17440 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55bcb0c194d0 0x55bcb0c7b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa96d7fa640 0 client.4905 ms_handle_reset on v2:172.21.15.164:6834/2707317155 2023-08-25T09:50:28.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55bcb0c21a50 0x55bcb0c81570 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 --2- 172.21.15.71:0/2776690350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55bcb0bd7bd0 0x55bcb0bd7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 >> 172.21.15.71:0/2776690350 conn(0x55bcb0bd6630 msgr2=0x55bcb0c15d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:28.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 shutdown_connections 2023-08-25T09:50:28.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.817+0000 7fa99739f740 1 -- 172.21.15.71:0/2776690350 wait complete. 2023-08-25T09:50:28.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.818+0000 7fa99739f740 20 client.4905 trim_cache size 0 max 16384 2023-08-25T09:50:28.826 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:50:28.826 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "d3b06f22-7372-48e3-a060-a2382f01af3f", 2023-08-25T09:50:28.826 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 4, 2023-08-25T09:50:28.826 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4238, 2023-08-25T09:50:28.826 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2023-08-25T09:50:28.826 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 11.083928082, 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 17, 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 31, 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 31, 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 61.059200095999998 2023-08-25T09:50:28.827 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:50:28.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa97006a050 msgr2=0x7fa97006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:28.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa97006a050 0x7fa97006c510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa988006e20 tx=0x7fa988008040 comp rx=0 tx=0).stop 2023-08-25T09:50:28.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 msgr2=0x7fa98c1f1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:28.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 0x7fa98c1f1710 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa97c02b9e0 tx=0x7fa97c004730 comp rx=0 tx=0).stop 2023-08-25T09:50:28.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 shutdown_connections 2023-08-25T09:50:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa98c150ce0 0x7fa98c1f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa97006a050 0x7fa97006c510 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa98c0c76b0 0x7fa98c1f1710 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 --2- 172.21.15.71:0/2040618150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa98c0c58c0 0x7fa98c0c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 >> 172.21.15.71:0/2040618150 conn(0x7fa98c06bff0 msgr2=0x7fa98c06c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 shutdown_connections 2023-08-25T09:50:28.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:28.819+0000 7fa99474e640 1 -- 172.21.15.71:0/2040618150 wait complete. 2023-08-25T09:50:28.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-08-25T09:50:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.060+0000 7f20cab42640 1 Processor -- start 2023-08-25T09:50:29.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.060+0000 7f20cab42640 1 -- start start 2023-08-25T09:50:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20cab42640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f20c4150ce0 0x7f20c40c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20cab42640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f20c40c76b0 0x7f20c41613f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20cab42640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40c2ba0 con 0x7f20c40c58c0 2023-08-25T09:50:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20cab42640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40c1770 con 0x7f20c4150ce0 2023-08-25T09:50:29.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20cab42640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40c1f30 con 0x7f20c40c76b0 2023-08-25T09:50:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20c88de640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f20c4150ce0 0x7f20c40c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20c88de640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f20c4150ce0 0x7f20c40c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51084/0 (socket says 172.21.15.71:51084) 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.062+0000 7f20c88de640 1 -- 172.21.15.71:0/99871051 learned_addr learned my addr 172.21.15.71:0/99871051 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.063+0000 7f20c90df640 1 --2- 172.21.15.71:0/99871051 >> v2:172.21.15.164:3300/0 conn(0x7f20c40c76b0 0x7f20c41613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.063+0000 7f20c88de640 1 -- 172.21.15.71:0/99871051 >> v2:172.21.15.164:3300/0 conn(0x7f20c40c76b0 msgr2=0x7f20c41613f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.063+0000 7f20c88de640 1 --2- 172.21.15.71:0/99871051 >> v2:172.21.15.164:3300/0 conn(0x7f20c40c76b0 0x7f20c41613f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.063+0000 7f20c88de640 1 -- 172.21.15.71:0/99871051 >> v1:172.21.15.71:6789/0 conn(0x7f20c40c58c0 legacy=0x7f20c40c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.063+0000 7f20c88de640 1 -- 172.21.15.71:0/99871051 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40c6400 con 0x7f20c4150ce0 2023-08-25T09:50:29.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.063+0000 7f20c37fe640 1 -- 172.21.15.71:0/99871051 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331808047 0 0) 0x7f20b0001110 con 0x7f20c40c58c0 2023-08-25T09:50:29.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.064+0000 7f20c88de640 1 --2- 172.21.15.71:0/99871051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c4150ce0 0x7f20c40c5380 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f20c40cb330 tx=0x7f20b802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970a9ace964ad1aa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.064+0000 7f20c37fe640 1 -- 172.21.15.71:0/99871051 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8004230 con 0x7f20c4150ce0 2023-08-25T09:50:29.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.064+0000 7f20c37fe640 1 -- 172.21.15.71:0/99871051 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b8004390 con 0x7f20c4150ce0 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.064+0000 7f20c37fe640 1 -- 172.21.15.71:0/99871051 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b802f830 con 0x7f20c4150ce0 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 -- 172.21.15.71:0/99871051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c4150ce0 msgr2=0x7f20c40c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 --2- 172.21.15.71:0/99871051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c4150ce0 0x7f20c40c5380 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f20c40cb330 tx=0x7f20b802ba10 comp rx=0 tx=0).stop 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 -- 172.21.15.71:0/99871051 shutdown_connections 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 --2- 172.21.15.71:0/99871051 >> v2:172.21.15.164:3300/0 conn(0x7f20c40c76b0 0x7f20c41613f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 --2- 172.21.15.71:0/99871051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c4150ce0 0x7f20c40c5380 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 -- 172.21.15.71:0/99871051 >> 172.21.15.71:0/99871051 conn(0x7f20c406bff0 msgr2=0x7f20c406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 -- 172.21.15.71:0/99871051 shutdown_connections 2023-08-25T09:50:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.065+0000 7f20cab42640 1 -- 172.21.15.71:0/99871051 wait complete. 2023-08-25T09:50:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.067+0000 7f20cab42640 1 Processor -- start 2023-08-25T09:50:29.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.067+0000 7f20cab42640 1 -- start start 2023-08-25T09:50:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20cab42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 0x7f20c4148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20cab42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40c76b0 0x7f20c4148590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20cab42640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4150ce0 0x7f20c40c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20cab42640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c41517e0 con 0x7f20c40c58c0 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20cab42640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c41523a0 con 0x7f20c4150ce0 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20cab42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40caed0 con 0x7f20c40c76b0 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 0x7f20c4148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 0x7f20c4148ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51100/0 (socket says 172.21.15.71:51100) 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 -- 172.21.15.71:0/1312214486 learned_addr learned my addr 172.21.15.71:0/1312214486 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 -- 172.21.15.71:0/1312214486 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40c76b0 msgr2=0x7f20c4148590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40c76b0 0x7f20c4148590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 -- 172.21.15.71:0/1312214486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4150ce0 msgr2=0x7f20c40c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4150ce0 0x7f20c40c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.068+0000 7f20c88de640 1 -- 172.21.15.71:0/1312214486 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b8009460 con 0x7f20c40c58c0 2023-08-25T09:50:29.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.069+0000 7f20c88de640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 0x7f20c4148ea0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f20b802ba50 tx=0x7f20b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.069+0000 7f20c17fa640 1 -- 172.21.15.71:0/1312214486 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b803d070 con 0x7f20c40c58c0 2023-08-25T09:50:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.069+0000 7f20c17fa640 1 -- 172.21.15.71:0/1312214486 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b8004500 con 0x7f20c40c58c0 2023-08-25T09:50:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.069+0000 7f20c17fa640 1 -- 172.21.15.71:0/1312214486 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b802fdf0 con 0x7f20c40c58c0 2023-08-25T09:50:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.069+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c40cb170 con 0x7f20c40c58c0 2023-08-25T09:50:29.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.069+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c414ab50 con 0x7f20c40c58c0 2023-08-25T09:50:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.071+0000 7f20c17fa640 1 -- 172.21.15.71:0/1312214486 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f20b8003610 con 0x7f20c40c58c0 2023-08-25T09:50:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.072+0000 7f20c17fa640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20a806a080 0x7f20a806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.072+0000 7f20c17fa640 1 -- 172.21.15.71:0/1312214486 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f20b80aa780 con 0x7f20c40c58c0 2023-08-25T09:50:29.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.072+0000 7f20c3fff640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20a806a080 0x7f20a806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.073+0000 7f20c3fff640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20a806a080 0x7f20a806c540 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f20c40c6160 tx=0x7f20b00077c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.073+0000 7f20cd78a740 1 Processor -- start 2023-08-25T09:50:29.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.073+0000 7f20cd78a740 1 -- start start 2023-08-25T09:50:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905aeabd0 0x559905aeafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905aeb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905ae9630 0x559905b2a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20cd78a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559905a58190 con 0x559905b34a50 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x559905a55880 con 0x559905aeabd0 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x559905a51170 con 0x559905ae9630 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20c3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905aeb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.074+0000 7f20c3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905aeb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51116/0 (socket says 172.21.15.71:51116) 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 -- 172.21.15.71:0/1099713909 learned_addr learned my addr 172.21.15.71:0/1099713909 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c90df640 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905ae9630 0x559905b2a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c88de640 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905aeabd0 0x559905aeafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 -- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905ae9630 msgr2=0x559905b2a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905ae9630 0x559905b2a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 -- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905aeabd0 msgr2=0x559905aeafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905aeabd0 0x559905aeafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 -- 172.21.15.71:0/1099713909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559905a4f7f0 con 0x559905b34a50 2023-08-25T09:50:29.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20c3fff640 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905aeb4f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f20b0005500 tx=0x7f20b000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b2c2f787760c3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20ae7fc640 1 -- 172.21.15.71:0/1099713909 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b001b070 con 0x559905b34a50 2023-08-25T09:50:29.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.075+0000 7f20ae7fc640 1 -- 172.21.15.71:0/1099713909 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b0012650 con 0x559905b34a50 2023-08-25T09:50:29.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20ae7fc640 1 -- 172.21.15.71:0/1099713909 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b0012a00 con 0x559905b34a50 2023-08-25T09:50:29.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 -- 172.21.15.71:0/1099713909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 msgr2=0x559905aeb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905aeb4f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f20b0005500 tx=0x7f20b000f000 comp rx=0 tx=0).stop 2023-08-25T09:50:29.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 -- 172.21.15.71:0/1099713909 shutdown_connections 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905ae9630 0x559905b2a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905aeb4f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 --2- 172.21.15.71:0/1099713909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905aeabd0 0x559905aeafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 -- 172.21.15.71:0/1099713909 >> 172.21.15.71:0/1099713909 conn(0x559905ae7340 msgr2=0x559905ae9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 -- 172.21.15.71:0/1099713909 shutdown_connections 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.076+0000 7f20cd78a740 1 -- 172.21.15.71:0/1099713909 wait complete. 2023-08-25T09:50:29.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.077+0000 7f20cd78a740 1 -- start start 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.077+0000 7f20cd78a740 1 Processor -- start 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.078+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 0x559905b10070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.078+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905b34a50 0x559905b13970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.078+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905b3b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.078+0000 7f20cd78a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599059f0d20 con 0x559905aeabd0 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.078+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599059f09c0 con 0x559905b2beb0 2023-08-25T09:50:29.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.078+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5599057b6f20 con 0x559905b34a50 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20cd78a740 10 client.0 _ll_register_callbacks cb 0x559905a08630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 0x559905b10070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 0x559905b10070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51118/0 (socket says 172.21.15.71:51118) 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 -- 172.21.15.71:0/4243967199 learned_addr learned my addr 172.21.15.71:0/4243967199 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 -- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905b34a50 msgr2=0x559905b13970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905b34a50 0x559905b13970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 -- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 msgr2=0x559905b3b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c90df640 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905b3b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905b3b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 -- 172.21.15.71:0/4243967199 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b00051b0 con 0x559905aeabd0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20c88de640 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 0x559905b10070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f20b8000c00 tx=0x7f20b807d6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43155cfd5300abe server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.079+0000 7f20a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20a37fe640 1 -- 172.21.15.71:0/4243967199 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8078ee0 con 0x559905aeabd0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20a37fe640 1 -- 172.21.15.71:0/4243967199 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f20b807d380 con 0x559905aeabd0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559905a25540 con 0x559905aeabd0 2023-08-25T09:50:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20a37fe640 1 -- 172.21.15.71:0/4243967199 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20b8081070 con 0x559905aeabd0 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 10 client.4920 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559905a48f30 con 0x559905aeabd0 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20a37fe640 1 -- 172.21.15.71:0/4243967199 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559905a25540 con 0x559905aeabd0 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20a37fe640 1 -- 172.21.15.71:0/4243967199 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f20b80a8ba0 con 0x559905aeabd0 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 10 client.4920 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 10 client.4920 resolve_mds: resolved name 'a' to daemon mds.a 2023-08-25T09:50:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 20 client.4920 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b262e0 0x559905b4a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 4 client.4920 mds_command: new command op to 4287 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T09:50:29.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.080+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 --> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559905a4a150 con 0x559905b262e0 2023-08-25T09:50:29.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.081+0000 7f20c3fff640 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b262e0 0x559905b4a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.081+0000 7f20c3fff640 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b262e0 0x559905b4a280 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5d8c4dfcd67868b0 server_cookie=52026e5c732e96f5 in_seq=0 out_seq=0 2023-08-25T09:50:29.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.081+0000 7f20a37fe640 10 client.4920 ms_handle_connect on v2:172.21.15.71:6834/1556118065 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20a37fe640 1 -- 172.21.15.71:0/4243967199 <== mds.3 v2:172.21.15.71:6834/1556118065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559905a4a150 con 0x559905b262e0 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20a37fe640 10 client.4920 handle_command_reply: tid=0 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 client.4920 shutdown 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 msgr2=0x559905b10070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 0x559905b10070 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f20b8000c00 tx=0x7f20b807d6f0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 shutdown_connections 2023-08-25T09:50:29.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b262e0 0x559905b4a280 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905b3b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905b34a50 0x559905b13970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 --2- 172.21.15.71:0/4243967199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905aeabd0 0x559905b10070 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20a37fe640 0 client.4920 ms_handle_reset on v2:172.21.15.71:6834/1556118065 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 >> 172.21.15.71:0/4243967199 conn(0x559905ae9630 msgr2=0x559905ae4010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 shutdown_connections 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.082+0000 7f20cd78a740 1 -- 172.21.15.71:0/4243967199 wait complete. 2023-08-25T09:50:29.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.083+0000 7f20cd78a740 20 client.4920 trim_cache size 0 max 16384 2023-08-25T09:50:29.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.099+0000 7f20cd78a740 1 Processor -- start 2023-08-25T09:50:29.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.099+0000 7f20cd78a740 1 -- start start 2023-08-25T09:50:29.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 0x559905aea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b34a50 0x559905ae5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20cd78a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559905b123f0 con 0x559905b2beb0 2023-08-25T09:50:29.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x559905a530f0 con 0x559905b34a50 2023-08-25T09:50:29.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x559905ae4600 con 0x559905aeabd0 2023-08-25T09:50:29.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20c3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 0x559905aea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20c3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 0x559905aea1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51124/0 (socket says 172.21.15.71:51124) 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.100+0000 7f20c3fff640 1 -- 172.21.15.71:0/3261451557 learned_addr learned my addr 172.21.15.71:0/3261451557 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c88de640 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c90df640 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b34a50 0x559905ae5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c3fff640 1 -- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 msgr2=0x559905ae9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c3fff640 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c3fff640 1 -- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b34a50 msgr2=0x559905ae5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c3fff640 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b34a50 0x559905ae5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c3fff640 1 -- 172.21.15.71:0/3261451557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599059efa10 con 0x559905b2beb0 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20c3fff640 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 0x559905aea1c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f20b00054d0 tx=0x7f20b0005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7dc8b99a16a66d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20aeffd640 1 -- 172.21.15.71:0/3261451557 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b001b030 con 0x559905b2beb0 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20aeffd640 1 -- 172.21.15.71:0/3261451557 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b0028250 con 0x559905b2beb0 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.101+0000 7f20aeffd640 1 -- 172.21.15.71:0/3261451557 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b0027070 con 0x559905b2beb0 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 -- 172.21.15.71:0/3261451557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 msgr2=0x559905aea1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 0x559905aea1c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f20b00054d0 tx=0x7f20b0005000 comp rx=0 tx=0).stop 2023-08-25T09:50:29.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 -- 172.21.15.71:0/3261451557 shutdown_connections 2023-08-25T09:50:29.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b34a50 0x559905ae5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b2beb0 0x559905aea1c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 --2- 172.21.15.71:0/3261451557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 -- 172.21.15.71:0/3261451557 >> 172.21.15.71:0/3261451557 conn(0x559905ae9630 msgr2=0x559905b3b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 -- 172.21.15.71:0/3261451557 shutdown_connections 2023-08-25T09:50:29.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.102+0000 7f20cd78a740 1 -- 172.21.15.71:0/3261451557 wait complete. 2023-08-25T09:50:29.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.103+0000 7f20cd78a740 1 -- start start 2023-08-25T09:50:29.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.103+0000 7f20cd78a740 1 Processor -- start 2023-08-25T09:50:29.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905ae6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905b8e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x559905b97e60 con 0x559905b34a50 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x559905ae5750 con 0x559905b2beb0 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x559905b26f90 con 0x559905aeabd0 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20cd78a740 10 client.0 _ll_register_callbacks cb 0x7f20b0007eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20c90df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905b8e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20c90df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905b8e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51138/0 (socket says 172.21.15.71:51138) 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20c90df640 1 -- 172.21.15.71:0/84325529 learned_addr learned my addr 172.21.15.71:0/84325529 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20c3fff640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905ae6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.104+0000 7f20c88de640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c90df640 1 -- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 msgr2=0x559905ae5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c90df640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c90df640 1 -- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 msgr2=0x559905ae6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c90df640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905ae6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c90df640 1 -- 172.21.15.71:0/84325529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559905a4f7f0 con 0x559905b34a50 2023-08-25T09:50:29.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c3fff640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905ae6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20c90df640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905b8e560 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x559905ae6c90 tx=0x7f20bc007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ddd48a75e506dd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20a37fe640 1 -- 172.21.15.71:0/84325529 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20bc009e50 con 0x559905b34a50 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20b0012650 con 0x559905b34a50 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20a37fe640 1 -- 172.21.15.71:0/84325529 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f20bc011910 con 0x559905b34a50 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.105+0000 7f20a37fe640 1 -- 172.21.15.71:0/84325529 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20bc011ad0 con 0x559905b34a50 2023-08-25T09:50:29.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 10 client.4926 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x559905a14de0 con 0x559905b34a50 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20a37fe640 1 -- 172.21.15.71:0/84325529 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b0012650 con 0x559905b34a50 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20a37fe640 1 -- 172.21.15.71:0/84325529 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f20bc014070 con 0x559905b34a50 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 10 client.4926 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 10 client.4926 resolve_mds: resolved name 'a' to daemon mds.a 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 20 client.4926 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b2a570 0x559905b2a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 4 client.4926 mds_command: new command op to 4287 tid=0[{"prefix": "status"}] 2023-08-25T09:50:29.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 --> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5599057005e0 con 0x559905b2a570 2023-08-25T09:50:29.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.106+0000 7f20c88de640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b2a570 0x559905b2a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20c88de640 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b2a570 0x559905b2a950 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=79f1ca39fa63b3b server_cookie=c7e93e9a24784410 in_seq=0 out_seq=0 2023-08-25T09:50:29.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20a37fe640 10 client.4926 ms_handle_connect on v2:172.21.15.71:6834/1556118065 2023-08-25T09:50:29.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20a37fe640 1 -- 172.21.15.71:0/84325529 <== mds.3 v2:172.21.15.71:6834/1556118065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5599057005e0 con 0x559905b2a570 2023-08-25T09:50:29.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20a37fe640 10 client.4926 handle_command_reply: tid=0 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 client.4926 shutdown 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 msgr2=0x559905b8e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905b8e560 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x559905ae6c90 tx=0x7f20bc007db0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 shutdown_connections 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:6834/1556118065,v1:172.21.15.71:6836/1556118065] conn(0x559905b2a570 0x559905b2a950 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559905b2beb0 0x559905ae6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559905b34a50 0x559905b8e560 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 --2- 172.21.15.71:0/84325529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x559905aeabd0 0x559905ae5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20a37fe640 0 client.4926 ms_handle_reset on v2:172.21.15.71:6834/1556118065 2023-08-25T09:50:29.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.107+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 >> 172.21.15.71:0/84325529 conn(0x559905ae9630 msgr2=0x559905aeafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 shutdown_connections 2023-08-25T09:50:29.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cd78a740 1 -- 172.21.15.71:0/84325529 wait complete. 2023-08-25T09:50:29.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cd78a740 20 client.4926 trim_cache size 0 max 16384 2023-08-25T09:50:29.122 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:50:29.122 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "d3b06f22-7372-48e3-a060-a2382f01af3f", 2023-08-25T09:50:29.122 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 3, 2023-08-25T09:50:29.122 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4287, 2023-08-25T09:50:29.122 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2023-08-25T09:50:29.122 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 16.562092182000001, 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 17, 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 31, 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 31, 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 61.382215791999997 2023-08-25T09:50:29.123 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:50:29.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20a806a080 msgr2=0x7f20a806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20a806a080 0x7f20a806c540 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f20c40c6160 tx=0x7f20b00077c0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 msgr2=0x7f20c4148ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 0x7f20c4148ea0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f20b802ba50 tx=0x7f20b80046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 shutdown_connections 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4150ce0 0x7f20c40c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20a806a080 0x7f20a806c540 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40c76b0 0x7f20c4148590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 --2- 172.21.15.71:0/1312214486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20c40c58c0 0x7f20c4148ea0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.108+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 >> 172.21.15.71:0/1312214486 conn(0x7f20c406bff0 msgr2=0x7f20c4069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.109+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 shutdown_connections 2023-08-25T09:50:29.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.109+0000 7f20cab42640 1 -- 172.21.15.71:0/1312214486 wait complete. 2023-08-25T09:50:29.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-08-25T09:50:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.338+0000 7f3b615c2640 1 Processor -- start 2023-08-25T09:50:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.338+0000 7f3b615c2640 1 -- start start 2023-08-25T09:50:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b615c2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b5c152be0 0x7f3b5c06aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b615c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c0c4c10 0x7f3b5c069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b615c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c159730 con 0x7f3b5c069d90 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b615c2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c159c40 con 0x7f3b5c152be0 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b615c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c15d190 con 0x7f3b5c0c4c10 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b5affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b5c152be0 0x7f3b5c06aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b5affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3b5c152be0 0x7f3b5c06aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51154/0 (socket says 172.21.15.71:51154) 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b5affd640 1 -- 172.21.15.71:0/751045709 learned_addr learned my addr 172.21.15.71:0/751045709 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b527fc640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c0c4c10 0x7f3b5c069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.340+0000 7f3b5affd640 1 -- 172.21.15.71:0/751045709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c0c4c10 msgr2=0x7f3b5c069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.341+0000 7f3b5affd640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c0c4c10 0x7f3b5c069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.341+0000 7f3b5affd640 1 -- 172.21.15.71:0/751045709 >> v1:172.21.15.164:6789/0 conn(0x7f3b5c069d90 legacy=0x7f3b5c0c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.341+0000 7f3b5affd640 1 -- 172.21.15.71:0/751045709 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c159960 con 0x7f3b5c152be0 2023-08-25T09:50:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.341+0000 7f3b527fc640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c0c4c10 0x7f3b5c069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.342+0000 7f3b5affd640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c06aa60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009860 tx=0x7f3b4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f93ac5731e0f9d6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.342+0000 7f3b5a7fc640 1 -- 172.21.15.71:0/751045709 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c031640 con 0x7f3b5c152be0 2023-08-25T09:50:29.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.342+0000 7f3b5a7fc640 1 -- 172.21.15.71:0/751045709 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c0317a0 con 0x7f3b5c152be0 2023-08-25T09:50:29.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.342+0000 7f3b5a7fc640 1 -- 172.21.15.71:0/751045709 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c031b30 con 0x7f3b5c152be0 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 -- 172.21.15.71:0/751045709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 msgr2=0x7f3b5c06aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c06aa60 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009860 tx=0x7f3b4c02b8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 -- 172.21.15.71:0/751045709 shutdown_connections 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c0c4c10 0x7f3b5c069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 --2- 172.21.15.71:0/751045709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c06aa60 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 -- 172.21.15.71:0/751045709 >> 172.21.15.71:0/751045709 conn(0x7f3b5c06bff0 msgr2=0x7f3b5c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 -- 172.21.15.71:0/751045709 shutdown_connections 2023-08-25T09:50:29.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.343+0000 7f3b615c2640 1 -- 172.21.15.71:0/751045709 wait complete. 2023-08-25T09:50:29.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.344+0000 7f3b615c2640 1 Processor -- start 2023-08-25T09:50:29.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.345+0000 7f3b615c2640 1 -- start start 2023-08-25T09:50:29.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b615c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c069d90 0x7f3b5c1ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b615c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b5c0c4c10 0x7f3b5c1547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b615c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b615c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c1f1950 con 0x7f3b5c152be0 2023-08-25T09:50:29.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b615c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c1f21f0 con 0x7f3b5c0c4c10 2023-08-25T09:50:29.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b615c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0c97a0 con 0x7f3b5c069d90 2023-08-25T09:50:29.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51160/0 (socket says 172.21.15.71:51160) 2023-08-25T09:50:29.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/282811290 learned_addr learned my addr 172.21.15.71:0/282811290 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5affd640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c069d90 0x7f3b5c1ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b527fc640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b5c0c4c10 0x7f3b5c1547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c069d90 msgr2=0x7f3b5c1ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c069d90 0x7f3b5c1ee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b5c0c4c10 msgr2=0x7f3b5c1547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b5c0c4c10 0x7f3b5c1547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.346+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/282811290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c009470 con 0x7f3b5c152be0 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.347+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c156c60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400be60 tx=0x7f3b5400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.347+0000 7f3b53fff640 1 -- 172.21.15.71:0/282811290 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54014440 con 0x7f3b5c152be0 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.347+0000 7f3b53fff640 1 -- 172.21.15.71:0/282811290 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b540145a0 con 0x7f3b5c152be0 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.347+0000 7f3b53fff640 1 -- 172.21.15.71:0/282811290 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54014990 con 0x7f3b5c152be0 2023-08-25T09:50:29.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.347+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c0c9a80 con 0x7f3b5c152be0 2023-08-25T09:50:29.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.347+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c1f1e80 con 0x7f3b5c152be0 2023-08-25T09:50:29.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.350+0000 7f3b53fff640 1 -- 172.21.15.71:0/282811290 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3b5401d3b0 con 0x7f3b5c152be0 2023-08-25T09:50:29.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.351+0000 7f3b53fff640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b4006a050 0x7f3b4006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.351+0000 7f3b53fff640 1 -- 172.21.15.71:0/282811290 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3b54088ad0 con 0x7f3b5c152be0 2023-08-25T09:50:29.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.351+0000 7f3b5affd640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b4006a050 0x7f3b4006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.351+0000 7f3b6420f740 1 Processor -- start 2023-08-25T09:50:29.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.351+0000 7f3b6420f740 1 -- start start 2023-08-25T09:50:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b5affd640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b4006a050 0x7f3b4006c510 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c036000 tx=0x7f3b4c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x564601632fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x5646016334f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601631630 0x564601672550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b6420f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646015a0190 con 0x56460167ca50 2023-08-25T09:50:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x56460159d880 con 0x564601631630 2023-08-25T09:50:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.352+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x564601599170 con 0x564601632bd0 2023-08-25T09:50:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x5646016334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x5646016334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51172/0 (socket says 172.21.15.71:51172) 2023-08-25T09:50:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 -- 172.21.15.71:0/140058770 learned_addr learned my addr 172.21.15.71:0/140058770 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b5affd640 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x564601632fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601631630 0x564601672550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 -- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 msgr2=0x564601632fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x564601632fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 -- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601631630 msgr2=0x564601672550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601631630 0x564601672550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 -- 172.21.15.71:0/140058770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646015977f0 con 0x56460167ca50 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601631630 0x564601672550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.353+0000 7f3b527fc640 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x5646016334f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400d900 tx=0x7f3b4400ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d871624a9615f78a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b50ff9640 1 -- 172.21.15.71:0/140058770 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400fa80 con 0x56460167ca50 2023-08-25T09:50:29.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b50ff9640 1 -- 172.21.15.71:0/140058770 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4400c030 con 0x56460167ca50 2023-08-25T09:50:29.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b50ff9640 1 -- 172.21.15.71:0/140058770 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400fde0 con 0x56460167ca50 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 -- 172.21.15.71:0/140058770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 msgr2=0x5646016334f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x5646016334f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400d900 tx=0x7f3b4400ddc0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 -- 172.21.15.71:0/140058770 shutdown_connections 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601631630 0x564601672550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x5646016334f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 --2- 172.21.15.71:0/140058770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x564601632fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 -- 172.21.15.71:0/140058770 >> 172.21.15.71:0/140058770 conn(0x56460162f340 msgr2=0x564601631c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.354+0000 7f3b6420f740 1 -- 172.21.15.71:0/140058770 shutdown_connections 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.355+0000 7f3b6420f740 1 -- 172.21.15.71:0/140058770 wait complete. 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.355+0000 7f3b6420f740 1 -- start start 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.355+0000 7f3b6420f740 1 Processor -- start 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x56460167b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x56460166ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x5646016870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x564601538d20 con 0x56460167ca50 2023-08-25T09:50:29.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646015389c0 con 0x564601686c20 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646012fef20 con 0x564601632bd0 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.356+0000 7f3b6420f740 10 client.0 _ll_register_callbacks cb 0x564601550630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x56460166ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x56460166ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51188/0 (socket says 172.21.15.71:51188) 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 -- 172.21.15.71:0/593746453 learned_addr learned my addr 172.21.15.71:0/593746453 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b5affd640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x56460167b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 -- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 msgr2=0x56460167b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x5646016870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x56460167b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 -- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 msgr2=0x5646016870a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x5646016870a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 -- 172.21.15.71:0/593746453 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4400e040 con 0x56460167ca50 2023-08-25T09:50:29.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x5646016870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b527fc640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x56460166ef40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400f5b0 tx=0x7f3b44012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596091d9c08e194b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.357+0000 7f3b49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b49ffb640 1 -- 172.21.15.71:0/593746453 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44021070 con 0x56460167ca50 2023-08-25T09:50:29.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b49ffb640 1 -- 172.21.15.71:0/593746453 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3b44004260 con 0x56460167ca50 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56460156d540 con 0x56460167ca50 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b49ffb640 1 -- 172.21.15.71:0/593746453 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b44004420 con 0x56460167ca50 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 10 client.4938 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564601590f30 con 0x56460167ca50 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b49ffb640 1 -- 172.21.15.71:0/593746453 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56460156d540 con 0x56460167ca50 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b49ffb640 1 -- 172.21.15.71:0/593746453 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f3b44003610 con 0x56460167ca50 2023-08-25T09:50:29.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 10 client.4938 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 10 client.4938 resolve_mds: resolved name 'c' to daemon mds.c 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 20 client.4938 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x5646016838c0 0x564601683ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 4 client.4938 mds_command: new command op to 4290 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.358+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 --> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564601592150 con 0x5646016838c0 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.359+0000 7f3b5affd640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x5646016838c0 0x564601683ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.359+0000 7f3b5affd640 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x5646016838c0 0x564601683ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6bc479e4b12d63b server_cookie=c98d309782b0f186 in_seq=0 out_seq=0 2023-08-25T09:50:29.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.359+0000 7f3b49ffb640 10 client.4938 ms_handle_connect on v2:172.21.15.71:6835/3404832976 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b49ffb640 1 -- 172.21.15.71:0/593746453 <== mds.2 v2:172.21.15.71:6835/3404832976 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564601592150 con 0x5646016838c0 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b49ffb640 10 client.4938 handle_command_reply: tid=0 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 client.4938 shutdown 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 msgr2=0x56460166ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x56460166ef40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400f5b0 tx=0x7f3b44012000 comp rx=0 tx=0).stop 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 shutdown_connections 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x5646016838c0 0x564601683ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x5646016870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56460167ca50 0x56460166ef40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 --2- 172.21.15.71:0/593746453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x564601632bd0 0x56460167b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b49ffb640 0 client.4938 ms_handle_reset on v2:172.21.15.71:6835/3404832976 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.360+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 >> 172.21.15.71:0/593746453 conn(0x564601631630 msgr2=0x56460162c010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.361+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 shutdown_connections 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.361+0000 7f3b6420f740 1 -- 172.21.15.71:0/593746453 wait complete. 2023-08-25T09:50:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.361+0000 7f3b6420f740 20 client.4938 trim_cache size 0 max 16384 2023-08-25T09:50:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.377+0000 7f3b6420f740 1 Processor -- start 2023-08-25T09:50:29.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 -- start start 2023-08-25T09:50:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601632bd0 0x56460162c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x564601632fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 0x5646016334f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x564601670390 con 0x564601686c20 2023-08-25T09:50:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x564601598310 con 0x564601632bd0 2023-08-25T09:50:29.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.378+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x564601676330 con 0x56460167ca50 2023-08-25T09:50:29.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 0x5646016334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 0x5646016334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51202/0 (socket says 172.21.15.71:51202) 2023-08-25T09:50:29.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/2358697187 learned_addr learned my addr 172.21.15.71:0/2358697187 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b527fc640 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x564601632fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5affd640 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601632bd0 0x56460162c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 msgr2=0x564601632fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x564601632fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601632bd0 msgr2=0x56460162c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601632bd0 0x56460162c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 -- 172.21.15.71:0/2358697187 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564601537a10 con 0x564601686c20 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.379+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 0x5646016334f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400be30 tx=0x7f3b54057720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85f646ffbfc18de server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b517fa640 1 -- 172.21.15.71:0/2358697187 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54055e50 con 0x564601686c20 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b517fa640 1 -- 172.21.15.71:0/2358697187 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b540541c0 con 0x564601686c20 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b517fa640 1 -- 172.21.15.71:0/2358697187 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5405f550 con 0x564601686c20 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 -- 172.21.15.71:0/2358697187 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 msgr2=0x5646016334f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 0x5646016334f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400be30 tx=0x7f3b54057720 comp rx=0 tx=0).stop 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 -- 172.21.15.71:0/2358697187 shutdown_connections 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601686c20 0x5646016334f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x564601632fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2358697187 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601632bd0 0x56460162c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 -- 172.21.15.71:0/2358697187 >> 172.21.15.71:0/2358697187 conn(0x564601631630 msgr2=0x564601685cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 -- 172.21.15.71:0/2358697187 shutdown_connections 2023-08-25T09:50:29.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.380+0000 7f3b6420f740 1 -- 172.21.15.71:0/2358697187 wait complete. 2023-08-25T09:50:29.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.381+0000 7f3b6420f740 1 -- start start 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.381+0000 7f3b6420f740 1 Processor -- start 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 0x5646016df170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x5646016e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x564601674e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646016e5320 con 0x564601632bd0 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646016e5490 con 0x564601686c20 2023-08-25T09:50:29.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x564601674ca0 con 0x56460167ca50 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b6420f740 10 client.0 _ll_register_callbacks cb 0x7f3b4400d280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b5affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 0x5646016df170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b5affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 0x5646016df170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51210/0 (socket says 172.21.15.71:51210) 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.382+0000 7f3b5affd640 1 -- 172.21.15.71:0/2979849002 learned_addr learned my addr 172.21.15.71:0/2979849002 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5b7fe640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x564601674e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b527fc640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x5646016e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5affd640 1 -- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 msgr2=0x5646016e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5affd640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x5646016e4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5affd640 1 -- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 msgr2=0x564601674e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5affd640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x564601674e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5affd640 1 -- 172.21.15.71:0/2979849002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c1f1e80 con 0x564601632bd0 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b5affd640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 0x5646016df170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009980 tx=0x7f3b4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4578d282ea0a9e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b49ffb640 1 -- 172.21.15.71:0/2979849002 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c04a070 con 0x564601632bd0 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b4400c030 con 0x564601632bd0 2023-08-25T09:50:29.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.383+0000 7f3b49ffb640 1 -- 172.21.15.71:0/2979849002 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3b4c03f070 con 0x564601632bd0 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b49ffb640 1 -- 172.21.15.71:0/2979849002 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b4c03a420 con 0x564601632bd0 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 10 client.4944 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x56460155cde0 con 0x564601632bd0 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b49ffb640 1 -- 172.21.15.71:0/2979849002 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4400c030 con 0x564601632bd0 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b49ffb640 1 -- 172.21.15.71:0/2979849002 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f3b4c03a580 con 0x564601632bd0 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 10 client.4944 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 10 client.4944 resolve_mds: resolved name 'c' to daemon mds.c 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 20 client.4944 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x56460166d300 0x56460166d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 4 client.4944 mds_command: new command op to 4290 tid=0[{"prefix": "status"}] 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 --> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5646012485e0 con 0x56460166d300 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.384+0000 7f3b527fc640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x56460166d300 0x56460166d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b527fc640 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x56460166d300 0x56460166d6e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76085a4c64a100ee server_cookie=3d81f7244909e071 in_seq=0 out_seq=0 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b49ffb640 10 client.4944 ms_handle_connect on v2:172.21.15.71:6835/3404832976 2023-08-25T09:50:29.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b49ffb640 1 -- 172.21.15.71:0/2979849002 <== mds.2 v2:172.21.15.71:6835/3404832976 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5646012485e0 con 0x56460166d300 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b49ffb640 10 client.4944 handle_command_reply: tid=0 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 client.4944 shutdown 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 msgr2=0x5646016df170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 0x5646016df170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009980 tx=0x7f3b4c004730 comp rx=0 tx=0).stop 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 shutdown_connections 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:6835/3404832976,v1:172.21.15.71:6837/3404832976] conn(0x56460166d300 0x56460166d6e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564601686c20 0x564601674e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x56460167ca50 0x5646016e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 --2- 172.21.15.71:0/2979849002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564601632bd0 0x5646016df170 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b49ffb640 0 client.4944 ms_handle_reset on v2:172.21.15.71:6835/3404832976 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.385+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 >> 172.21.15.71:0/2979849002 conn(0x564601631630 msgr2=0x564601685cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 shutdown_connections 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b6420f740 1 -- 172.21.15.71:0/2979849002 wait complete. 2023-08-25T09:50:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b6420f740 20 client.4944 trim_cache size 0 max 16384 2023-08-25T09:50:29.396 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:50:29.396 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "d3b06f22-7372-48e3-a060-a2382f01af3f", 2023-08-25T09:50:29.396 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 2, 2023-08-25T09:50:29.396 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4290, 2023-08-25T09:50:29.396 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2023-08-25T09:50:29.396 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 22.662848434000001, 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 17, 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 31, 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 31, 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 61.658022146 2023-08-25T09:50:29.397 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:50:29.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b4006a050 msgr2=0x7f3b4006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b4006a050 0x7f3b4006c510 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c036000 tx=0x7f3b4c03c040 comp rx=0 tx=0).stop 2023-08-25T09:50:29.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 msgr2=0x7f3b5c156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c156c60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3b5400be60 tx=0x7f3b5400bf50 comp rx=0 tx=0).stop 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 shutdown_connections 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b5c0c4c10 0x7f3b5c1547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3b4006a050 0x7f3b4006c510 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b5c152be0 0x7f3b5c156c60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 --2- 172.21.15.71:0/282811290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b5c069d90 0x7f3b5c1ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.386+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 >> 172.21.15.71:0/282811290 conn(0x7f3b5c06bff0 msgr2=0x7f3b5c150400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.387+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 shutdown_connections 2023-08-25T09:50:29.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.387+0000 7f3b615c2640 1 -- 172.21.15.71:0/282811290 wait complete. 2023-08-25T09:50:29.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-08-25T09:50:29.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.619+0000 7f1ba2922640 1 Processor -- start 2023-08-25T09:50:29.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.620+0000 7f1ba2922640 1 -- start start 2023-08-25T09:50:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.621+0000 7f1ba2922640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b9c0c56c0 0x7f1b9c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.621+0000 7f1ba2922640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c157bf0 con 0x7f1b9c1542d0 2023-08-25T09:50:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.621+0000 7f1ba2922640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c06a130 con 0x7f1b9c069850 2023-08-25T09:50:29.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.621+0000 7f1ba2922640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9c0c4b60 con 0x7f1b9c0c56c0 2023-08-25T09:50:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba1920640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1b9c1542d0 0x7f1b9c0c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36844/0 (socket says 172.21.15.71:36844) 2023-08-25T09:50:29.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba1920640 1 -- 172.21.15.71:0/4132744030 learned_addr learned my addr 172.21.15.71:0/4132744030 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba2121640 1 --2- 172.21.15.71:0/4132744030 >> v2:172.21.15.164:3300/0 conn(0x7f1b9c0c56c0 0x7f1b9c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183132123 0 0) 0x7f1b98001110 con 0x7f1b9c1542d0 2023-08-25T09:50:29.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b88003520 con 0x7f1b9c1542d0 2023-08-25T09:50:29.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4183673931 0 0) 0x7f1b88003520 con 0x7f1b9c1542d0 2023-08-25T09:50:29.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 >> v2:172.21.15.164:3300/0 conn(0x7f1b9c0c56c0 msgr2=0x7f1b9c1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 --2- 172.21.15.71:0/4132744030 >> v2:172.21.15.164:3300/0 conn(0x7f1b9c0c56c0 0x7f1b9c1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 >> v1:172.21.15.164:6789/0 conn(0x7f1b9c069850 legacy=0x7f1b9c0c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c1546b0 con 0x7f1b9c1542d0 2023-08-25T09:50:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.622+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f1b98001800 con 0x7f1b9c1542d0 2023-08-25T09:50:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 >> v1:172.21.15.71:6789/0 conn(0x7f1b9c1542d0 legacy=0x7f1b9c0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b880039e0 0x7f1b88003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b880047c0 0x7f1b88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b8800fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0c4b60 con 0x7f1b88005180 2023-08-25T09:50:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b880104c0 con 0x7f1b880047c0 2023-08-25T09:50:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88010680 con 0x7f1b880039e0 2023-08-25T09:50:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba2121640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b8800fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba111f640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b880047c0 0x7f1b88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba1920640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b880039e0 0x7f1b88003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba2121640 1 -- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b880039e0 msgr2=0x7f1b88003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba2121640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b880039e0 0x7f1b88003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba2121640 1 -- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b880047c0 msgr2=0x7f1b88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba2121640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b880047c0 0x7f1b88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.623+0000 7f1ba2121640 1 -- 172.21.15.71:0/4132744030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b88003520 con 0x7f1b88005180 2023-08-25T09:50:29.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.624+0000 7f1ba1920640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b880039e0 0x7f1b88003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.624+0000 7f1ba2121640 1 -- 172.21.15.71:0/4132744030 reap_dead start 2023-08-25T09:50:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba2121640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b8800fd90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009fc0 tx=0x7f1b880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f098ec95ceeaac server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9402cc50 con 0x7f1b88005180 2023-08-25T09:50:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9400f040 con 0x7f1b88005180 2023-08-25T09:50:29.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba091e640 1 -- 172.21.15.71:0/4132744030 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94003990 con 0x7f1b88005180 2023-08-25T09:50:29.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba2922640 1 -- 172.21.15.71:0/4132744030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 msgr2=0x7f1b8800fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba2922640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b8800fd90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009fc0 tx=0x7f1b880057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba2922640 1 -- 172.21.15.71:0/4132744030 shutdown_connections 2023-08-25T09:50:29.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba2922640 1 --2- 172.21.15.71:0/4132744030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b8800fd90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.625+0000 7f1ba2922640 1 -- 172.21.15.71:0/4132744030 >> 172.21.15.71:0/4132744030 conn(0x7f1b9c06c280 msgr2=0x7f1b9c06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.626+0000 7f1ba2922640 1 -- 172.21.15.71:0/4132744030 shutdown_connections 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.626+0000 7f1ba2922640 1 -- 172.21.15.71:0/4132744030 wait complete. 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.627+0000 7f1ba2922640 1 Processor -- start 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.627+0000 7f1ba2922640 1 -- start start 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.628+0000 7f1ba2922640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b9c0c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.628+0000 7f1ba2922640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c157d90 0x7f1b9c0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.628+0000 7f1ba2922640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b9c1ea390 0x7f1b9c1ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.628+0000 7f1ba2922640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0c1210 con 0x7f1b88005180 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.628+0000 7f1ba2922640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0c06a0 con 0x7f1b9c1ea390 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.628+0000 7f1ba2922640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c1ead80 con 0x7f1b9c157d90 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b9c0c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b9c0c0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51224/0 (socket says 172.21.15.71:51224) 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 -- 172.21.15.71:0/2824647053 learned_addr learned my addr 172.21.15.71:0/2824647053 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba2121640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b9c1ea390 0x7f1b9c1ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba111f640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c157d90 0x7f1b9c0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 -- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c157d90 msgr2=0x7f1b9c0ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c157d90 0x7f1b9c0ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 -- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b9c1ea390 msgr2=0x7f1b9c1ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b9c1ea390 0x7f1b9c1ea840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.629+0000 7f1ba1920640 1 -- 172.21.15.71:0/2824647053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c1ed620 con 0x7f1b88005180 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.630+0000 7f1ba1920640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b9c0c0cd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800d7a0 tx=0x7f1b9800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.630+0000 7f1b927fc640 1 -- 172.21.15.71:0/2824647053 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b980074e0 con 0x7f1b88005180 2023-08-25T09:50:29.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.630+0000 7f1b927fc640 1 -- 172.21.15.71:0/2824647053 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98007640 con 0x7f1b88005180 2023-08-25T09:50:29.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.630+0000 7f1b927fc640 1 -- 172.21.15.71:0/2824647053 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98004a00 con 0x7f1b88005180 2023-08-25T09:50:29.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.630+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c1ed8e0 con 0x7f1b88005180 2023-08-25T09:50:29.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.630+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c1edea0 con 0x7f1b88005180 2023-08-25T09:50:29.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.632+0000 7f1b927fc640 1 -- 172.21.15.71:0/2824647053 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1b98004b60 con 0x7f1b88005180 2023-08-25T09:50:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.634+0000 7f1b927fc640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b8006a050 0x7f1b8006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.634+0000 7f1b927fc640 1 -- 172.21.15.71:0/2824647053 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1b98088140 con 0x7f1b88005180 2023-08-25T09:50:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.634+0000 7f1ba111f640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b8006a050 0x7f1b8006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.635+0000 7f1ba67d5740 1 Processor -- start 2023-08-25T09:50:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.635+0000 7f1ba111f640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b8006a050 0x7f1b8006c510 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0c3be0 tx=0x7f1b8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.635+0000 7f1ba67d5740 1 -- start start 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b0071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b0076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b005820 0x55660b046740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660af74190 con 0x55660b006dc0 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660af71880 con 0x55660b005820 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55660af6d170 con 0x55660b050c40 2023-08-25T09:50:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b0071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b0071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51236/0 (socket says 172.21.15.71:51236) 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 -- 172.21.15.71:0/3168392487 learned_addr learned my addr 172.21.15.71:0/3168392487 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba2121640 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b005820 0x55660b046740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba111f640 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b0076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 -- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 msgr2=0x55660b0076e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b0076e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 -- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b005820 msgr2=0x55660b046740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b005820 0x55660b046740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba1920640 1 -- 172.21.15.71:0/3168392487 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55660af6b7f0 con 0x55660b006dc0 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.636+0000 7f1ba2121640 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b005820 0x55660b046740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.637+0000 7f1ba1920640 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b0071a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1b98001ac0 tx=0x7f1b980577e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298a162cab43dc45 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.637+0000 7f1b7f7fe640 1 -- 172.21.15.71:0/3168392487 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98004e10 con 0x55660b006dc0 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.637+0000 7f1b7f7fe640 1 -- 172.21.15.71:0/3168392487 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98055430 con 0x55660b006dc0 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.637+0000 7f1b7f7fe640 1 -- 172.21.15.71:0/3168392487 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b980537a0 con 0x55660b006dc0 2023-08-25T09:50:29.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 -- 172.21.15.71:0/3168392487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 msgr2=0x55660b0071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b0071a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1b98001ac0 tx=0x7f1b980577e0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 -- 172.21.15.71:0/3168392487 shutdown_connections 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b005820 0x55660b046740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b0076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/3168392487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b0071a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 -- 172.21.15.71:0/3168392487 >> 172.21.15.71:0/3168392487 conn(0x55660b003530 msgr2=0x55660b005e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.645 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 -- 172.21.15.71:0/3168392487 shutdown_connections 2023-08-25T09:50:29.646 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.638+0000 7f1ba67d5740 1 -- 172.21.15.71:0/3168392487 wait complete. 2023-08-25T09:50:29.648 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.639+0000 7f1ba67d5740 1 -- start start 2023-08-25T09:50:29.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.639+0000 7f1ba67d5740 1 Processor -- start 2023-08-25T09:50:29.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b001660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b041e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b02ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660af0cd20 con 0x55660b02a8d0 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660af0c9c0 con 0x55660b006dc0 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55660acd2f20 con 0x55660b050c40 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba67d5740 10 client.0 _ll_register_callbacks cb 0x55660af24630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b02ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b02ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51246/0 (socket says 172.21.15.71:51246) 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 -- 172.21.15.71:0/1006626492 learned_addr learned my addr 172.21.15.71:0/1006626492 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 -- 172.21.15.71:0/1006626492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 msgr2=0x55660b041e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b041e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 -- 172.21.15.71:0/1006626492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 msgr2=0x55660b001660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b001660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.640+0000 7f1ba2121640 1 -- 172.21.15.71:0/1006626492 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c1edea0 con 0x55660b02a8d0 2023-08-25T09:50:29.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1ba2121640 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b02ad50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400e230 tx=0x7f1b94004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771733464bf3a848 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1b7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1b7cff9640 1 -- 172.21.15.71:0/1006626492 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94003da0 con 0x55660b02a8d0 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55660af41540 con 0x55660b02a8d0 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1b7cff9640 1 -- 172.21.15.71:0/1006626492 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1b9402ce30 con 0x55660b02a8d0 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1b7cff9640 1 -- 172.21.15.71:0/1006626492 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b940346f0 con 0x55660b02a8d0 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1ba67d5740 10 client.4956 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.641+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55660af64f30 con 0x55660b02a8d0 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1b7cff9640 1 -- 172.21.15.71:0/1006626492 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55660af41540 con 0x55660b02a8d0 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1b7cff9640 1 -- 172.21.15.71:0/1006626492 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f1b9403c050 con 0x55660b02a8d0 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba67d5740 10 client.4956 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba67d5740 10 client.4956 resolve_mds: resolved name 'e' to daemon mds.e 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba67d5740 20 client.4956 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b02f440 0x55660b044920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba67d5740 4 client.4956 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T09:50:29.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 --> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55660af66150 con 0x55660b02f440 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.642+0000 7f1ba1920640 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b02f440 0x55660b044920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.643+0000 7f1ba1920640 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b02f440 0x55660b044920 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6dd0ab718610697 server_cookie=1bd1a694d2d8c442 in_seq=0 out_seq=0 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.643+0000 7f1b7cff9640 10 client.4956 ms_handle_connect on v2:172.21.15.71:6838/3369232025 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.643+0000 7f1b7cff9640 1 -- 172.21.15.71:0/1006626492 <== mds.1 v2:172.21.15.71:6838/3369232025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55660af66150 con 0x55660b02f440 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.643+0000 7f1b7cff9640 10 client.4956 handle_command_reply: tid=0 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.643+0000 7f1ba67d5740 1 client.4956 shutdown 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 msgr2=0x55660b02ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b02ad50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1b9400e230 tx=0x7f1b94004ab0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 shutdown_connections 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b02f440 0x55660b044920 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b02ad50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b041e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1006626492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b001660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1b7cff9640 0 client.4956 ms_handle_reset on v2:172.21.15.71:6838/3369232025 2023-08-25T09:50:29.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 >> 172.21.15.71:0/1006626492 conn(0x55660b005820 msgr2=0x55660b008680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 shutdown_connections 2023-08-25T09:50:29.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1006626492 wait complete. 2023-08-25T09:50:29.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.644+0000 7f1ba67d5740 20 client.4956 trim_cache size 0 max 16384 2023-08-25T09:50:29.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.660+0000 7f1ba67d5740 1 Processor -- start 2023-08-25T09:50:29.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 -- start start 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b02da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b02a8d0 0x55660b042ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b050c40 0x55660b043420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660afa0a60 con 0x55660b006dc0 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660b02edc0 con 0x55660b050c40 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.661+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55660affd2e0 con 0x55660b02a8d0 2023-08-25T09:50:29.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b02da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b02da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51260/0 (socket says 172.21.15.71:51260) 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 -- 172.21.15.71:0/1038600850 learned_addr learned my addr 172.21.15.71:0/1038600850 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba111f640 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b02a8d0 0x55660b042ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 -- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b02a8d0 msgr2=0x55660b042ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b02a8d0 0x55660b042ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 -- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b050c40 msgr2=0x55660b043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b050c40 0x55660b043420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 -- 172.21.15.71:0/1038600850 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55660af0ba10 con 0x55660b006dc0 2023-08-25T09:50:29.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.662+0000 7f1ba1920640 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b02da00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b980578d0 tx=0x7f1b9801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47f60bc1b5594cd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1b7ffff640 1 -- 172.21.15.71:0/1038600850 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98056bd0 con 0x55660b006dc0 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1b7ffff640 1 -- 172.21.15.71:0/1038600850 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9806e120 con 0x55660b006dc0 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1b7ffff640 1 -- 172.21.15.71:0/1038600850 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9806e290 con 0x55660b006dc0 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1038600850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 msgr2=0x55660b02da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b02da00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b980578d0 tx=0x7f1b9801e000 comp rx=0 tx=0).stop 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1038600850 shutdown_connections 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b050c40 0x55660b043420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b02a8d0 0x55660b042ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/1038600850 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b006dc0 0x55660b02da00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1038600850 >> 172.21.15.71:0/1038600850 conn(0x55660b005820 msgr2=0x55660b06a590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1038600850 shutdown_connections 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.663+0000 7f1ba67d5740 1 -- 172.21.15.71:0/1038600850 wait complete. 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.664+0000 7f1ba67d5740 1 -- start start 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.664+0000 7f1ba67d5740 1 Processor -- start 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b0aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b05a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b05a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660b059fd0 con 0x55660b02a8d0 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660b043690 con 0x55660b006dc0 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x55660b0421e0 con 0x55660b050c40 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba67d5740 10 client.0 _ll_register_callbacks cb 0x7f1b980533d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.665+0000 7f1ba111f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b05a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b05a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51268/0 (socket says 172.21.15.71:51268) 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 -- 172.21.15.71:0/2793357358 learned_addr learned my addr 172.21.15.71:0/2793357358 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba1920640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b0aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba2121640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b05a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 -- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 msgr2=0x55660b05a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b05a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 -- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 msgr2=0x55660b0aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b0aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 -- 172.21.15.71:0/2793357358 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55660af6b7f0 con 0x55660b02a8d0 2023-08-25T09:50:29.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1ba111f640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b05a450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c007fd0 tx=0x7f1b8c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f13ec12cdaa4ef server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1b7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.666+0000 7f1b7cff9640 1 -- 172.21.15.71:0/2793357358 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8c026030 con 0x55660b02a8d0 2023-08-25T09:50:29.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1b7cff9640 1 -- 172.21.15.71:0/2793357358 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1b8c027070 con 0x55660b02a8d0 2023-08-25T09:50:29.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b98055430 con 0x55660b02a8d0 2023-08-25T09:50:29.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1b7cff9640 1 -- 172.21.15.71:0/2793357358 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b8c014c10 con 0x55660b02a8d0 2023-08-25T09:50:29.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 10 client.4962 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55660af30de0 con 0x55660b02a8d0 2023-08-25T09:50:29.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1b7cff9640 1 -- 172.21.15.71:0/2793357358 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8c031020 con 0x55660b02a8d0 2023-08-25T09:50:29.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1b7cff9640 1 -- 172.21.15.71:0/2793357358 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f1b8c02e670 con 0x55660b02a8d0 2023-08-25T09:50:29.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 10 client.4962 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 10 client.4962 resolve_mds: resolved name 'e' to daemon mds.e 2023-08-25T09:50:29.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 20 client.4962 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b0512a0 0x55660b051680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 4 client.4962 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.667+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 --> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55660ac1c5e0 con 0x55660b0512a0 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba1920640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b0512a0 0x55660b051680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba1920640 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b0512a0 0x55660b051680 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71004dced1e926fe server_cookie=2a864ef5853b2262 in_seq=0 out_seq=0 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1b7cff9640 10 client.4962 ms_handle_connect on v2:172.21.15.71:6838/3369232025 2023-08-25T09:50:29.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1b7cff9640 1 -- 172.21.15.71:0/2793357358 <== mds.1 v2:172.21.15.71:6838/3369232025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55660ac1c5e0 con 0x55660b0512a0 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1b7cff9640 10 client.4962 handle_command_reply: tid=0 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 client.4962 shutdown 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 msgr2=0x55660b05a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b05a450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c007fd0 tx=0x7f1b8c001e50 comp rx=0 tx=0).stop 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 shutdown_connections 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:6838/3369232025,v1:172.21.15.71:6839/3369232025] conn(0x55660b0512a0 0x55660b051680 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55660b050c40 0x55660b05a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55660b02a8d0 0x55660b05a450 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1b7cff9640 0 client.4962 ms_handle_reset on v2:172.21.15.71:6838/3369232025 2023-08-25T09:50:29.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 --2- 172.21.15.71:0/2793357358 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55660b006dc0 0x55660b0aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.668+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 >> 172.21.15.71:0/2793357358 conn(0x55660b005820 msgr2=0x55660b05b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 shutdown_connections 2023-08-25T09:50:29.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba67d5740 1 -- 172.21.15.71:0/2793357358 wait complete. 2023-08-25T09:50:29.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba67d5740 20 client.4962 trim_cache size 0 max 16384 2023-08-25T09:50:29.680 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:50:29.680 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "d3b06f22-7372-48e3-a060-a2382f01af3f", 2023-08-25T09:50:29.680 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 1, 2023-08-25T09:50:29.680 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4293, 2023-08-25T09:50:29.680 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2023-08-25T09:50:29.680 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 29.157302349999998, 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 17, 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 31, 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 31, 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 61.929769542000003 2023-08-25T09:50:29.681 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:50:29.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b8006a050 msgr2=0x7f1b8006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba2922640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b8006a050 0x7f1b8006c510 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0c3be0 tx=0x7f1b8c009040 comp rx=0 tx=0).stop 2023-08-25T09:50:29.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 msgr2=0x7f1b9c0c0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba2922640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b9c0c0cd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800d7a0 tx=0x7f1b9800dc60 comp rx=0 tx=0).stop 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.669+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 shutdown_connections 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b9c1ea390 0x7f1b9c1ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b8006a050 0x7f1b8006c510 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b9c157d90 0x7f1b9c0ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 --2- 172.21.15.71:0/2824647053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b88005180 0x7f1b9c0c0cd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 >> 172.21.15.71:0/2824647053 conn(0x7f1b9c06c280 msgr2=0x7f1b9c06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 shutdown_connections 2023-08-25T09:50:29.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.670+0000 7f1ba2922640 1 -- 172.21.15.71:0/2824647053 wait complete. 2023-08-25T09:50:29.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-08-25T09:50:29.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.887+0000 7f2db035d640 1 Processor -- start 2023-08-25T09:50:29.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.887+0000 7f2db035d640 1 -- start start 2023-08-25T09:50:29.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2db035d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2da8150ce0 0x7f2da80c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2db035d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c58c0 0x7f2da80c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2db035d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80c2ba0 con 0x7f2da80c76b0 2023-08-25T09:50:29.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2db035d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da80c1770 con 0x7f2da8150ce0 2023-08-25T09:50:29.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2db035d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80c1f30 con 0x7f2da80c58c0 2023-08-25T09:50:29.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2dae0f9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2da8150ce0 0x7f2da80c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2dae8fa640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f2da80c76b0 0x7f2da815a060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36856/0 (socket says 172.21.15.71:36856) 2023-08-25T09:50:29.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.889+0000 7f2dae8fa640 1 -- 172.21.15.71:0/847475070 learned_addr learned my addr 172.21.15.71:0/847475070 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:29.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.890+0000 7f2dae0f9640 1 -- 172.21.15.71:0/847475070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c58c0 msgr2=0x7f2da80c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.890+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/847475070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c58c0 0x7f2da80c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.890+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/847475070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c58c0 0x7f2da80c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.890+0000 7f2dae0f9640 1 -- 172.21.15.71:0/847475070 >> v1:172.21.15.71:6789/0 conn(0x7f2da80c76b0 legacy=0x7f2da815a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.890+0000 7f2dae0f9640 1 -- 172.21.15.71:0/847475070 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da80c6400 con 0x7f2da8150ce0 2023-08-25T09:50:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.890+0000 7f2dad0f7640 1 -- 172.21.15.71:0/847475070 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516038042 0 0) 0x7f2da0001110 con 0x7f2da80c76b0 2023-08-25T09:50:29.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.891+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/847475070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da8150ce0 0x7f2da80c5380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009de0 tx=0x7f2da402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6172dfc8f3b9f5a8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.891+0000 7f2dad0f7640 1 -- 172.21.15.71:0/847475070 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402cd30 con 0x7f2da8150ce0 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.891+0000 7f2dad0f7640 1 -- 172.21.15.71:0/847475070 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da402ce90 con 0x7f2da8150ce0 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.891+0000 7f2dad0f7640 1 -- 172.21.15.71:0/847475070 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4034660 con 0x7f2da8150ce0 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 -- 172.21.15.71:0/847475070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da8150ce0 msgr2=0x7f2da80c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 --2- 172.21.15.71:0/847475070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da8150ce0 0x7f2da80c5380 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009de0 tx=0x7f2da402bf80 comp rx=0 tx=0).stop 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 -- 172.21.15.71:0/847475070 shutdown_connections 2023-08-25T09:50:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 --2- 172.21.15.71:0/847475070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c58c0 0x7f2da80c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 --2- 172.21.15.71:0/847475070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da8150ce0 0x7f2da80c5380 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 -- 172.21.15.71:0/847475070 >> 172.21.15.71:0/847475070 conn(0x7f2da806bff0 msgr2=0x7f2da806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 -- 172.21.15.71:0/847475070 shutdown_connections 2023-08-25T09:50:29.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.892+0000 7f2db035d640 1 -- 172.21.15.71:0/847475070 wait complete. 2023-08-25T09:50:29.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.894+0000 7f2db035d640 1 Processor -- start 2023-08-25T09:50:29.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.894+0000 7f2db035d640 1 -- start start 2023-08-25T09:50:29.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2db035d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 0x7f2da81f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2db035d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c76b0 0x7f2da806f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2db035d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da8150ce0 0x7f2da806f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2db035d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8152600 con 0x7f2da80c58c0 2023-08-25T09:50:29.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2db035d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8070490 con 0x7f2da8150ce0 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2db035d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80c87a0 con 0x7f2da80c76b0 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 0x7f2da81f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 0x7f2da81f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51300/0 (socket says 172.21.15.71:51300) 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 -- 172.21.15.71:0/17072058 learned_addr learned my addr 172.21.15.71:0/17072058 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 -- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c76b0 msgr2=0x7f2da806f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c76b0 0x7f2da806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da8150ce0 0x7f2da806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 -- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da8150ce0 msgr2=0x7f2da806f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da8150ce0 0x7f2da806f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.895+0000 7f2dae0f9640 1 -- 172.21.15.71:0/17072058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da4009a70 con 0x7f2da80c58c0 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.896+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 0x7f2da81f1a20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009de0 tx=0x7f2da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.896+0000 7f2d9effd640 1 -- 172.21.15.71:0/17072058 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da403f680 con 0x7f2da80c58c0 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.896+0000 7f2d9effd640 1 -- 172.21.15.71:0/17072058 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da403f7e0 con 0x7f2da80c58c0 2023-08-25T09:50:29.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.896+0000 7f2d9effd640 1 -- 172.21.15.71:0/17072058 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da403fb90 con 0x7f2da80c58c0 2023-08-25T09:50:29.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.896+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da80c8230 con 0x7f2da80c58c0 2023-08-25T09:50:29.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.896+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da81f2370 con 0x7f2da80c58c0 2023-08-25T09:50:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.899+0000 7f2d9effd640 1 -- 172.21.15.71:0/17072058 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2da403b040 con 0x7f2da80c58c0 2023-08-25T09:50:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.900+0000 7f2d9effd640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d8c06a050 0x7f2d8c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.900+0000 7f2d9effd640 1 -- 172.21.15.71:0/17072058 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2da40ab8d0 con 0x7f2da80c58c0 2023-08-25T09:50:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.901+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d8c06a050 0x7f2d8c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.901+0000 7f2db2fb0740 1 Processor -- start 2023-08-25T09:50:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.901+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d8c06a050 0x7f2d8c06c510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2da80c89f0 tx=0x7f2d98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.901+0000 7f2db2fb0740 1 -- start start 2023-08-25T09:50:29.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2051fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a20524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2050630 0x5630a2091550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a1fbf190 con 0x5630a209ba50 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a1fbc880 con 0x5630a2050630 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630a1fb8170 con 0x5630a2051bd0 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.902+0000 7f2dad8f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a20524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a20524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51308/0 (socket says 172.21.15.71:51308) 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 -- 172.21.15.71:0/1054298293 learned_addr learned my addr 172.21.15.71:0/1054298293 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2050630 0x5630a2091550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 -- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 msgr2=0x5630a2051fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2051fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2051fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 -- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2050630 msgr2=0x5630a2091550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2050630 0x5630a2091550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 -- 172.21.15.71:0/1054298293 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630a1fb67f0 con 0x5630a209ba50 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2050630 0x5630a2091550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a20524f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800afd0 tx=0x7f2d98006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1093804c457312c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2d8bfff640 1 -- 172.21.15.71:0/1054298293 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98013e40 con 0x5630a209ba50 2023-08-25T09:50:29.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2d8bfff640 1 -- 172.21.15.71:0/1054298293 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9801e400 con 0x5630a209ba50 2023-08-25T09:50:29.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.903+0000 7f2d8bfff640 1 -- 172.21.15.71:0/1054298293 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9801e7b0 con 0x5630a209ba50 2023-08-25T09:50:29.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 -- 172.21.15.71:0/1054298293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 msgr2=0x5630a20524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a20524f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800afd0 tx=0x7f2d98006f90 comp rx=0 tx=0).stop 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 -- 172.21.15.71:0/1054298293 shutdown_connections 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2050630 0x5630a2091550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a20524f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/1054298293 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2051fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 -- 172.21.15.71:0/1054298293 >> 172.21.15.71:0/1054298293 conn(0x5630a204e340 msgr2=0x5630a2050c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 -- 172.21.15.71:0/1054298293 shutdown_connections 2023-08-25T09:50:29.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.904+0000 7f2db2fb0740 1 -- 172.21.15.71:0/1054298293 wait complete. 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.905+0000 7f2db2fb0740 1 -- start start 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.905+0000 7f2db2fb0740 1 Processor -- start 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2094590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a2077d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2079e00 0x5630a208dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a1f57d20 con 0x5630a209ba50 2023-08-25T09:50:29.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a1f579c0 con 0x5630a2079e00 2023-08-25T09:50:29.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630a1d1df20 con 0x5630a2051bd0 2023-08-25T09:50:29.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2db2fb0740 10 client.0 _ll_register_callbacks cb 0x5630a1f6f630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2dad8f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a2077d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2dad8f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a2077d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51310/0 (socket says 172.21.15.71:51310) 2023-08-25T09:50:29.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.906+0000 7f2dad8f8640 1 -- 172.21.15.71:0/2228784614 learned_addr learned my addr 172.21.15.71:0/2228784614 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2079e00 0x5630a208dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2094590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dad8f8640 1 -- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 msgr2=0x5630a2094590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2094590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dad8f8640 1 -- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2079e00 msgr2=0x5630a208dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2079e00 0x5630a208dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dad8f8640 1 -- 172.21.15.71:0/2228784614 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9800c7a0 con 0x5630a209ba50 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2079e00 0x5630a208dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a2077d10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d98006f60 tx=0x7f2d98013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca60a78d75b1377b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2d897fa640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2d897fa640 1 -- 172.21.15.71:0/2228784614 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9801ee90 con 0x5630a209ba50 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630a1f8c540 con 0x5630a209ba50 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.907+0000 7f2d897fa640 1 -- 172.21.15.71:0/2228784614 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2d9802a450 con 0x5630a209ba50 2023-08-25T09:50:29.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2d897fa640 1 -- 172.21.15.71:0/2228784614 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d9802a610 con 0x5630a209ba50 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 10 client.4977 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5630a1faff30 con 0x5630a209ba50 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2d897fa640 1 -- 172.21.15.71:0/2228784614 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5630a1f8c540 con 0x5630a209ba50 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2d897fa640 1 -- 172.21.15.71:0/2228784614 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f2d9802a770 con 0x5630a209ba50 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 10 client.4977 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 10 client.4977 resolve_mds: resolved name 'f' to daemon mds.f 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 20 client.4977 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20a6170 0x5630a20b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 4 client.4977 mds_command: new command op to 4306 tid=0[{"prefix": "get_command_descriptions"}] 2023-08-25T09:50:29.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.908+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 --> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630a1fb1150 con 0x5630a20a6170 2023-08-25T09:50:29.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.909+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20a6170 0x5630a20b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.910+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20a6170 0x5630a20b1040 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c6a0d7b3b8fe010 server_cookie=90e33722a6e6f7a2 in_seq=0 out_seq=0 2023-08-25T09:50:29.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.910+0000 7f2d897fa640 10 client.4977 ms_handle_connect on v2:172.21.15.164:6836/881271092 2023-08-25T09:50:29.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.911+0000 7f2d897fa640 1 -- 172.21.15.71:0/2228784614 <== mds.0 v2:172.21.15.164:6836/881271092 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5630a1fb1150 con 0x5630a20a6170 2023-08-25T09:50:29.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.911+0000 7f2d897fa640 10 client.4977 handle_command_reply: tid=0 2023-08-25T09:50:29.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 client.4977 shutdown 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 msgr2=0x5630a2077d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a2077d10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2d98006f60 tx=0x7f2d98013000 comp rx=0 tx=0).stop 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 shutdown_connections 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20a6170 0x5630a20b1040 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2079e00 0x5630a208dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a209ba50 0x5630a2077d10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/2228784614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2094590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 >> 172.21.15.71:0/2228784614 conn(0x5630a2050630 msgr2=0x5630a204f480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2d897fa640 0 client.4977 ms_handle_reset on v2:172.21.15.164:6836/881271092 2023-08-25T09:50:29.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 shutdown_connections 2023-08-25T09:50:29.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 1 -- 172.21.15.71:0/2228784614 wait complete. 2023-08-25T09:50:29.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.912+0000 7f2db2fb0740 20 client.4977 trim_cache size 0 max 16384 2023-08-25T09:50:29.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.929+0000 7f2db2fb0740 1 Processor -- start 2023-08-25T09:50:29.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.929+0000 7f2db2fb0740 1 -- start start 2023-08-25T09:50:29.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2051bd0 0x5630a204b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2079e00 0x5630a204b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 0x5630a204ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a20482e0 con 0x5630a208fcd0 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a209b210 con 0x5630a2051bd0 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630a1f8c540 con 0x5630a2079e00 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 0x5630a204ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 0x5630a204ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51318/0 (socket says 172.21.15.71:51318) 2023-08-25T09:50:29.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 -- 172.21.15.71:0/4271830796 learned_addr learned my addr 172.21.15.71:0/4271830796 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2051bd0 0x5630a204b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 -- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2079e00 msgr2=0x5630a204b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2079e00 0x5630a204b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 -- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2051bd0 msgr2=0x5630a204b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2051bd0 0x5630a204b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae8fa640 1 -- 172.21.15.71:0/4271830796 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630a1f56a10 con 0x5630a208fcd0 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.930+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2051bd0 0x5630a204b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.931+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 0x5630a204ce20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2da0001620 tx=0x7f2da0001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef993e74ad457314 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.931+0000 7f2d9ce1b640 1 -- 172.21.15.71:0/4271830796 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0018070 con 0x5630a208fcd0 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.931+0000 7f2d9ce1b640 1 -- 172.21.15.71:0/4271830796 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da0001d30 con 0x5630a208fcd0 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.931+0000 7f2d9ce1b640 1 -- 172.21.15.71:0/4271830796 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0007500 con 0x5630a208fcd0 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.931+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4271830796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 msgr2=0x5630a204ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:29.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.931+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 0x5630a204ce20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2da0001620 tx=0x7f2da0001ae0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4271830796 shutdown_connections 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a208fcd0 0x5630a204ce20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2079e00 0x5630a204b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4271830796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a2051bd0 0x5630a204b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4271830796 >> 172.21.15.71:0/4271830796 conn(0x5630a2050630 msgr2=0x5630a2047730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4271830796 shutdown_connections 2023-08-25T09:50:29.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4271830796 wait complete. 2023-08-25T09:50:29.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.932+0000 7f2db2fb0740 1 -- start start 2023-08-25T09:50:29.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 Processor -- start 2023-08-25T09:50:29.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2052500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 0x5630a208d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a208fcd0 0x5630a20fe850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a20a4950 con 0x5630a2079e00 2023-08-25T09:50:29.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a20a4d70 con 0x5630a208fcd0 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.933+0000 7f2db2fb0740 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630a20fc0b0 con 0x5630a2051bd0 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2db2fb0740 10 client.0 _ll_register_callbacks cb 0x7f2d9800a4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 0x5630a208d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 0x5630a208d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51326/0 (socket says 172.21.15.71:51326) 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 -- 172.21.15.71:0/4095211361 learned_addr learned my addr 172.21.15.71:0/4095211361 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 -- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 msgr2=0x5630a2052500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2052500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a208fcd0 0x5630a20fe850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 -- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a208fcd0 msgr2=0x5630a20fe850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a208fcd0 0x5630a20fe850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 -- 172.21.15.71:0/4095211361 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da0007dd0 con 0x5630a2079e00 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dae8fa640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a208fcd0 0x5630a20fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2dad8f8640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 0x5630a208d2a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2d9802c040 tx=0x7f2d9800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4a68c91f255fa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.934+0000 7f2d897fa640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2d897fa640 1 -- 172.21.15.71:0/4095211361 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d980137f0 con 0x5630a2079e00 2023-08-25T09:50:29.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2d897fa640 1 -- 172.21.15.71:0/4095211361 <== mon.0 v2:172.21.15.71:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2d98032e40 con 0x5630a2079e00 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d9801e400 con 0x5630a2079e00 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2d897fa640 1 -- 172.21.15.71:0/4095211361 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d9800a630 con 0x5630a2079e00 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 10 client.4983 fetch_fsmap learned FSMap version 17 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5630a1f7bde0 con 0x5630a2079e00 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2d897fa640 1 -- 172.21.15.71:0/4095211361 <== mon.0 v2:172.21.15.71:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9801e400 con 0x5630a2079e00 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2d897fa640 1 -- 172.21.15.71:0/4095211361 <== mon.0 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) v1 ==== 4500+0+0 (secure 0 0 0) 0x7f2d9802a660 con 0x5630a2079e00 2023-08-25T09:50:29.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 10 client.4983 fetch_fsmap finished waiting for FSMap version 17 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 10 client.4983 resolve_mds: resolved name 'f' to daemon mds.f 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 20 client.4983 populate_metadata read hostname 'smithi071' 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20ff2c0 0x5630a20f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 4 client.4983 mds_command: new command op to 4306 tid=0[{"prefix": "status"}] 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.935+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 --> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5630a1c675e0 con 0x5630a20ff2c0 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.936+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20ff2c0 0x5630a20f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.937+0000 7f2dae0f9640 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20ff2c0 0x5630a20f4c10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0a29887a33a6fa server_cookie=d34b990c432f0c57 in_seq=0 out_seq=0 2023-08-25T09:50:29.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:29.937+0000 7f2d897fa640 10 client.4983 ms_handle_connect on v2:172.21.15.164:6836/881271092 2023-08-25T09:50:30.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2d897fa640 1 -- 172.21.15.71:0/4095211361 <== mds.0 v2:172.21.15.164:6836/881271092 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5630a1c675e0 con 0x5630a20ff2c0 2023-08-25T09:50:30.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2d897fa640 10 client.4983 handle_command_reply: tid=0 2023-08-25T09:50:30.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 client.4983 shutdown 2023-08-25T09:50:30.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 msgr2=0x5630a208d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 0x5630a208d2a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2d9802c040 tx=0x7f2d9800f000 comp rx=0 tx=0).stop 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 shutdown_connections 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:6836/881271092,v1:172.21.15.164:6837/881271092] conn(0x5630a20ff2c0 0x5630a20f4c10 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5630a208fcd0 0x5630a20fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5630a2079e00 0x5630a208d2a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 --2- 172.21.15.71:0/4095211361 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x5630a2051bd0 0x5630a2052500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2d897fa640 0 client.4983 ms_handle_reset on v2:172.21.15.164:6836/881271092 2023-08-25T09:50:30.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 >> 172.21.15.71:0/4095211361 conn(0x5630a2050630 msgr2=0x5630a204b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:30.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 shutdown_connections 2023-08-25T09:50:30.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.433+0000 7f2db2fb0740 1 -- 172.21.15.71:0/4095211361 wait complete. 2023-08-25T09:50:30.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.434+0000 7f2db2fb0740 20 client.4983 trim_cache size 0 max 16384 2023-08-25T09:50:30.439 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:50:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "cluster_fsid": "d3b06f22-7372-48e3-a060-a2382f01af3f", 2023-08-25T09:50:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "whoami": 0, 2023-08-25T09:50:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4306, 2023-08-25T09:50:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "want_state": "up:active", 2023-08-25T09:50:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "state": "up:active", 2023-08-25T09:50:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs", 2023-08-25T09:50:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "rank_uptime": 55.82133674, 2023-08-25T09:50:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "mdsmap_epoch": 17, 2023-08-25T09:50:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch": 31, 2023-08-25T09:50:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_epoch_barrier": 31, 2023-08-25T09:50:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "uptime": 62.667538880999999 2023-08-25T09:50:30.441 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:50:30.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d8c06a050 msgr2=0x7f2d8c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:30.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d8c06a050 0x7f2d8c06c510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2da80c89f0 tx=0x7f2d98008040 comp rx=0 tx=0).stop 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 msgr2=0x7f2da81f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 0x7f2da81f1a20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009de0 tx=0x7f2da4004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 shutdown_connections 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da8150ce0 0x7f2da806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d8c06a050 0x7f2d8c06c510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da80c76b0 0x7f2da806f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 --2- 172.21.15.71:0/17072058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2da80c58c0 0x7f2da81f1a20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 >> 172.21.15.71:0/17072058 conn(0x7f2da806bff0 msgr2=0x7f2da814fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:30.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 shutdown_connections 2023-08-25T09:50:30.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.435+0000 7f2db035d640 1 -- 172.21.15.71:0/17072058 wait complete. 2023-08-25T09:50:30.466 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-08-25T09:50:30.477 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-08-25T09:50:30.478 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-08-25T09:50:30.478 INFO:tasks.ceph_fuse:client.0 config is {} 2023-08-25T09:50:30.479 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-08-25T09:50:30.479 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:30.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:30.480 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list 2023-08-25T09:50:30.542 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-08-25T09:50:30.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:30.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-08-25T09:50:30.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:30.647 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link delete ceph-brx 2023-08-25T09:50:30.768 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-08-25T09:50:30.768 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-08-25T09:50:30.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.960+0000 7f1d6cbc9640 1 Processor -- start 2023-08-25T09:50:30.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.960+0000 7f1d6cbc9640 1 -- start start 2023-08-25T09:50:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.961+0000 7f1d6cbc9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1d681594d0 0x7f1d680c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.961+0000 7f1d6cbc9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1d680c3940 0x7f1d681598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.961+0000 7f1d6cbc9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6806a960 con 0x7f1d6806cc70 2023-08-25T09:50:30.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.961+0000 7f1d6cbc9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6806ac20 con 0x7f1d680c3940 2023-08-25T09:50:30.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.961+0000 7f1d6cbc9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6806af00 con 0x7f1d681594d0 2023-08-25T09:50:30.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1d680c3940 0x7f1d681598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:30.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1d680c3940 0x7f1d681598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54808/0 (socket says 172.21.15.71:54808) 2023-08-25T09:50:30.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 -- 172.21.15.71:0/69054460 learned_addr learned my addr 172.21.15.71:0/69054460 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d6659c640 1 --2- 172.21.15.71:0/69054460 >> v2:172.21.15.164:3300/0 conn(0x7f1d681594d0 0x7f1d680c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 -- 172.21.15.71:0/69054460 >> v2:172.21.15.164:3300/0 conn(0x7f1d681594d0 msgr2=0x7f1d680c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 --2- 172.21.15.71:0/69054460 >> v2:172.21.15.164:3300/0 conn(0x7f1d681594d0 0x7f1d680c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 -- 172.21.15.71:0/69054460 >> v1:172.21.15.164:6789/0 conn(0x7f1d6806cc70 legacy=0x7f1d6814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.962+0000 7f1d66d9d640 1 -- 172.21.15.71:0/69054460 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d680c2e60 con 0x7f1d680c3940 2023-08-25T09:50:30.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.963+0000 7f1d66d9d640 1 --2- 172.21.15.71:0/69054460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d680c3940 0x7f1d681598b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1d68071d10 tx=0x7f1d6002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f8a5df1ec261ef server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:30.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d65d9b640 1 -- 172.21.15.71:0/69054460 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6002ca80 con 0x7f1d680c3940 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d65d9b640 1 -- 172.21.15.71:0/69054460 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6002cbe0 con 0x7f1d680c3940 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d65d9b640 1 -- 172.21.15.71:0/69054460 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600343f0 con 0x7f1d680c3940 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/69054460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d680c3940 msgr2=0x7f1d681598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/69054460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d680c3940 0x7f1d681598b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1d68071d10 tx=0x7f1d6002bee0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/69054460 shutdown_connections 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/69054460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d680c3940 0x7f1d681598b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/69054460 >> v2:172.21.15.164:3300/0 conn(0x7f1d681594d0 0x7f1d680c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/69054460 >> 172.21.15.71:0/69054460 conn(0x7f1d680654f0 msgr2=0x7f1d68065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.964+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/69054460 shutdown_connections 2023-08-25T09:50:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.965+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/69054460 wait complete. 2023-08-25T09:50:30.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.966+0000 7f1d6cbc9640 1 Processor -- start 2023-08-25T09:50:30.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.966+0000 7f1d6cbc9640 1 -- start start 2023-08-25T09:50:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.967+0000 7f1d6cbc9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 0x7f1d680c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.967+0000 7f1d6cbc9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d680c3940 0x7f1d6813e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.967+0000 7f1d6cbc9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d681594d0 0x7f1d68069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.967+0000 7f1d6cbc9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806f8d0 con 0x7f1d6806cc70 2023-08-25T09:50:30.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.967+0000 7f1d6cbc9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680c8630 con 0x7f1d681594d0 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.967+0000 7f1d6cbc9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681ec610 con 0x7f1d680c3940 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 0x7f1d680c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 0x7f1d680c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54814/0 (socket says 172.21.15.71:54814) 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 -- 172.21.15.71:0/1592785861 learned_addr learned my addr 172.21.15.71:0/1592785861 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d5ffff640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d680c3940 0x7f1d6813e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d66d9d640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d681594d0 0x7f1d68069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:30.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 -- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d680c3940 msgr2=0x7f1d6813e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d680c3940 0x7f1d6813e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 -- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d681594d0 msgr2=0x7f1d68069ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d681594d0 0x7f1d68069ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d6659c640 1 -- 172.21.15.71:0/1592785861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d60009b30 con 0x7f1d6806cc70 2023-08-25T09:50:30.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.968+0000 7f1d66d9d640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d681594d0 0x7f1d68069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:30.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.969+0000 7f1d6659c640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 0x7f1d680c80f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800d830 tx=0x7f1d5800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:30.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.969+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5800b6c0 con 0x7f1d6806cc70 2023-08-25T09:50:30.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.969+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5800b820 con 0x7f1d6806cc70 2023-08-25T09:50:30.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.969+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d681ecb40 con 0x7f1d6806cc70 2023-08-25T09:50:30.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.969+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5800bbb0 con 0x7f1d6806cc70 2023-08-25T09:50:30.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.969+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6806e650 con 0x7f1d6806cc70 2023-08-25T09:50:30.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.970+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d68066510 con 0x7f1d6806cc70 2023-08-25T09:50:30.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.971+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1d5801e460 con 0x7f1d6806cc70 2023-08-25T09:50:30.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.972+0000 7f1d5effd640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d4806a050 0x7f1d4806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:30.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.972+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1d58088fe0 con 0x7f1d6806cc70 2023-08-25T09:50:30.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.973+0000 7f1d5ffff640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d4806a050 0x7f1d4806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:30.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.973+0000 7f1d5ffff640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d4806a050 0x7f1d4806c510 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1d50006fd0 tx=0x7f1d50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:30.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:30.977+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1d580543f0 con 0x7f1d6806cc70 2023-08-25T09:50:31.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.202+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1d68069160 con 0x7f1d6806cc70 2023-08-25T09:50:31.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.203+0000 7f1d5effd640 1 -- 172.21.15.71:0/1592785861 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1d58057a10 con 0x7f1d6806cc70 2023-08-25T09:50:31.206 INFO:teuthology.orchestra.run.smithi071.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-08-25T09:50:31.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d4806a050 msgr2=0x7f1d4806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:31.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d4806a050 0x7f1d4806c510 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1d50006fd0 tx=0x7f1d50008040 comp rx=0 tx=0).stop 2023-08-25T09:50:31.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 msgr2=0x7f1d680c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:31.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 0x7f1d680c80f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1d5800d830 tx=0x7f1d5800dcf0 comp rx=0 tx=0).stop 2023-08-25T09:50:31.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 shutdown_connections 2023-08-25T09:50:31.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d681594d0 0x7f1d68069ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:31.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d4806a050 0x7f1d4806c510 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:31.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d680c3940 0x7f1d6813e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:31.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 --2- 172.21.15.71:0/1592785861 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d6806cc70 0x7f1d680c80f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:31.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 >> 172.21.15.71:0/1592785861 conn(0x7f1d680654f0 msgr2=0x7f1d681430b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:31.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 shutdown_connections 2023-08-25T09:50:31.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:31.206+0000 7f1d6cbc9640 1 -- 172.21.15.71:0/1592785861 wait complete. 2023-08-25T09:50:31.227 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-08-25T09:50:31.227 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-08-25T09:50:31.228 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi071.front.sepia.ceph.com 2023-08-25T09:50:31.228 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-08-25T09:50:31.228 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:31.228 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-08-25T09:50:31.229 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-08-25T09:50:31.229 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-08-25T09:50:31.229 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-08-25T09:50:31.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.230 DEBUG:teuthology.orchestra.run.smithi071:> ip addr 2023-08-25T09:50:31.281 INFO:teuthology.orchestra.run.smithi071.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-08-25T09:50:31.281 INFO:teuthology.orchestra.run.smithi071.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-08-25T09:50:31.282 INFO:teuthology.orchestra.run.smithi071.stdout: inet 127.0.0.1/8 scope host lo 2023-08-25T09:50:31.282 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2023-08-25T09:50:31.282 INFO:teuthology.orchestra.run.smithi071.stdout: inet6 ::1/128 scope host 2023-08-25T09:50:31.282 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2023-08-25T09:50:31.282 INFO:teuthology.orchestra.run.smithi071.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T09:50:31.282 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:d8:aa:4a brd ff:ff:ff:ff:ff:ff 2023-08-25T09:50:31.283 INFO:teuthology.orchestra.run.smithi071.stdout: altname enp7s0f0 2023-08-25T09:50:31.283 INFO:teuthology.orchestra.run.smithi071.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T09:50:31.283 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:d8:aa:4b brd ff:ff:ff:ff:ff:ff 2023-08-25T09:50:31.284 INFO:teuthology.orchestra.run.smithi071.stdout: altname enp7s0f1 2023-08-25T09:50:31.284 INFO:teuthology.orchestra.run.smithi071.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-08-25T09:50:31.284 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:8f:d7:64 brd ff:ff:ff:ff:ff:ff 2023-08-25T09:50:31.284 INFO:teuthology.orchestra.run.smithi071.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-08-25T09:50:31.285 INFO:teuthology.orchestra.run.smithi071.stdout: link/ether 0c:c4:7a:8f:d7:65 brd ff:ff:ff:ff:ff:ff 2023-08-25T09:50:31.285 INFO:teuthology.orchestra.run.smithi071.stdout: inet 172.21.15.71/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-08-25T09:50:31.285 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft 42286sec preferred_lft 42286sec 2023-08-25T09:50:31.285 INFO:teuthology.orchestra.run.smithi071.stdout: inet6 fe80::ec4:7aff:fe8f:d765/64 scope link noprefixroute 2023-08-25T09:50:31.286 INFO:teuthology.orchestra.run.smithi071.stdout: valid_lft forever preferred_lft forever 2023-08-25T09:50:31.286 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-08-25T09:50:31.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.287 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:50:31.288 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:50:31.288 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link add name ceph-brx type bridge 2023-08-25T09:50:31.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip addr flush dev ceph-brx 2023-08-25T09:50:31.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set ceph-brx up 2023-08-25T09:50:31.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-08-25T09:50:31.290 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:50:31.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.522 DEBUG:teuthology.orchestra.run.smithi071:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-08-25T09:50:31.600 INFO:teuthology.orchestra.run.smithi071.stdout:1 2023-08-25T09:50:31.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.603 DEBUG:teuthology.orchestra.run.smithi071:> route 2023-08-25T09:50:31.650 INFO:teuthology.orchestra.run.smithi071.stdout:Kernel IP routing table 2023-08-25T09:50:31.651 INFO:teuthology.orchestra.run.smithi071.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-08-25T09:50:31.651 INFO:teuthology.orchestra.run.smithi071.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-08-25T09:50:31.651 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-08-25T09:50:31.651 INFO:teuthology.orchestra.run.smithi071.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-08-25T09:50:31.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.652 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:50:31.652 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:50:31.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-08-25T09:50:31.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-08-25T09:50:31.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-08-25T09:50:31.653 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:50:31.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.885 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list 2023-08-25T09:50:31.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.941 DEBUG:teuthology.orchestra.run.smithi071:> ip netns list-id 2023-08-25T09:50:31.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:31.995 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:50:31.995 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:50:31.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-08-25T09:50:31.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-08-25T09:50:31.996 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:50:32.141 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-08-25T09:50:32.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:32.142 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:50:32.142 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:50:32.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-08-25T09:50:32.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-08-25T09:50:32.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-08-25T09:50:32.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-08-25T09:50:32.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-08-25T09:50:32.143 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:50:32.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:32.404 DEBUG:teuthology.orchestra.run.smithi071:> (cd / && exec bash -c ' 2023-08-25T09:50:32.405 DEBUG:teuthology.orchestra.run.smithi071:> set -e 2023-08-25T09:50:32.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set brx.0 up 2023-08-25T09:50:32.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo ip link set dev brx.0 master ceph-brx 2023-08-25T09:50:32.405 DEBUG:teuthology.orchestra.run.smithi071:> ') 2023-08-25T09:50:32.574 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-08-25T09:50:32.574 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-08-25T09:50:32.575 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:32.624 INFO:teuthology.orchestra.run.smithi071.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-08-25T09:50:32.625 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-08-25T09:50:32.625 DEBUG:teuthology.orchestra.run.smithi071:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:32.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe fuse 2023-08-25T09:50:32.778 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-08-25T09:50:32.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-08-25T09:50:32.872 INFO:teuthology.orchestra.run.smithi071.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-08-25T09:50:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-08-25T09:50:32.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:32.877 DEBUG:teuthology.orchestra.run.smithi071:> ls /sys/fs/fuse/connections 2023-08-25T09:50:32.931 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-08-25T09:50:32.932 DEBUG:teuthology.orchestra.run.smithi071:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-08-25T09:50:32.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe fuse 2023-08-25T09:50:33.015 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-08-25T09:50:33.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-08-25T09:50:33.087 INFO:teuthology.orchestra.run.smithi071.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-08-25T09:50:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-08-25T09:50:33.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:33.091 DEBUG:teuthology.orchestra.run.smithi071:> ls /sys/fs/fuse/connections 2023-08-25T09:50:33.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi071.stderr:2023-08-25T09:50:33.125+0000 7f3969a3e480 -1 init, newargv = 0x55d4f69e23c0 newargc=15 2023-08-25T09:50:33.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi071.stderr:ceph-fuse[98567]: starting ceph client 2023-08-25T09:50:33.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi071.stderr:ceph-fuse[98567]: starting fuse 2023-08-25T09:50:34.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo modprobe fuse 2023-08-25T09:50:34.338 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-08-25T09:50:34.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-08-25T09:50:34.436 INFO:teuthology.orchestra.run.smithi071.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-08-25T09:50:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-08-25T09:50:34.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:34.441 DEBUG:teuthology.orchestra.run.smithi071:> ls /sys/fs/fuse/connections 2023-08-25T09:50:34.494 INFO:teuthology.orchestra.run.smithi071.stdout:39 2023-08-25T09:50:34.495 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-08-25T09:50:34.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo stdin-killer -- python3 -c ' 2023-08-25T09:50:34.495 DEBUG:teuthology.orchestra.run.smithi071:> import glob 2023-08-25T09:50:34.496 DEBUG:teuthology.orchestra.run.smithi071:> import re 2023-08-25T09:50:34.496 DEBUG:teuthology.orchestra.run.smithi071:> import os 2023-08-25T09:50:34.496 DEBUG:teuthology.orchestra.run.smithi071:> import subprocess 2023-08-25T09:50:34.497 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:50:34.497 DEBUG:teuthology.orchestra.run.smithi071:> def _find_admin_socket(client_name): 2023-08-25T09:50:34.497 DEBUG:teuthology.orchestra.run.smithi071:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-08-25T09:50:34.498 DEBUG:teuthology.orchestra.run.smithi071:> files = glob.glob(asok_path) 2023-08-25T09:50:34.498 DEBUG:teuthology.orchestra.run.smithi071:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-08-25T09:50:34.498 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:50:34.498 DEBUG:teuthology.orchestra.run.smithi071:> # Given a non-glob path, it better be there 2023-08-25T09:50:34.498 DEBUG:teuthology.orchestra.run.smithi071:> if "*" not in asok_path: 2023-08-25T09:50:34.499 DEBUG:teuthology.orchestra.run.smithi071:> assert(len(files) == 1) 2023-08-25T09:50:34.499 DEBUG:teuthology.orchestra.run.smithi071:> return files[0] 2023-08-25T09:50:34.499 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:50:34.499 DEBUG:teuthology.orchestra.run.smithi071:> for f in files: 2023-08-25T09:50:34.500 DEBUG:teuthology.orchestra.run.smithi071:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-08-25T09:50:34.500 DEBUG:teuthology.orchestra.run.smithi071:> if os.path.exists("/proc/{0}".format(pid)): 2023-08-25T09:50:34.500 DEBUG:teuthology.orchestra.run.smithi071:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-08-25T09:50:34.500 DEBUG:teuthology.orchestra.run.smithi071:> contents = proc_f.read() 2023-08-25T09:50:34.500 DEBUG:teuthology.orchestra.run.smithi071:> if mountpoint in contents: 2023-08-25T09:50:34.501 DEBUG:teuthology.orchestra.run.smithi071:> return f 2023-08-25T09:50:34.501 DEBUG:teuthology.orchestra.run.smithi071:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-08-25T09:50:34.501 DEBUG:teuthology.orchestra.run.smithi071:> 2023-08-25T09:50:34.501 DEBUG:teuthology.orchestra.run.smithi071:> print(_find_admin_socket("client.0")) 2023-08-25T09:50:34.502 DEBUG:teuthology.orchestra.run.smithi071:> ' 2023-08-25T09:50:34.690 INFO:teuthology.orchestra.run.smithi071.stdout:/var/run/ceph/ceph-client.0.98567.asok 2023-08-25T09:50:34.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-08-25T09:50:34.707 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.98567.asok 2023-08-25T09:50:34.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:34.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.98567.asok status 2023-08-25T09:50:34.948 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2023-08-25T09:50:34.949 INFO:teuthology.orchestra.run.smithi071.stdout: "metadata": { 2023-08-25T09:50:34.949 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_sha1": "ba6b39e4f4f77fb95c747099b53db0467d435f56", 2023-08-25T09:50:34.949 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_version": "ceph version 18.0.0-5703-gba6b39e4 (ba6b39e4f4f77fb95c747099b53db0467d435f56) reef (dev)", 2023-08-25T09:50:34.949 INFO:teuthology.orchestra.run.smithi071.stdout: "entity_id": "0", 2023-08-25T09:50:34.949 INFO:teuthology.orchestra.run.smithi071.stdout: "hostname": "smithi071", 2023-08-25T09:50:34.949 INFO:teuthology.orchestra.run.smithi071.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: "pid": "98567", 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: "root": "/" 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: "dentry_count": 0, 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: "dentry_pinned_count": 0, 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: "id": 4992, 2023-08-25T09:50:34.950 INFO:teuthology.orchestra.run.smithi071.stdout: "inst": { 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: "name": { 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "client", 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: "num": 4992 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": { 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "v1", 2023-08-25T09:50:34.951 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": "192.168.0.1:0", 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: "nonce": 2788445468 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: } 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": { 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: "type": "v1", 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: "addr": "192.168.0.1:0", 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: "nonce": 2788445468 2023-08-25T09:50:34.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "inst_str": "client.4992 192.168.0.1:0/2788445468", 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "addr_str": "192.168.0.1:0/2788445468", 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "inode_count": 1, 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_epoch": 17, 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_epoch": 31, 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_epoch_barrier": 0, 2023-08-25T09:50:34.953 INFO:teuthology.orchestra.run.smithi071.stdout: "blocklisted": false, 2023-08-25T09:50:34.954 INFO:teuthology.orchestra.run.smithi071.stdout: "fs_name": "cephfs" 2023-08-25T09:50:34.954 INFO:teuthology.orchestra.run.smithi071.stdout:} 2023-08-25T09:50:34.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:34.963 DEBUG:teuthology.orchestra.run.smithi071:> stat --file-system '--printf=%T 2023-08-25T09:50:34.963 DEBUG:teuthology.orchestra.run.smithi071:> ' -- /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:35.017 INFO:teuthology.orchestra.run.smithi071.stdout:fuseblk 2023-08-25T09:50:35.018 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:35.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T09:50:35.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:35.113 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-08-25T09:50:35.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:35.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.345+0000 7fab7f70f640 1 Processor -- start 2023-08-25T09:50:35.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.345+0000 7fab7f70f640 1 -- start start 2023-08-25T09:50:35.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7f70f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 0x7fab780c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7f70f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780c1dc0 con 0x7fab781509d0 2023-08-25T09:50:35.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7f70f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab78156e00 con 0x7fab780c3a40 2023-08-25T09:50:35.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7f70f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab78157a80 con 0x7fab780c3660 2023-08-25T09:50:35.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7dcac640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fab781509d0 0x7fab780c5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47340/0 (socket says 172.21.15.71:47340) 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7dcac640 1 -- 172.21.15.71:0/747761206 learned_addr learned my addr 172.21.15.71:0/747761206 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 0x7fab780c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.347+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708963169 0 0) 0x7fab780c1dc0 con 0x7fab781509d0 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab64003520 con 0x7fab781509d0 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4175114344 0 0) 0x7fab64003520 con 0x7fab781509d0 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 msgr2=0x7fab780c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 0x7fab780c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 >> v1:172.21.15.164:6789/0 conn(0x7fab780c3a40 legacy=0x7fab7806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab781570e0 con 0x7fab781509d0 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fab700019f0 con 0x7fab781509d0 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 >> v1:172.21.15.71:6789/0 conn(0x7fab781509d0 legacy=0x7fab780c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab64003a10 0x7fab64003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 0x7fab640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64004d00 0x7fab64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab78157a80 con 0x7fab64003a10 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6400bb20 con 0x7fab64004d00 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6400bce0 con 0x7fab780c3660 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.348+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab64003a10 0x7fab64003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7dcac640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64004d00 0x7fab64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7ccaa640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 0x7fab640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7d4ab640 1 -- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 msgr2=0x7fab640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab780c3660 0x7fab640047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7d4ab640 1 -- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64004d00 msgr2=0x7fab64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:35.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64004d00 0x7fab64005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.349+0000 7fab7d4ab640 1 -- 172.21.15.71:0/747761206 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab64003520 con 0x7fab64003a10 2023-08-25T09:50:35.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.350+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab64003a10 0x7fab64003e30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fab74009a90 tx=0x7fab640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88309a4ebd051d6e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:35.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.350+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7402c5c0 con 0x7fab64003a10 2023-08-25T09:50:35.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.350+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7402c720 con 0x7fab64003a10 2023-08-25T09:50:35.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.350+0000 7fab6ffff640 1 -- 172.21.15.71:0/747761206 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7402cab0 con 0x7fab64003a10 2023-08-25T09:50:35.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7f70f640 1 -- 172.21.15.71:0/747761206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab64003a10 msgr2=0x7fab64003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7f70f640 1 --2- 172.21.15.71:0/747761206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab64003a10 0x7fab64003e30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fab74009a90 tx=0x7fab640057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7dcac640 1 -- 172.21.15.71:0/747761206 reap_dead start 2023-08-25T09:50:35.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7f70f640 1 -- 172.21.15.71:0/747761206 shutdown_connections 2023-08-25T09:50:35.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7f70f640 1 -- 172.21.15.71:0/747761206 >> 172.21.15.71:0/747761206 conn(0x7fab780654f0 msgr2=0x7fab78065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:35.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7f70f640 1 -- 172.21.15.71:0/747761206 shutdown_connections 2023-08-25T09:50:35.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.351+0000 7fab7f70f640 1 -- 172.21.15.71:0/747761206 wait complete. 2023-08-25T09:50:35.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.353+0000 7fab7f70f640 1 Processor -- start 2023-08-25T09:50:35.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.353+0000 7fab7f70f640 1 -- start start 2023-08-25T09:50:35.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7f70f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64003a10 0x7fab7806a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7f70f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab78153ce0 0x7fab7815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7f70f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 0x7fab78153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7f70f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab78154540 con 0x7fab7815a620 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7f70f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab781f2750 con 0x7fab64003a10 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7f70f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7814dab0 con 0x7fab78153ce0 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 0x7fab78153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 0x7fab78153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54828/0 (socket says 172.21.15.71:54828) 2023-08-25T09:50:35.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 -- 172.21.15.71:0/872097638 learned_addr learned my addr 172.21.15.71:0/872097638 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:35.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64003a10 0x7fab7806a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7ccaa640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab78153ce0 0x7fab7815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 -- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab78153ce0 msgr2=0x7fab7815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab78153ce0 0x7fab7815a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 -- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64003a10 msgr2=0x7fab7806a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64003a10 0x7fab7806a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.354+0000 7fab7dcac640 1 -- 172.21.15.71:0/872097638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab781f22c0 con 0x7fab7815a620 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.355+0000 7fab7dcac640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 0x7fab78153020 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fab78071cc0 tx=0x7fab7000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.355+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7000ebe0 con 0x7fab7815a620 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.355+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7000ed60 con 0x7fab7815a620 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.355+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70014670 con 0x7fab7815a620 2023-08-25T09:50:35.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.355+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab78154260 con 0x7fab7815a620 2023-08-25T09:50:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.355+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab781f1c70 con 0x7fab7815a620 2023-08-25T09:50:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.356+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab781f2450 con 0x7fab7815a620 2023-08-25T09:50:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.357+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fab700147d0 con 0x7fab7815a620 2023-08-25T09:50:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.358+0000 7fab6dffb640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.358+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fab700885d0 con 0x7fab7815a620 2023-08-25T09:50:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.359+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.359+0000 7fab7d4ab640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab74036000 tx=0x7fab74030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:35.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.363+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fab70053990 con 0x7fab7815a620 2023-08-25T09:50:35.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.553+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab78069160 con 0x7fab7815a620 2023-08-25T09:50:35.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.555+0000 7fab6dffb640 1 -- 172.21.15.71:0/872097638 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7fab70056fb0 con 0x7fab7815a620 2023-08-25T09:50:35.557 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:35.558 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:35.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.558+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 msgr2=0x7fab5c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.558+0000 7fab7f70f640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab74036000 tx=0x7fab74030000 comp rx=0 tx=0).stop 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.558+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 msgr2=0x7fab78153020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.558+0000 7fab7f70f640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 0x7fab78153020 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fab78071cc0 tx=0x7fab7000e530 comp rx=0 tx=0).stop 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.558+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 shutdown_connections 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fab5c06a080 0x7fab5c06c540 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab7815a620 0x7fab78153020 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab78153ce0 0x7fab7815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 --2- 172.21.15.71:0/872097638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab64003a10 0x7fab7806a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 >> 172.21.15.71:0/872097638 conn(0x7fab780654f0 msgr2=0x7fab780c9420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 shutdown_connections 2023-08-25T09:50:35.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.559+0000 7fab7f70f640 1 -- 172.21.15.71:0/872097638 wait complete. 2023-08-25T09:50:35.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:50:35.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:50:35.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.799+0000 7f1239353640 1 Processor -- start 2023-08-25T09:50:35.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.799+0000 7f1239353640 1 -- start start 2023-08-25T09:50:35.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1239353640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f12340c76d0 0x7f1234152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1239353640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123406aed0 0x7f12341532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1239353640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12340c1060 con 0x7f12340c72f0 2023-08-25T09:50:35.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1239353640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12340adf70 con 0x7f12340c76d0 2023-08-25T09:50:35.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1239353640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12340af730 con 0x7f123406aed0 2023-08-25T09:50:35.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f122bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f12340c76d0 0x7f1234152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1233fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f12340c72f0 0x7f12340acdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47354/0 (socket says 172.21.15.71:47354) 2023-08-25T09:50:35.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.801+0000 7f1233fff640 1 -- 172.21.15.71:0/490539771 learned_addr learned my addr 172.21.15.71:0/490539771 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f122bfff640 1 -- 172.21.15.71:0/490539771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123406aed0 msgr2=0x7f12341532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f1238b52640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123406aed0 0x7f12341532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f12337fe640 1 -- 172.21.15.71:0/490539771 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940659841 0 0) 0x7f12340c1060 con 0x7f12340c72f0 2023-08-25T09:50:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f122bfff640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123406aed0 0x7f12341532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f122bfff640 1 -- 172.21.15.71:0/490539771 >> v1:172.21.15.71:6789/0 conn(0x7f12340c72f0 legacy=0x7f12340acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f122bfff640 1 -- 172.21.15.71:0/490539771 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123406b4a0 con 0x7f12340c76d0 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.802+0000 7f1238b52640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123406aed0 0x7f12341532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.803+0000 7f122bfff640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12340c76d0 0x7f1234152da0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f121c0097a0 tx=0x7f121c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c57c7a31c8bd25 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.803+0000 7f12337fe640 1 -- 172.21.15.71:0/490539771 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c030800 con 0x7f12340c76d0 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.803+0000 7f12337fe640 1 -- 172.21.15.71:0/490539771 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c030960 con 0x7f12340c76d0 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.803+0000 7f12337fe640 1 -- 172.21.15.71:0/490539771 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c030cf0 con 0x7f12340c76d0 2023-08-25T09:50:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 -- 172.21.15.71:0/490539771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12340c76d0 msgr2=0x7f1234152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12340c76d0 0x7f1234152da0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f121c0097a0 tx=0x7f121c02b940 comp rx=0 tx=0).stop 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 -- 172.21.15.71:0/490539771 shutdown_connections 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f123406aed0 0x7f12341532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 --2- 172.21.15.71:0/490539771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12340c76d0 0x7f1234152da0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 -- 172.21.15.71:0/490539771 >> 172.21.15.71:0/490539771 conn(0x7f123406c280 msgr2=0x7f123406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 -- 172.21.15.71:0/490539771 shutdown_connections 2023-08-25T09:50:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.804+0000 7f1239353640 1 -- 172.21.15.71:0/490539771 wait complete. 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.806+0000 7f1239353640 1 Processor -- start 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.806+0000 7f1239353640 1 -- start start 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1239353640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 0x7f1234154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1239353640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12340c72f0 0x7f12341e25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1239353640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12340c76d0 0x7f12341e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1239353640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12340c5600 con 0x7f123406aed0 2023-08-25T09:50:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1239353640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123406ac80 con 0x7f12340c72f0 2023-08-25T09:50:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1239353640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12341ee7c0 con 0x7f12340c76d0 2023-08-25T09:50:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 0x7f1234154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 0x7f1234154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54850/0 (socket says 172.21.15.71:54850) 2023-08-25T09:50:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 -- 172.21.15.71:0/2587121167 learned_addr learned my addr 172.21.15.71:0/2587121167 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 -- 172.21.15.71:0/2587121167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12340c76d0 msgr2=0x7f12341e2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12340c76d0 0x7f12341e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 -- 172.21.15.71:0/2587121167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12340c72f0 msgr2=0x7f12341e25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.807+0000 7f1233fff640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12340c72f0 0x7f12341e25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f1233fff640 1 -- 172.21.15.71:0/2587121167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c009400 con 0x7f123406aed0 2023-08-25T09:50:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f1233fff640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 0x7f1234154bf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f122400c4d0 tx=0x7f122400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12240074e0 con 0x7f123406aed0 2023-08-25T09:50:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1224007660 con 0x7f123406aed0 2023-08-25T09:50:35.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1224004a20 con 0x7f123406aed0 2023-08-25T09:50:35.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1234155a50 con 0x7f123406aed0 2023-08-25T09:50:35.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.808+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12340c5830 con 0x7f123406aed0 2023-08-25T09:50:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.809+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123404e420 con 0x7f123406aed0 2023-08-25T09:50:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.810+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1224004b80 con 0x7f123406aed0 2023-08-25T09:50:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.811+0000 7f12317fa640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f121406a050 0x7f121406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.811+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f1224087b60 con 0x7f123406aed0 2023-08-25T09:50:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.812+0000 7f122bfff640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f121406a050 0x7f121406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.812+0000 7f122bfff640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f121406a050 0x7f121406c510 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f121c036000 tx=0x7f121c030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:35.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:35.816+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1224052e90 con 0x7f123406aed0 2023-08-25T09:50:36.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.005+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f123406c700 con 0x7f123406aed0 2023-08-25T09:50:36.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.005+0000 7f12317fa640 1 -- 172.21.15.71:0/2587121167 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1224018070 con 0x7f123406aed0 2023-08-25T09:50:36.007 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:50:36.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f121406a050 msgr2=0x7f121406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f121406a050 0x7f121406c510 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f121c036000 tx=0x7f121c030000 comp rx=0 tx=0).stop 2023-08-25T09:50:36.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 msgr2=0x7f1234154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 0x7f1234154bf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f122400c4d0 tx=0x7f122400c990 comp rx=0 tx=0).stop 2023-08-25T09:50:36.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 shutdown_connections 2023-08-25T09:50:36.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12340c72f0 0x7f12341e25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f121406a050 0x7f121406c510 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12340c76d0 0x7f12341e2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 --2- 172.21.15.71:0/2587121167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f123406aed0 0x7f1234154bf0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 >> 172.21.15.71:0/2587121167 conn(0x7f123406c280 msgr2=0x7f1234152b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:36.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 shutdown_connections 2023-08-25T09:50:36.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.008+0000 7f1239353640 1 -- 172.21.15.71:0/2587121167 wait complete. 2023-08-25T09:50:36.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:50:36.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.250+0000 7f2b47fff640 1 Processor -- start 2023-08-25T09:50:36.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.250+0000 7f2b47fff640 1 -- start start 2023-08-25T09:50:36.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b47fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2b480af290 0x7f2b4806b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b47fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2b48071760 0x7f2b4806a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b47fff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b480c0480 con 0x7f2b4806ad90 2023-08-25T09:50:36.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b47fff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4806b1a0 con 0x7f2b480af290 2023-08-25T09:50:36.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b47fff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b480bf500 con 0x7f2b48071760 2023-08-25T09:50:36.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b46ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2b480af290 0x7f2b4806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b46ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2b480af290 0x7f2b4806b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54860/0 (socket says 172.21.15.71:54860) 2023-08-25T09:50:36.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b46ffd640 1 -- 172.21.15.71:0/3856606759 learned_addr learned my addr 172.21.15.71:0/3856606759 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:36.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.252+0000 7f2b467fc640 1 --2- 172.21.15.71:0/3856606759 >> v2:172.21.15.164:3300/0 conn(0x7f2b48071760 0x7f2b4806a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.253+0000 7f2b46ffd640 1 -- 172.21.15.71:0/3856606759 >> v2:172.21.15.164:3300/0 conn(0x7f2b48071760 msgr2=0x7f2b4806a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:36.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.253+0000 7f2b46ffd640 1 --2- 172.21.15.71:0/3856606759 >> v2:172.21.15.164:3300/0 conn(0x7f2b48071760 0x7f2b4806a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.253+0000 7f2b46ffd640 1 -- 172.21.15.71:0/3856606759 >> v1:172.21.15.164:6789/0 conn(0x7f2b4806ad90 legacy=0x7f2b4815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:36.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.253+0000 7f2b46ffd640 1 -- 172.21.15.71:0/3856606759 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48069850 con 0x7f2b480af290 2023-08-25T09:50:36.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.254+0000 7f2b46ffd640 1 --2- 172.21.15.71:0/3856606759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b480af290 0x7f2b4806b300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b4815a0e0 tx=0x7f2b3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17c4b75088e05c7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:36.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.254+0000 7f2b45ffb640 1 -- 172.21.15.71:0/3856606759 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38031870 con 0x7f2b480af290 2023-08-25T09:50:36.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.254+0000 7f2b45ffb640 1 -- 172.21.15.71:0/3856606759 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380319d0 con 0x7f2b480af290 2023-08-25T09:50:36.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.254+0000 7f2b45ffb640 1 -- 172.21.15.71:0/3856606759 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38031d60 con 0x7f2b480af290 2023-08-25T09:50:36.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 -- 172.21.15.71:0/3856606759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b480af290 msgr2=0x7f2b4806b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3856606759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b480af290 0x7f2b4806b300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b4815a0e0 tx=0x7f2b3802b910 comp rx=0 tx=0).stop 2023-08-25T09:50:36.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 -- 172.21.15.71:0/3856606759 shutdown_connections 2023-08-25T09:50:36.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3856606759 >> v2:172.21.15.164:3300/0 conn(0x7f2b48071760 0x7f2b4806a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3856606759 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b480af290 0x7f2b4806b300 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 -- 172.21.15.71:0/3856606759 >> 172.21.15.71:0/3856606759 conn(0x7f2b4806c280 msgr2=0x7f2b4806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:36.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 -- 172.21.15.71:0/3856606759 shutdown_connections 2023-08-25T09:50:36.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.255+0000 7f2b47fff640 1 -- 172.21.15.71:0/3856606759 wait complete. 2023-08-25T09:50:36.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.257+0000 7f2b47fff640 1 Processor -- start 2023-08-25T09:50:36.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.257+0000 7f2b47fff640 1 -- start start 2023-08-25T09:50:36.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.257+0000 7f2b47fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b4806ad90 0x7f2b48070a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b47fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 0x7f2b48070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b47fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b480af290 0x7f2b481f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b47fff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b481eee70 con 0x7f2b48071760 2023-08-25T09:50:36.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b47fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480c41a0 con 0x7f2b480af290 2023-08-25T09:50:36.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b47fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4815b6a0 con 0x7f2b4806ad90 2023-08-25T09:50:36.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 0x7f2b48070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 0x7f2b48070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54870/0 (socket says 172.21.15.71:54870) 2023-08-25T09:50:36.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 -- 172.21.15.71:0/3583925335 learned_addr learned my addr 172.21.15.71:0/3583925335 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:36.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 -- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b4806ad90 msgr2=0x7f2b48070a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b46ffd640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b4806ad90 0x7f2b48070a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b4806ad90 0x7f2b48070a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 -- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b480af290 msgr2=0x7f2b481f16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b480af290 0x7f2b481f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.258+0000 7f2b467fc640 1 -- 172.21.15.71:0/3583925335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380094d0 con 0x7f2b48071760 2023-08-25T09:50:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.259+0000 7f2b467fc640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 0x7f2b48070f40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00d830 tx=0x7f2b3c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:36.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.259+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c00b6c0 con 0x7f2b48071760 2023-08-25T09:50:36.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.259+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c00b820 con 0x7f2b48071760 2023-08-25T09:50:36.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.259+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c00bbb0 con 0x7f2b48071760 2023-08-25T09:50:36.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.259+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4806ab30 con 0x7f2b48071760 2023-08-25T09:50:36.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.259+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b481ee1f0 con 0x7f2b48071760 2023-08-25T09:50:36.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.260+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4804e420 con 0x7f2b48071760 2023-08-25T09:50:36.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.261+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2b3c01e460 con 0x7f2b48071760 2023-08-25T09:50:36.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.262+0000 7f2b2f7fe640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b3406a050 0x7f2b3406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.262+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f2b3c0890c0 con 0x7f2b48071760 2023-08-25T09:50:36.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.263+0000 7f2b46ffd640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b3406a050 0x7f2b3406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.263+0000 7f2b46ffd640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b3406a050 0x7f2b3406c510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b38036000 tx=0x7f2b3803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:36.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.267+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2b3c0543e0 con 0x7f2b48071760 2023-08-25T09:50:36.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.460+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2b48071520 con 0x7f2b48071760 2023-08-25T09:50:36.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.461+0000 7f2b2f7fe640 1 -- 172.21.15.71:0/3583925335 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2b3c057a00 con 0x7f2b48071760 2023-08-25T09:50:36.462 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:50:36.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b3406a050 msgr2=0x7f2b3406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b3406a050 0x7f2b3406c510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b38036000 tx=0x7f2b3803c040 comp rx=0 tx=0).stop 2023-08-25T09:50:36.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 msgr2=0x7f2b48070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 0x7f2b48070f40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00d830 tx=0x7f2b3c00dcf0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 shutdown_connections 2023-08-25T09:50:36.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b480af290 0x7f2b481f16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b3406a050 0x7f2b3406c510 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b48071760 0x7f2b48070f40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 --2- 172.21.15.71:0/3583925335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b4806ad90 0x7f2b48070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 >> 172.21.15.71:0/3583925335 conn(0x7f2b4806c280 msgr2=0x7f2b480c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:36.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 shutdown_connections 2023-08-25T09:50:36.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.464+0000 7f2b47fff640 1 -- 172.21.15.71:0/3583925335 wait complete. 2023-08-25T09:50:36.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:50:36.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.700+0000 7f9e3716b640 1 Processor -- start 2023-08-25T09:50:36.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.700+0000 7f9e3716b640 1 -- start start 2023-08-25T09:50:36.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e3716b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 0x7f9e300c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e3716b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3006a960 con 0x7f9e300c3940 2023-08-25T09:50:36.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e3716b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3006ac20 con 0x7f9e3006cc70 2023-08-25T09:50:36.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e3716b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3006af00 con 0x7f9e301594d0 2023-08-25T09:50:36.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e35708640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9e300c3940 0x7f9e301598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47370/0 (socket says 172.21.15.71:47370) 2023-08-25T09:50:36.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e35708640 1 -- 172.21.15.71:0/3902302840 learned_addr learned my addr 172.21.15.71:0/3902302840 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:36.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.702+0000 7f9e34f07640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 0x7f9e300c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253291807 0 0) 0x7f9e3006a960 con 0x7f9e300c3940 2023-08-25T09:50:36.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e18003520 con 0x7f9e300c3940 2023-08-25T09:50:36.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327402531 0 0) 0x7f9e18003520 con 0x7f9e300c3940 2023-08-25T09:50:36.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 msgr2=0x7f9e300c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:36.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 0x7f9e300c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 >> v1:172.21.15.164:6789/0 conn(0x7f9e3006cc70 legacy=0x7f9e3014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e300c2e60 con 0x7f9e300c3940 2023-08-25T09:50:36.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f9e280019f0 con 0x7f9e300c3940 2023-08-25T09:50:36.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 >> v1:172.21.15.71:6789/0 conn(0x7f9e300c3940 legacy=0x7f9e301598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 0x7f9e18003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18003f90 0x7f9e180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3006af00 con 0x7f9e18004d00 2023-08-25T09:50:36.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1800dcf0 con 0x7f9e18003f90 2023-08-25T09:50:36.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1800deb0 con 0x7f9e301594d0 2023-08-25T09:50:36.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.703+0000 7f9e35708640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e2ffff640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18003f90 0x7f9e180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e35708640 1 -- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 msgr2=0x7f9e18003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:36.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e35708640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e301594d0 0x7f9e18003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e35708640 1 -- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18003f90 msgr2=0x7f9e180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e35708640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18003f90 0x7f9e180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e35708640 1 -- 172.21.15.71:0/3902302840 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18003520 con 0x7f9e18004d00 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.704+0000 7f9e2ffff640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18003f90 0x7f9e180047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.705+0000 7f9e35708640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800a830 tx=0x7f9e180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441c96c15169a340 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.705+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280072d0 con 0x7f9e18004d00 2023-08-25T09:50:36.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.705+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2802f070 con 0x7f9e18004d00 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.705+0000 7f9e2f7fe640 1 -- 172.21.15.71:0/3902302840 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28007690 con 0x7f9e18004d00 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e3716b640 1 -- 172.21.15.71:0/3902302840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e18004d00 msgr2=0x7f9e18005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3902302840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e18004d00 0x7f9e18005180 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800a830 tx=0x7f9e180057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e35708640 1 -- 172.21.15.71:0/3902302840 reap_dead start 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e3716b640 1 -- 172.21.15.71:0/3902302840 shutdown_connections 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e3716b640 1 -- 172.21.15.71:0/3902302840 >> 172.21.15.71:0/3902302840 conn(0x7f9e300654f0 msgr2=0x7f9e30065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e3716b640 1 -- 172.21.15.71:0/3902302840 shutdown_connections 2023-08-25T09:50:36.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.706+0000 7f9e3716b640 1 -- 172.21.15.71:0/3902302840 wait complete. 2023-08-25T09:50:36.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.708+0000 7f9e3716b640 1 Processor -- start 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.708+0000 7f9e3716b640 1 -- start start 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e3716b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18004d00 0x7f9e30144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e3716b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e30158db0 0x7f9e301e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e3716b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 0x7f9e301e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e3716b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e30158260 con 0x7f9e300c8b00 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e3716b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3014f390 con 0x7f9e18004d00 2023-08-25T09:50:36.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e3716b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e301eda30 con 0x7f9e30158db0 2023-08-25T09:50:36.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e35708640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 0x7f9e301e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e35708640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 0x7f9e301e9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54886/0 (socket says 172.21.15.71:54886) 2023-08-25T09:50:36.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e35708640 1 -- 172.21.15.71:0/3026093634 learned_addr learned my addr 172.21.15.71:0/3026093634 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:36.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e2ffff640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e30158db0 0x7f9e301e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.709+0000 7f9e34f07640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18004d00 0x7f9e30144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e35708640 1 -- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e30158db0 msgr2=0x7f9e301e99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e35708640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e30158db0 0x7f9e301e99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e35708640 1 -- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18004d00 msgr2=0x7f9e30144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e35708640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18004d00 0x7f9e30144670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e35708640 1 -- 172.21.15.71:0/3026093634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e301edec0 con 0x7f9e300c8b00 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e35708640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 0x7f9e301e9f30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9e2802cb70 tx=0x7f9e28001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28001250 con 0x7f9e300c8b00 2023-08-25T09:50:36.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e280070d0 con 0x7f9e300c8b00 2023-08-25T09:50:36.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28007440 con 0x7f9e300c8b00 2023-08-25T09:50:36.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3014f0b0 con 0x7f9e300c8b00 2023-08-25T09:50:36.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.710+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e301ed3e0 con 0x7f9e300c8b00 2023-08-25T09:50:36.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.712+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e301edbe0 con 0x7f9e300c8b00 2023-08-25T09:50:36.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.712+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9e28031030 con 0x7f9e300c8b00 2023-08-25T09:50:36.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.713+0000 7f9e2d7fa640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9e1c06a050 0x7f9e1c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:36.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.714+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9e280a9dd0 con 0x7f9e300c8b00 2023-08-25T09:50:36.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.714+0000 7f9e34f07640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9e1c06a050 0x7f9e1c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:36.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.714+0000 7f9e34f07640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9e1c06a050 0x7f9e1c06c510 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9e24004770 tx=0x7f9e24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:36.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.718+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9e301edbe0 con 0x7f9e300c8b00 2023-08-25T09:50:36.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.911+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e301edbe0 con 0x7f9e300c8b00 2023-08-25T09:50:36.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.911+0000 7f9e2d7fa640 1 -- 172.21.15.71:0/3026093634 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v31) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9e301edbe0 con 0x7f9e300c8b00 2023-08-25T09:50:36.913 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:36.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9e1c06a050 msgr2=0x7f9e1c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9e1c06a050 0x7f9e1c06c510 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9e24004770 tx=0x7f9e24008040 comp rx=0 tx=0).stop 2023-08-25T09:50:36.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 msgr2=0x7f9e301e9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:36.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 0x7f9e301e9f30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9e2802cb70 tx=0x7f9e28001d80 comp rx=0 tx=0).stop 2023-08-25T09:50:36.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 shutdown_connections 2023-08-25T09:50:36.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9e1c06a050 0x7f9e1c06c510 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e300c8b00 0x7f9e301e9f30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e30158db0 0x7f9e301e99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 --2- 172.21.15.71:0/3026093634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e18004d00 0x7f9e30144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 >> 172.21.15.71:0/3026093634 conn(0x7f9e300654f0 msgr2=0x7f9e3014d470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 shutdown_connections 2023-08-25T09:50:36.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:36.914+0000 7f9e3716b640 1 -- 172.21.15.71:0/3026093634 wait complete. 2023-08-25T09:50:36.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:50:37.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.145+0000 7f3c2ff5d640 1 Processor -- start 2023-08-25T09:50:37.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.145+0000 7f3c2ff5d640 1 -- start start 2023-08-25T09:50:37.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.146+0000 7f3c2ff5d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c280c4560 0x7f3c2806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.146+0000 7f3c2ff5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c281637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2ff5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2806eb60 con 0x7f3c28152d80 2023-08-25T09:50:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2ff5d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2806ca30 con 0x7f3c280c4560 2023-08-25T09:50:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2ff5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2806cf90 con 0x7f3c28070cf0 2023-08-25T09:50:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2d4f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c280c4560 0x7f3c2806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2d4f8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c280c4560 0x7f3c2806e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54894/0 (socket says 172.21.15.71:54894) 2023-08-25T09:50:37.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3398022775 learned_addr learned my addr 172.21.15.71:0/3398022775 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:37.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2e4fa640 1 --2- 172.21.15.71:0/3398022775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c281637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3398022775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 msgr2=0x7f3c281637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2d4f8640 1 --2- 172.21.15.71:0/3398022775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c281637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.147+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3398022775 >> v1:172.21.15.164:6789/0 conn(0x7f3c28152d80 legacy=0x7f3c280c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.148+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3398022775 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28158fd0 con 0x7f3c280c4560 2023-08-25T09:50:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.149+0000 7f3c2d4f8640 1 --2- 172.21.15.71:0/3398022775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2806e620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009870 tx=0x7f3c1802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46e7579ae0de7e3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.149+0000 7f3c2ccf7640 1 -- 172.21.15.71:0/3398022775 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18031640 con 0x7f3c280c4560 2023-08-25T09:50:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.149+0000 7f3c2ccf7640 1 -- 172.21.15.71:0/3398022775 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c180317a0 con 0x7f3c280c4560 2023-08-25T09:50:37.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.149+0000 7f3c2ccf7640 1 -- 172.21.15.71:0/3398022775 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18031b30 con 0x7f3c280c4560 2023-08-25T09:50:37.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3398022775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 msgr2=0x7f3c2806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3398022775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2806e620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009870 tx=0x7f3c1802b8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3398022775 shutdown_connections 2023-08-25T09:50:37.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3398022775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c281637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3398022775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2806e620 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3398022775 >> 172.21.15.71:0/3398022775 conn(0x7f3c280654f0 msgr2=0x7f3c28065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:37.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3398022775 shutdown_connections 2023-08-25T09:50:37.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.150+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3398022775 wait complete. 2023-08-25T09:50:37.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.152+0000 7f3c2ff5d640 1 Processor -- start 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.152+0000 7f3c2ff5d640 1 -- start start 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2ff5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c280c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2ff5d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2814a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2ff5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c28152d80 0x7f3c28157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2ff5d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c280c9c60 con 0x7f3c280c4560 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2ff5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28149c90 con 0x7f3c28152d80 2023-08-25T09:50:37.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2ff5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28153720 con 0x7f3c28070cf0 2023-08-25T09:50:37.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2d4f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2814a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2d4f8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2814a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54908/0 (socket says 172.21.15.71:54908) 2023-08-25T09:50:37.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3522411651 learned_addr learned my addr 172.21.15.71:0/3522411651 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:37.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 msgr2=0x7f3c280c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2dcf9640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c280c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2e4fa640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c28152d80 0x7f3c28157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.153+0000 7f3c2d4f8640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c280c32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c28152d80 msgr2=0x7f3c28157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c2d4f8640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c28152d80 0x7f3c28157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c2d4f8640 1 -- 172.21.15.71:0/3522411651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c180094d0 con 0x7f3c280c4560 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c2d4f8640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2814a120 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3c18000c00 tx=0x7f3c180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1803ac60 con 0x7f3c280c4560 2023-08-25T09:50:37.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1803adc0 con 0x7f3c280c4560 2023-08-25T09:50:37.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18044630 con 0x7f3c280c4560 2023-08-25T09:50:37.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c281566a0 con 0x7f3c280c4560 2023-08-25T09:50:37.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.154+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c281539e0 con 0x7f3c280c4560 2023-08-25T09:50:37.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.155+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c28069160 con 0x7f3c280c4560 2023-08-25T09:50:37.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.156+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3c1802f050 con 0x7f3c280c4560 2023-08-25T09:50:37.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.157+0000 7f3c1e7fc640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c0c06a050 0x7f3c0c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.157+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f3c180aa7c0 con 0x7f3c280c4560 2023-08-25T09:50:37.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.158+0000 7f3c2dcf9640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c0c06a050 0x7f3c0c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.158+0000 7f3c2dcf9640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c0c06a050 0x7f3c0c06c510 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3c24000f60 tx=0x7f3c24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:37.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.161+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3c18075ae0 con 0x7f3c280c4560 2023-08-25T09:50:37.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.353+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c2806e940 con 0x7f3c280c4560 2023-08-25T09:50:37.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.353+0000 7f3c1e7fc640 1 -- 172.21.15.71:0/3522411651 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v31) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c18031e60 con 0x7f3c280c4560 2023-08-25T09:50:37.355 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:50:37.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c0c06a050 msgr2=0x7f3c0c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c0c06a050 0x7f3c0c06c510 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3c24000f60 tx=0x7f3c24009040 comp rx=0 tx=0).stop 2023-08-25T09:50:37.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 msgr2=0x7f3c2814a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2814a120 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3c18000c00 tx=0x7f3c180046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 shutdown_connections 2023-08-25T09:50:37.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c28152d80 0x7f3c28157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c0c06a050 0x7f3c0c06c510 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c280c4560 0x7f3c2814a120 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 --2- 172.21.15.71:0/3522411651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c28070cf0 0x7f3c280c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 >> 172.21.15.71:0/3522411651 conn(0x7f3c280654f0 msgr2=0x7f3c280c8570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:37.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 shutdown_connections 2023-08-25T09:50:37.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.357+0000 7f3c2ff5d640 1 -- 172.21.15.71:0/3522411651 wait complete. 2023-08-25T09:50:37.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:50:37.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.591+0000 7f9ff1ead640 1 Processor -- start 2023-08-25T09:50:37.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.591+0000 7f9ff1ead640 1 -- start start 2023-08-25T09:50:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff1ead640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fec1542d0 0x7f9fec1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff1ead640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fec069850 0x7f9fec0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff1ead640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0c75c0 0x7f9fec0c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff1ead640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec0c4b60 con 0x7f9fec1542d0 2023-08-25T09:50:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff1ead640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec157bf0 con 0x7f9fec069850 2023-08-25T09:50:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff1ead640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec06a130 con 0x7f9fec0c75c0 2023-08-25T09:50:37.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.593+0000 7f9ff0eab640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fec1542d0 0x7f9fec1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fec1542d0 0x7f9fec1576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54920/0 (socket says 172.21.15.71:54920) 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 -- 172.21.15.71:0/45553657 learned_addr learned my addr 172.21.15.71:0/45553657 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9febfff640 1 --2- 172.21.15.71:0/45553657 >> v2:172.21.15.164:3300/0 conn(0x7f9fec069850 0x7f9fec0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff16ac640 1 --2- 172.21.15.71:0/45553657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0c75c0 0x7f9fec0c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 -- 172.21.15.71:0/45553657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0c75c0 msgr2=0x7f9fec0c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 --2- 172.21.15.71:0/45553657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0c75c0 0x7f9fec0c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 -- 172.21.15.71:0/45553657 >> v2:172.21.15.164:3300/0 conn(0x7f9fec069850 msgr2=0x7f9fec0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 --2- 172.21.15.71:0/45553657 >> v2:172.21.15.164:3300/0 conn(0x7f9fec069850 0x7f9fec0c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9ff0eab640 1 -- 172.21.15.71:0/45553657 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec1546b0 con 0x7f9fec1542d0 2023-08-25T09:50:37.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.594+0000 7f9febfff640 1 --2- 172.21.15.71:0/45553657 >> v2:172.21.15.164:3300/0 conn(0x7f9fec069850 0x7f9fec0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:50:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.596+0000 7f9ff0eab640 1 --2- 172.21.15.71:0/45553657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec1542d0 0x7f9fec1576b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0c13c0 tx=0x7f9fe002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac851c9c7c2118e4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.596+0000 7f9feb7fe640 1 -- 172.21.15.71:0/45553657 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0031870 con 0x7f9fec1542d0 2023-08-25T09:50:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.596+0000 7f9feb7fe640 1 -- 172.21.15.71:0/45553657 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe00319d0 con 0x7f9fec1542d0 2023-08-25T09:50:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.596+0000 7f9feb7fe640 1 -- 172.21.15.71:0/45553657 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0031d60 con 0x7f9fec1542d0 2023-08-25T09:50:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 -- 172.21.15.71:0/45553657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec1542d0 msgr2=0x7f9fec1576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/45553657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec1542d0 0x7f9fec1576b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9fec0c13c0 tx=0x7f9fe002b910 comp rx=0 tx=0).stop 2023-08-25T09:50:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 -- 172.21.15.71:0/45553657 shutdown_connections 2023-08-25T09:50:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/45553657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec0c75c0 0x7f9fec0c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/45553657 >> v2:172.21.15.164:3300/0 conn(0x7f9fec069850 0x7f9fec0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/45553657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec1542d0 0x7f9fec1576b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 -- 172.21.15.71:0/45553657 >> 172.21.15.71:0/45553657 conn(0x7f9fec06c280 msgr2=0x7f9fec06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 -- 172.21.15.71:0/45553657 shutdown_connections 2023-08-25T09:50:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.597+0000 7f9ff1ead640 1 -- 172.21.15.71:0/45553657 wait complete. 2023-08-25T09:50:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.599+0000 7f9ff1ead640 1 Processor -- start 2023-08-25T09:50:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.599+0000 7f9ff1ead640 1 -- start start 2023-08-25T09:50:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff1ead640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 0x7f9fec1ece90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff1ead640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec0c75c0 0x7f9fec14b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff1ead640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec1542d0 0x7f9fec148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff1ead640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec0719a0 con 0x7f9fec069850 2023-08-25T09:50:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff1ead640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec155880 con 0x7f9fec0c75c0 2023-08-25T09:50:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff1ead640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec1552c0 con 0x7f9fec1542d0 2023-08-25T09:50:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff0eab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 0x7f9fec1ece90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff0eab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 0x7f9fec1ece90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54928/0 (socket says 172.21.15.71:54928) 2023-08-25T09:50:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.601+0000 7f9ff0eab640 1 -- 172.21.15.71:0/1973627054 learned_addr learned my addr 172.21.15.71:0/1973627054 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9febfff640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec0c75c0 0x7f9fec14b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff16ac640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec1542d0 0x7f9fec148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff0eab640 1 -- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec1542d0 msgr2=0x7f9fec148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff0eab640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec1542d0 0x7f9fec148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff0eab640 1 -- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec0c75c0 msgr2=0x7f9fec14b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff0eab640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec0c75c0 0x7f9fec14b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff0eab640 1 -- 172.21.15.71:0/1973627054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe00094d0 con 0x7f9fec069850 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.602+0000 7f9ff0eab640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 0x7f9fec1ece90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9fe002b950 tx=0x7f9fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.603+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe003ad30 con 0x7f9fec069850 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.603+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe00443f0 con 0x7f9fec069850 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.603+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00447a0 con 0x7f9fec069850 2023-08-25T09:50:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.603+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec1555a0 con 0x7f9fec069850 2023-08-25T09:50:37.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.603+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec1ec970 con 0x7f9fec069850 2023-08-25T09:50:37.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.604+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec04e420 con 0x7f9fec069850 2023-08-25T09:50:37.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.605+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9fe0044900 con 0x7f9fec069850 2023-08-25T09:50:37.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.607+0000 7f9fe97fa640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd806a050 0x7f9fd806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:37.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.607+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9fe002d040 con 0x7f9fec069850 2023-08-25T09:50:37.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.607+0000 7f9febfff640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd806a050 0x7f9fd806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:37.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.608+0000 7f9febfff640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd806a050 0x7f9fd806c510 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc006fd0 tx=0x7f9fdc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:37.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.611+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9fe00762b0 con 0x7f9fec069850 2023-08-25T09:50:37.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.803+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9fec0af380 con 0x7f9fec069850 2023-08-25T09:50:37.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.804+0000 7f9fe97fa640 1 -- 172.21.15.71:0/1973627054 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9fe00798d0 con 0x7f9fec069850 2023-08-25T09:50:37.806 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd806a050 msgr2=0x7f9fd806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd806a050 0x7f9fd806c510 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc006fd0 tx=0x7f9fdc008040 comp rx=0 tx=0).stop 2023-08-25T09:50:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 msgr2=0x7f9fec1ece90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 0x7f9fec1ece90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9fe002b950 tx=0x7f9fe00046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 shutdown_connections 2023-08-25T09:50:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fec0c75c0 0x7f9fec14b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd806a050 0x7f9fd806c510 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fec1542d0 0x7f9fec148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 --2- 172.21.15.71:0/1973627054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fec069850 0x7f9fec1ece90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:37.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 >> 172.21.15.71:0/1973627054 conn(0x7f9fec06c280 msgr2=0x7f9fec152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:37.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 shutdown_connections 2023-08-25T09:50:37.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:37.806+0000 7f9ff1ead640 1 -- 172.21.15.71:0/1973627054 wait complete. 2023-08-25T09:50:37.827 INFO:tasks.mds_thrash:mds thrasher using random seed: 1692957037 2023-08-25T09:50:37.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:38.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.037+0000 7f7681bf3640 1 Processor -- start 2023-08-25T09:50:38.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.038+0000 7f7681bf3640 1 -- start start 2023-08-25T09:50:38.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.039+0000 7f7681bf3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 0x7f767c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.039+0000 7f7681bf3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c06a960 con 0x7f767c0c3940 2023-08-25T09:50:38.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.039+0000 7f7681bf3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c06ac20 con 0x7f767c1594d0 2023-08-25T09:50:38.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.039+0000 7f7681bf3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c06af00 con 0x7f767c06cc70 2023-08-25T09:50:38.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f767c0c3940 0x7f767c1598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47386/0 (socket says 172.21.15.71:47386) 2023-08-25T09:50:38.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767bfff640 1 -- 172.21.15.71:0/3881406594 learned_addr learned my addr 172.21.15.71:0/3881406594 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:38.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f7673fff640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 0x7f767c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717680632 0 0) 0x7f767c06a960 con 0x7f767c0c3940 2023-08-25T09:50:38.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7668003520 con 0x7f767c0c3940 2023-08-25T09:50:38.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2601842388 0 0) 0x7f7668003520 con 0x7f767c0c3940 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 msgr2=0x7f767c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 0x7f767c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 >> v1:172.21.15.164:6789/0 conn(0x7f767c1594d0 legacy=0x7f767c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c0c2e60 con 0x7f767c0c3940 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f76740019f0 con 0x7f767c0c3940 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.040+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 >> v1:172.21.15.71:6789/0 conn(0x7f767c0c3940 legacy=0x7f767c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767affd640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7668003a10 0x7f7668003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767affd640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76680047c0 0x7f7668004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767affd640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 0x7f7668005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c06af00 con 0x7f76680047c0 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766800dc90 con 0x7f7668003a10 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766800de50 con 0x7f767c06cc70 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f7673fff640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76680047c0 0x7f7668004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f7673fff640 1 -- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 msgr2=0x7f7668005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767bfff640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 0x7f7668005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f767b7fe640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7668003a10 0x7f7668003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f7673fff640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c06cc70 0x7f7668005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f7673fff640 1 -- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7668003a10 msgr2=0x7f7668003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f7673fff640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7668003a10 0x7f7668003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.041+0000 7f7673fff640 1 -- 172.21.15.71:0/3881406594 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7668003520 con 0x7f76680047c0 2023-08-25T09:50:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.042+0000 7f7673fff640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76680047c0 0x7f7668004c40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7664009ac0 tx=0x7f76680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1429e9c54956ab server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766402c6b0 con 0x7f76680047c0 2023-08-25T09:50:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766402c810 con 0x7f76680047c0 2023-08-25T09:50:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f767affd640 1 -- 172.21.15.71:0/3881406594 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766402cba0 con 0x7f76680047c0 2023-08-25T09:50:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f7681bf3640 1 -- 172.21.15.71:0/3881406594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76680047c0 msgr2=0x7f7668004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f7681bf3640 1 --2- 172.21.15.71:0/3881406594 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76680047c0 0x7f7668004c40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7664009ac0 tx=0x7f76680057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f767bfff640 1 -- 172.21.15.71:0/3881406594 reap_dead start 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f7681bf3640 1 -- 172.21.15.71:0/3881406594 shutdown_connections 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f7681bf3640 1 -- 172.21.15.71:0/3881406594 >> 172.21.15.71:0/3881406594 conn(0x7f767c0654f0 msgr2=0x7f767c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f7681bf3640 1 -- 172.21.15.71:0/3881406594 shutdown_connections 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.043+0000 7f7681bf3640 1 -- 172.21.15.71:0/3881406594 wait complete. 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.045+0000 7f7681bf3640 1 Processor -- start 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.045+0000 7f7681bf3640 1 -- start start 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7681bf3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76680047c0 0x7f767c1eac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7681bf3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 0x7f767c0c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7681bf3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c0c1b80 0x7f767c0c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7681bf3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c14be10 con 0x7f767c0c62a0 2023-08-25T09:50:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7681bf3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c1dfae0 con 0x7f76680047c0 2023-08-25T09:50:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7681bf3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c1ea1b0 con 0x7f767c0c1b80 2023-08-25T09:50:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7673fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 0x7f767c0c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7673fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 0x7f767c0c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54942/0 (socket says 172.21.15.71:54942) 2023-08-25T09:50:38.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.046+0000 7f7673fff640 1 -- 172.21.15.71:0/1492966474 learned_addr learned my addr 172.21.15.71:0/1492966474 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f767b7fe640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76680047c0 0x7f767c1eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f767bfff640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c0c1b80 0x7f767c0c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7673fff640 1 -- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c0c1b80 msgr2=0x7f767c0c2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7673fff640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c0c1b80 0x7f767c0c2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7673fff640 1 -- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76680047c0 msgr2=0x7f767c1eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7673fff640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76680047c0 0x7f767c1eac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7673fff640 1 -- 172.21.15.71:0/1492966474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c1dfd80 con 0x7f767c0c62a0 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f767b7fe640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76680047c0 0x7f767c1eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7673fff640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 0x7f767c0c9c00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7664009ac0 tx=0x7f7664004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664003810 con 0x7f767c0c62a0 2023-08-25T09:50:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7664003970 con 0x7f767c0c62a0 2023-08-25T09:50:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767c1e0060 con 0x7f767c0c62a0 2023-08-25T09:50:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.047+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664003d00 con 0x7f767c0c62a0 2023-08-25T09:50:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.048+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c1ea470 con 0x7f767c0c62a0 2023-08-25T09:50:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.048+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767c069160 con 0x7f767c0c62a0 2023-08-25T09:50:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.049+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7664040450 con 0x7f767c0c62a0 2023-08-25T09:50:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.050+0000 7f7678ff9640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f765c06a080 0x7f765c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.051+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f76640aa8a0 con 0x7f767c0c62a0 2023-08-25T09:50:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.051+0000 7f767b7fe640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f765c06a080 0x7f765c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.051+0000 7f767b7fe640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f765c06a080 0x7f765c06c540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f766c0016f0 tx=0x7f766c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.054+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7664075c50 con 0x7f767c0c62a0 2023-08-25T09:50:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.243+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f767c0c18d0 con 0x7f767c0c62a0 2023-08-25T09:50:38.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.246+0000 7f7678ff9640 1 -- 172.21.15.71:0/1492966474 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7f76640044e0 con 0x7f767c0c62a0 2023-08-25T09:50:38.248 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:38.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:38.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f765c06a080 msgr2=0x7f765c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f765c06a080 0x7f765c06c540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f766c0016f0 tx=0x7f766c009040 comp rx=0 tx=0).stop 2023-08-25T09:50:38.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 msgr2=0x7f767c0c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 0x7f767c0c9c00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7664009ac0 tx=0x7f7664004730 comp rx=0 tx=0).stop 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 shutdown_connections 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f765c06a080 0x7f765c06c540 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f767c0c1b80 0x7f767c0c2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f767c0c62a0 0x7f767c0c9c00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 --2- 172.21.15.71:0/1492966474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76680047c0 0x7f767c1eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 >> 172.21.15.71:0/1492966474 conn(0x7f767c0654f0 msgr2=0x7f767c1477a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:38.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 shutdown_connections 2023-08-25T09:50:38.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.249+0000 7f7681bf3640 1 -- 172.21.15.71:0/1492966474 wait complete. 2023-08-25T09:50:38.269 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:50:38.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:50:38.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.484+0000 7f14f0355640 1 Processor -- start 2023-08-25T09:50:38.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.485+0000 7f14f0355640 1 -- start start 2023-08-25T09:50:38.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.486+0000 7f14f0355640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f14e81510e0 0x7f14e80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.486+0000 7f14f0355640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80c1dc0 con 0x7f14e80c3a40 2023-08-25T09:50:38.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.487+0000 7f14f0355640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e8156e00 con 0x7f14e80c3660 2023-08-25T09:50:38.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.487+0000 7f14f0355640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e8157a80 con 0x7f14e81510e0 2023-08-25T09:50:38.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.487+0000 7f14ee8f2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f14e81510e0 0x7f14e80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.487+0000 7f14ee8f2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f14e81510e0 0x7f14e80c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54954/0 (socket says 172.21.15.71:54954) 2023-08-25T09:50:38.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.487+0000 7f14ee8f2640 1 -- 172.21.15.71:0/994687801 learned_addr learned my addr 172.21.15.71:0/994687801 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:38.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.488+0000 7f14ee8f2640 1 -- 172.21.15.71:0/994687801 >> v1:172.21.15.164:6789/0 conn(0x7f14e80c3660 legacy=0x7f14e806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.488+0000 7f14ee8f2640 1 -- 172.21.15.71:0/994687801 >> v1:172.21.15.71:6789/0 conn(0x7f14e80c3a40 legacy=0x7f14e81509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.488+0000 7f14ee8f2640 1 -- 172.21.15.71:0/994687801 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e81570e0 con 0x7f14e81510e0 2023-08-25T09:50:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.489+0000 7f14ee8f2640 1 --2- 172.21.15.71:0/994687801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e81510e0 0x7f14e80c1710 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009e50 tx=0x7f14e002c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724bab12426ba855 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.489+0000 7f14ed0ef640 1 -- 172.21.15.71:0/994687801 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e002cc30 con 0x7f14e81510e0 2023-08-25T09:50:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.489+0000 7f14ed0ef640 1 -- 172.21.15.71:0/994687801 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e002cd90 con 0x7f14e81510e0 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.489+0000 7f14ed0ef640 1 -- 172.21.15.71:0/994687801 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e0034610 con 0x7f14e81510e0 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.489+0000 7f14f0355640 1 -- 172.21.15.71:0/994687801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e81510e0 msgr2=0x7f14e80c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.489+0000 7f14f0355640 1 --2- 172.21.15.71:0/994687801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e81510e0 0x7f14e80c1710 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009e50 tx=0x7f14e002c090 comp rx=0 tx=0).stop 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.490+0000 7f14f0355640 1 -- 172.21.15.71:0/994687801 shutdown_connections 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.490+0000 7f14f0355640 1 --2- 172.21.15.71:0/994687801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e81510e0 0x7f14e80c1710 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.490+0000 7f14f0355640 1 -- 172.21.15.71:0/994687801 >> 172.21.15.71:0/994687801 conn(0x7f14e80654f0 msgr2=0x7f14e8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.490+0000 7f14f0355640 1 -- 172.21.15.71:0/994687801 shutdown_connections 2023-08-25T09:50:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.490+0000 7f14f0355640 1 -- 172.21.15.71:0/994687801 wait complete. 2023-08-25T09:50:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.491+0000 7f14f0355640 1 Processor -- start 2023-08-25T09:50:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 -- start start 2023-08-25T09:50:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e80c3660 0x7f14e8156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 0x7f14e8152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e81510e0 0x7f14e806f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e81566c0 con 0x7f14e80c3a40 2023-08-25T09:50:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e81f2970 con 0x7f14e80c3660 2023-08-25T09:50:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.492+0000 7f14f0355640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8153520 con 0x7f14e81510e0 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 0x7f14e8152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 0x7f14e8152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54958/0 (socket says 172.21.15.71:54958) 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 -- 172.21.15.71:0/2936929160 learned_addr learned my addr 172.21.15.71:0/2936929160 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ee0f1640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e80c3660 0x7f14e8156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ee8f2640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e81510e0 0x7f14e806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 -- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e81510e0 msgr2=0x7f14e806f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e81510e0 0x7f14e806f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 -- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e80c3660 msgr2=0x7f14e8156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e80c3660 0x7f14e8156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.493+0000 7f14ed8f0640 1 -- 172.21.15.71:0/2936929160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e0009b30 con 0x7f14e80c3a40 2023-08-25T09:50:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.494+0000 7f14ed8f0640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 0x7f14e8152fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f14d800c3c0 tx=0x7f14d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.494+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d80052d0 con 0x7f14e80c3a40 2023-08-25T09:50:38.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.494+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d8005450 con 0x7f14e80c3a40 2023-08-25T09:50:38.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.494+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e81f2690 con 0x7f14e80c3a40 2023-08-25T09:50:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.494+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8003830 con 0x7f14e80c3a40 2023-08-25T09:50:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.494+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e8151960 con 0x7f14e80c3a40 2023-08-25T09:50:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.495+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e80c9bb0 con 0x7f14e80c3a40 2023-08-25T09:50:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.496+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f14d8003990 con 0x7f14e80c3a40 2023-08-25T09:50:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.497+0000 7f14deffd640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a050 0x7f14cc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.497+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f14d8087ca0 con 0x7f14e80c3a40 2023-08-25T09:50:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.497+0000 7f14ee0f1640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a050 0x7f14cc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.498+0000 7f14ee0f1640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a050 0x7f14cc06c510 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f14e4000f60 tx=0x7f14e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.501+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f14d80530f0 con 0x7f14e80c3a40 2023-08-25T09:50:38.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.695+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14e8069160 con 0x7f14e80c3a40 2023-08-25T09:50:38.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.696+0000 7f14deffd640 1 -- 172.21.15.71:0/2936929160 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14d8017020 con 0x7f14e80c3a40 2023-08-25T09:50:38.697 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:50:38.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.698+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a050 msgr2=0x7f14cc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a050 0x7f14cc06c510 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f14e4000f60 tx=0x7f14e4009040 comp rx=0 tx=0).stop 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 msgr2=0x7f14e8152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 0x7f14e8152fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f14d800c3c0 tx=0x7f14d800c880 comp rx=0 tx=0).stop 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 shutdown_connections 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14cc06a050 0x7f14cc06c510 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e81510e0 0x7f14e806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e80c3a40 0x7f14e8152fe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 --2- 172.21.15.71:0/2936929160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e80c3660 0x7f14e8156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 >> 172.21.15.71:0/2936929160 conn(0x7f14e80654f0 msgr2=0x7f14e80c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:38.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 shutdown_connections 2023-08-25T09:50:38.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.699+0000 7f14f0355640 1 -- 172.21.15.71:0/2936929160 wait complete. 2023-08-25T09:50:38.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:50:38.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.936+0000 7ff0a34e0640 1 Processor -- start 2023-08-25T09:50:38.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.936+0000 7ff0a34e0640 1 -- start start 2023-08-25T09:50:38.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.937+0000 7ff0a34e0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff09c0c3660 0x7ff09c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.937+0000 7ff0a34e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 0x7ff09c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.937+0000 7ff0a34e0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0c1dc0 con 0x7ff09c0c3a40 2023-08-25T09:50:38.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.937+0000 7ff0a34e0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09c157a80 con 0x7ff09c0c3660 2023-08-25T09:50:38.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.937+0000 7ff0a34e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c151360 con 0x7ff09c1509d0 2023-08-25T09:50:38.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.938+0000 7ff0a0a7b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff09c0c3a40 0x7ff09c06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47406/0 (socket says 172.21.15.71:47406) 2023-08-25T09:50:38.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.938+0000 7ff0a0a7b640 1 -- 172.21.15.71:0/3620853455 learned_addr learned my addr 172.21.15.71:0/3620853455 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:38.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.938+0000 7ff0a127c640 1 --2- 172.21.15.71:0/3620853455 >> v2:172.21.15.164:3300/0 conn(0x7ff09c0c3660 0x7ff09c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.938+0000 7ff0a1a7d640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 0x7ff09c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.938+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55708308 0 0) 0x7ff09c0c1dc0 con 0x7ff09c0c3a40 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.938+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff088003520 con 0x7ff09c0c3a40 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288390963 0 0) 0x7ff088003520 con 0x7ff09c0c3a40 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 msgr2=0x7ff09c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 0x7ff09c150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 >> v2:172.21.15.164:3300/0 conn(0x7ff09c0c3660 msgr2=0x7ff09c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 --2- 172.21.15.71:0/3620853455 >> v2:172.21.15.164:3300/0 conn(0x7ff09c0c3660 0x7ff09c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c156e00 con 0x7ff09c0c3a40 2023-08-25T09:50:38.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff08c001670 con 0x7ff09c0c3a40 2023-08-25T09:50:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 >> v1:172.21.15.71:6789/0 conn(0x7ff09c0c3a40 legacy=0x7ff09c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff088003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0880047c0 0x7ff088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 0x7ff088005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c157a80 con 0x7ff0880039e0 2023-08-25T09:50:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c151360 con 0x7ff0880047c0 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08800b9d0 con 0x7ff09c1509d0 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff0a127c640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff088003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff0a127c640 1 -- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 msgr2=0x7ff088005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff0a127c640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c1509d0 0x7ff088005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff0a127c640 1 -- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0880047c0 msgr2=0x7ff088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.939+0000 7ff0a127c640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0880047c0 0x7ff088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.940+0000 7ff0a127c640 1 -- 172.21.15.71:0/3620853455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088003520 con 0x7ff0880039e0 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff0a127c640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff088003e00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff098009ac0 tx=0x7ff0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdfc8d312743f4e1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09802c710 con 0x7ff0880039e0 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09802c870 con 0x7ff0880039e0 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff093fff640 1 -- 172.21.15.71:0/3620853455 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09802cc00 con 0x7ff0880039e0 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff0a34e0640 1 -- 172.21.15.71:0/3620853455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 msgr2=0x7ff088003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/3620853455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff088003e00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff098009ac0 tx=0x7ff0880057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.941+0000 7ff0a1a7d640 1 -- 172.21.15.71:0/3620853455 reap_dead start 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.942+0000 7ff0a34e0640 1 -- 172.21.15.71:0/3620853455 shutdown_connections 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.942+0000 7ff0a34e0640 1 -- 172.21.15.71:0/3620853455 >> 172.21.15.71:0/3620853455 conn(0x7ff09c0654f0 msgr2=0x7ff09c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.942+0000 7ff0a34e0640 1 -- 172.21.15.71:0/3620853455 shutdown_connections 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.942+0000 7ff0a34e0640 1 -- 172.21.15.71:0/3620853455 wait complete. 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.943+0000 7ff0a34e0640 1 Processor -- start 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 -- start start 2023-08-25T09:50:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff09c153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c06ff80 0x7ff09c152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff09c1525c0 0x7ff09c145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c152a40 con 0x7ff0880039e0 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c152dd0 con 0x7ff09c1525c0 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.944+0000 7ff0a34e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c065b30 con 0x7ff09c06ff80 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff09c153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff09c153ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54974/0 (socket says 172.21.15.71:54974) 2023-08-25T09:50:38.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 -- 172.21.15.71:0/492039419 learned_addr learned my addr 172.21.15.71:0/492039419 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:38.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 -- 172.21.15.71:0/492039419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c06ff80 msgr2=0x7ff09c152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c06ff80 0x7ff09c152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 -- 172.21.15.71:0/492039419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff09c1525c0 msgr2=0x7ff09c145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:38.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff09c1525c0 0x7ff09c145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.945+0000 7ff0a127c640 1 -- 172.21.15.71:0/492039419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c151b10 con 0x7ff0880039e0 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.946+0000 7ff0a127c640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff09c153ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff09802c2b0 tx=0x7ff098004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.946+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09802de90 con 0x7ff0880039e0 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.946+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff098003610 con 0x7ff0880039e0 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.946+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980039c0 con 0x7ff0880039e0 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.946+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09c06b070 con 0x7ff0880039e0 2023-08-25T09:50:38.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.946+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09c06a8a0 con 0x7ff0880039e0 2023-08-25T09:50:38.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.947+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09c069160 con 0x7ff0880039e0 2023-08-25T09:50:38.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.948+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff098003b20 con 0x7ff0880039e0 2023-08-25T09:50:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.949+0000 7ff091ffb640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff08006a050 0x7ff08006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.949+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff0980aa710 con 0x7ff0880039e0 2023-08-25T09:50:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.949+0000 7ff0a0a7b640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff08006a050 0x7ff08006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.950+0000 7ff0a0a7b640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff08006a050 0x7ff08006c510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0c5eb0 tx=0x7ff08c001430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:38.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:38.954+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff098075b60 con 0x7ff0880039e0 2023-08-25T09:50:39.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.135+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff09c154680 con 0x7ff0880039e0 2023-08-25T09:50:39.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.136+0000 7ff091ffb640 1 -- 172.21.15.71:0/492039419 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff098079180 con 0x7ff0880039e0 2023-08-25T09:50:39.138 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:50:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff08006a050 msgr2=0x7ff08006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff08006a050 0x7ff08006c510 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0c5eb0 tx=0x7ff08c001430 comp rx=0 tx=0).stop 2023-08-25T09:50:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 msgr2=0x7ff09c153ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff09c153ef0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff09802c2b0 tx=0x7ff098004730 comp rx=0 tx=0).stop 2023-08-25T09:50:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 shutdown_connections 2023-08-25T09:50:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff09c1525c0 0x7ff09c145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff08006a050 0x7ff08006c510 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff09c06ff80 0x7ff09c152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 --2- 172.21.15.71:0/492039419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0880039e0 0x7ff09c153ef0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.139+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 >> 172.21.15.71:0/492039419 conn(0x7ff09c0654f0 msgr2=0x7ff09c0c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.140+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 shutdown_connections 2023-08-25T09:50:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.140+0000 7ff0a34e0640 1 -- 172.21.15.71:0/492039419 wait complete. 2023-08-25T09:50:39.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:50:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.368+0000 7ff3d2ca7640 1 Processor -- start 2023-08-25T09:50:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.368+0000 7ff3d2ca7640 1 -- start start 2023-08-25T09:50:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d2ca7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff3cc0cc6b0 0x7ff3cc156f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d2ca7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 0x7ff3cc070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d2ca7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0c58a0 con 0x7ff3cc0c6060 2023-08-25T09:50:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d2ca7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc071350 con 0x7ff3cc0cc6b0 2023-08-25T09:50:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d2ca7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc157540 con 0x7ff3cc0cca90 2023-08-25T09:50:39.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d1244640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff3cc0c6060 0x7ff3cc150db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47412/0 (socket says 172.21.15.71:47412) 2023-08-25T09:50:39.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.370+0000 7ff3d1244640 1 -- 172.21.15.71:0/772171984 learned_addr learned my addr 172.21.15.71:0/772171984 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:39.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cbfff640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 0x7ff3cc070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3d0a43640 1 --2- 172.21.15.71:0/772171984 >> v2:172.21.15.164:3300/0 conn(0x7ff3cc0cc6b0 0x7ff3cc156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616346328 0 0) 0x7ff3cc0c58a0 con 0x7ff3cc0c6060 2023-08-25T09:50:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc003520 con 0x7ff3cc0c6060 2023-08-25T09:50:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757038389 0 0) 0x7ff3bc003520 con 0x7ff3cc0c6060 2023-08-25T09:50:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 msgr2=0x7ff3cc070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 0x7ff3cc070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 >> v2:172.21.15.164:3300/0 conn(0x7ff3cc0cc6b0 msgr2=0x7ff3cc156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 --2- 172.21.15.71:0/772171984 >> v2:172.21.15.164:3300/0 conn(0x7ff3cc0cc6b0 0x7ff3cc156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc157820 con 0x7ff3cc0c6060 2023-08-25T09:50:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff3c40019f0 con 0x7ff3cc0c6060 2023-08-25T09:50:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 >> v1:172.21.15.71:6789/0 conn(0x7ff3cc0c6060 legacy=0x7ff3cc150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 0x7ff3bc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.371+0000 7ff3cb7fe640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc003f60 0x7ff3bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3cb7fe640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc004d00 0x7ff3bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc071350 con 0x7ff3bc004d00 2023-08-25T09:50:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc157540 con 0x7ff3bc003f60 2023-08-25T09:50:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc00ba30 con 0x7ff3cc0cca90 2023-08-25T09:50:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d1244640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc004d00 0x7ff3bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d0a43640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 0x7ff3bc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d1244640 1 -- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 msgr2=0x7ff3bc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:39.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d1244640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc0cca90 0x7ff3bc0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d1244640 1 -- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc003f60 msgr2=0x7ff3bc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:39.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3cbfff640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc003f60 0x7ff3bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d1244640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc003f60 0x7ff3bc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.372+0000 7ff3d1244640 1 -- 172.21.15.71:0/772171984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc003520 con 0x7ff3bc004d00 2023-08-25T09:50:39.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.373+0000 7ff3d1244640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc004d00 0x7ff3bc005180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400a860 tx=0x7ff3bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3953ab24e2c425cc server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:39.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.373+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4034630 con 0x7ff3bc004d00 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c402f070 con 0x7ff3bc004d00 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3cb7fe640 1 -- 172.21.15.71:0/772171984 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4034a30 con 0x7ff3bc004d00 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/772171984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc004d00 msgr2=0x7ff3bc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/772171984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3bc004d00 0x7ff3bc005180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400a860 tx=0x7ff3bc0057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d1244640 1 -- 172.21.15.71:0/772171984 reap_dead start 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/772171984 shutdown_connections 2023-08-25T09:50:39.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/772171984 >> 172.21.15.71:0/772171984 conn(0x7ff3cc069e80 msgr2=0x7ff3cc0654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/772171984 shutdown_connections 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.374+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/772171984 wait complete. 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.376+0000 7ff3d2ca7640 1 Processor -- start 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.376+0000 7ff3d2ca7640 1 -- start start 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3d2ca7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc004d00 0x7ff3cc156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3d2ca7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 0x7ff3cc1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3d2ca7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc14e3b0 0x7ff3cc06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3d2ca7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc06c1f0 con 0x7ff3cc0cb740 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3d2ca7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc156ce0 con 0x7ff3bc004d00 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3d2ca7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc1ee9b0 con 0x7ff3cc14e3b0 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 0x7ff3cc1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 0x7ff3cc1566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55006/0 (socket says 172.21.15.71:55006) 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.377+0000 7ff3cbfff640 1 -- 172.21.15.71:0/2109411845 learned_addr learned my addr 172.21.15.71:0/2109411845 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3d1244640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc14e3b0 0x7ff3cc06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3d0a43640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc004d00 0x7ff3cc156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3cbfff640 1 -- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc14e3b0 msgr2=0x7ff3cc06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3cbfff640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc14e3b0 0x7ff3cc06acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3cbfff640 1 -- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc004d00 msgr2=0x7ff3cc156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3cbfff640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc004d00 0x7ff3cc156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3cbfff640 1 -- 172.21.15.71:0/2109411845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc1eec90 con 0x7ff3cc0cb740 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3d1244640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc14e3b0 0x7ff3cc06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3d0a43640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc004d00 0x7ff3cc156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3cbfff640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 0x7ff3cc1566a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8010860 tx=0x7ff3b8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8009610 con 0x7ff3cc0cb740 2023-08-25T09:50:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b800eb60 con 0x7ff3cc0cb740 2023-08-25T09:50:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc06c460 con 0x7ff3cc0cb740 2023-08-25T09:50:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.378+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b801c400 con 0x7ff3cc0cb740 2023-08-25T09:50:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.379+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc15b270 con 0x7ff3cc0cb740 2023-08-25T09:50:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.380+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc1eee20 con 0x7ff3cc0cb740 2023-08-25T09:50:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.381+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff3b801c560 con 0x7ff3cc0cb740 2023-08-25T09:50:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.382+0000 7ff3c97fa640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff3b006a050 0x7ff3b006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.382+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7ff3b8088e70 con 0x7ff3cc0cb740 2023-08-25T09:50:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.382+0000 7ff3d0a43640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff3b006a050 0x7ff3b006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.383+0000 7ff3d0a43640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff3b006a050 0x7ff3b006c510 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009a60 tx=0x7ff3c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.386+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff3b80542e0 con 0x7ff3cc0cb740 2023-08-25T09:50:39.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.565+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3cc069160 con 0x7ff3cc0cb740 2023-08-25T09:50:39.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.566+0000 7ff3c97fa640 1 -- 172.21.15.71:0/2109411845 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v31) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3b8057900 con 0x7ff3cc0cb740 2023-08-25T09:50:39.568 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:39.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff3b006a050 msgr2=0x7ff3b006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff3b006a050 0x7ff3b006c510 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009a60 tx=0x7ff3c0008040 comp rx=0 tx=0).stop 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 msgr2=0x7ff3cc1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 0x7ff3cc1566a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8010860 tx=0x7ff3b8010d20 comp rx=0 tx=0).stop 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 shutdown_connections 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff3b006a050 0x7ff3b006c510 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3cc14e3b0 0x7ff3cc06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff3cc0cb740 0x7ff3cc1566a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 --2- 172.21.15.71:0/2109411845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3bc004d00 0x7ff3cc156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 >> 172.21.15.71:0/2109411845 conn(0x7ff3cc069e80 msgr2=0x7ff3cc0cab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:39.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 shutdown_connections 2023-08-25T09:50:39.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.569+0000 7ff3d2ca7640 1 -- 172.21.15.71:0/2109411845 wait complete. 2023-08-25T09:50:39.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:50:39.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.800+0000 7f9e16955640 1 Processor -- start 2023-08-25T09:50:39.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.800+0000 7f9e16955640 1 -- start start 2023-08-25T09:50:39.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e16955640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9e100c3a40 0x7f9e100c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e16955640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9e101509d0 0x7f9e10150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e16955640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100c1dc0 con 0x7f9e100c3660 2023-08-25T09:50:39.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e16955640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e10157a80 con 0x7f9e101509d0 2023-08-25T09:50:39.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e16955640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e10151320 con 0x7f9e100c3a40 2023-08-25T09:50:39.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e14ef2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9e101509d0 0x7f9e10150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e14ef2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9e101509d0 0x7f9e10150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55016/0 (socket says 172.21.15.71:55016) 2023-08-25T09:50:39.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.802+0000 7f9e14ef2640 1 -- 172.21.15.71:0/1752316755 learned_addr learned my addr 172.21.15.71:0/1752316755 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:39.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.803+0000 7f9e0f7fe640 1 --2- 172.21.15.71:0/1752316755 >> v2:172.21.15.164:3300/0 conn(0x7f9e100c3a40 0x7f9e100c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.803+0000 7f9e14ef2640 1 -- 172.21.15.71:0/1752316755 >> v2:172.21.15.164:3300/0 conn(0x7f9e100c3a40 msgr2=0x7f9e100c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.803+0000 7f9e14ef2640 1 --2- 172.21.15.71:0/1752316755 >> v2:172.21.15.164:3300/0 conn(0x7f9e100c3a40 0x7f9e100c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.803+0000 7f9e14ef2640 1 -- 172.21.15.71:0/1752316755 >> v1:172.21.15.164:6789/0 conn(0x7f9e100c3660 legacy=0x7f9e1006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.803+0000 7f9e14ef2640 1 -- 172.21.15.71:0/1752316755 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10156e00 con 0x7f9e101509d0 2023-08-25T09:50:39.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.804+0000 7f9e14ef2640 1 --2- 172.21.15.71:0/1752316755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e101509d0 0x7f9e10150de0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9e10067520 tx=0x7f9e0802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ea65a11fc31791 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:39.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e0effd640 1 -- 172.21.15.71:0/1752316755 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0802cb10 con 0x7f9e101509d0 2023-08-25T09:50:39.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e0effd640 1 -- 172.21.15.71:0/1752316755 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0802cc70 con 0x7f9e101509d0 2023-08-25T09:50:39.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e0effd640 1 -- 172.21.15.71:0/1752316755 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080343f0 con 0x7f9e101509d0 2023-08-25T09:50:39.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e16955640 1 -- 172.21.15.71:0/1752316755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e101509d0 msgr2=0x7f9e10150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e16955640 1 --2- 172.21.15.71:0/1752316755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e101509d0 0x7f9e10150de0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9e10067520 tx=0x7f9e0802bf70 comp rx=0 tx=0).stop 2023-08-25T09:50:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e16955640 1 -- 172.21.15.71:0/1752316755 shutdown_connections 2023-08-25T09:50:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e16955640 1 --2- 172.21.15.71:0/1752316755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e101509d0 0x7f9e10150de0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e16955640 1 --2- 172.21.15.71:0/1752316755 >> v2:172.21.15.164:3300/0 conn(0x7f9e100c3a40 0x7f9e100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.805+0000 7f9e16955640 1 -- 172.21.15.71:0/1752316755 >> 172.21.15.71:0/1752316755 conn(0x7f9e100654f0 msgr2=0x7f9e10065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.806+0000 7f9e16955640 1 -- 172.21.15.71:0/1752316755 shutdown_connections 2023-08-25T09:50:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.806+0000 7f9e16955640 1 -- 172.21.15.71:0/1752316755 wait complete. 2023-08-25T09:50:39.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.808+0000 7f9e16955640 1 Processor -- start 2023-08-25T09:50:39.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.808+0000 7f9e16955640 1 -- start start 2023-08-25T09:50:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e16955640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e100c3660 0x7f9e1014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e16955640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 0x7f9e1014c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e16955640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e101509d0 0x7f9e1014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e16955640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e101f2560 con 0x7f9e100c3a40 2023-08-25T09:50:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e16955640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1015aa30 con 0x7f9e101509d0 2023-08-25T09:50:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e16955640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1015ad10 con 0x7f9e100c3660 2023-08-25T09:50:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e0f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 0x7f9e1014c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e0f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 0x7f9e1014c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55026/0 (socket says 172.21.15.71:55026) 2023-08-25T09:50:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.809+0000 7f9e0f7fe640 1 -- 172.21.15.71:0/4113398140 learned_addr learned my addr 172.21.15.71:0/4113398140 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0ffff640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e100c3660 0x7f9e1014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0f7fe640 1 -- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e100c3660 msgr2=0x7f9e1014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0f7fe640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e100c3660 0x7f9e1014b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0f7fe640 1 -- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e101509d0 msgr2=0x7f9e1014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:39.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e14ef2640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e101509d0 0x7f9e1014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0f7fe640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e101509d0 0x7f9e1014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:39.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0f7fe640 1 -- 172.21.15.71:0/4113398140 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08009b30 con 0x7f9e100c3a40 2023-08-25T09:50:39.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e14ef2640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e101509d0 0x7f9e1014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:39.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0f7fe640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 0x7f9e1014c1f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d7d0 tx=0x7f9e0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:39.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.810+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400b6c0 con 0x7f9e100c3a40 2023-08-25T09:50:39.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.811+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0400b820 con 0x7f9e100c3a40 2023-08-25T09:50:39.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.811+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400bbb0 con 0x7f9e100c3a40 2023-08-25T09:50:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.811+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1015aff0 con 0x7f9e100c3a40 2023-08-25T09:50:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.811+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1014e960 con 0x7f9e100c3a40 2023-08-25T09:50:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.812+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e100c29c0 con 0x7f9e100c3a40 2023-08-25T09:50:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.813+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9e0400bd10 con 0x7f9e100c3a40 2023-08-25T09:50:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.814+0000 7f9e0cff9640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9dfc06a080 0x7f9dfc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.815+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9e04088ec0 con 0x7f9e100c3a40 2023-08-25T09:50:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.815+0000 7f9e0ffff640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9dfc06a080 0x7f9dfc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.815+0000 7f9e0ffff640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9dfc06a080 0x7f9dfc06c540 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000a750 tx=0x7f9e00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:39.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:39.819+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9e04054150 con 0x7f9e100c3a40 2023-08-25T09:50:40.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.006+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e10154680 con 0x7f9e100c3a40 2023-08-25T09:50:40.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.007+0000 7f9e0cff9640 1 -- 172.21.15.71:0/4113398140 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v31) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e04057770 con 0x7f9e100c3a40 2023-08-25T09:50:40.008 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:50:40.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9dfc06a080 msgr2=0x7f9dfc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9dfc06a080 0x7f9dfc06c540 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000a750 tx=0x7f9e00008040 comp rx=0 tx=0).stop 2023-08-25T09:50:40.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 msgr2=0x7f9e1014c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 0x7f9e1014c1f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d7d0 tx=0x7f9e0400dc90 comp rx=0 tx=0).stop 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 shutdown_connections 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9e101509d0 0x7f9e1014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9dfc06a080 0x7f9dfc06c540 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e100c3a40 0x7f9e1014c1f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 --2- 172.21.15.71:0/4113398140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9e100c3660 0x7f9e1014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.009+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 >> 172.21.15.71:0/4113398140 conn(0x7f9e100654f0 msgr2=0x7f9e10070630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:40.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.010+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 shutdown_connections 2023-08-25T09:50:40.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.010+0000 7f9e16955640 1 -- 172.21.15.71:0/4113398140 wait complete. 2023-08-25T09:50:40.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:50:40.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.249+0000 7f79d4f5b640 1 Processor -- start 2023-08-25T09:50:40.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.250+0000 7f79d4f5b640 1 -- start start 2023-08-25T09:50:40.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.251+0000 7f79d4f5b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f79d00cadd0 0x7f79d01575c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.251+0000 7f79d4f5b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f79d00c5e00 0x7f79d015a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.251+0000 7f79d4f5b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00c7480 con 0x7f79d00cb1b0 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.251+0000 7f79d4f5b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d00c55d0 con 0x7f79d00c5e00 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.251+0000 7f79d4f5b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d00c5b00 con 0x7f79d00cadd0 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f79d00c5e00 0x7f79d015a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f79d00c5e00 0x7f79d015a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52572/0 (socket says 172.21.15.71:52572) 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3938892300 learned_addr learned my addr 172.21.15.71:0/3938892300 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79ce59c640 1 --2- 172.21.15.71:0/3938892300 >> v2:172.21.15.164:3300/0 conn(0x7f79d00cadd0 0x7f79d01575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3938892300 >> v2:172.21.15.164:3300/0 conn(0x7f79d00cadd0 msgr2=0x7f79d01575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 --2- 172.21.15.71:0/3938892300 >> v2:172.21.15.164:3300/0 conn(0x7f79d00cadd0 0x7f79d01575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3938892300 >> v1:172.21.15.164:6789/0 conn(0x7f79d00cb1b0 legacy=0x7f79d0150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.252+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3938892300 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d00c5820 con 0x7f79d00c5e00 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.253+0000 7f79c5d9b640 1 --2- 172.21.15.71:0/3938892300 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00c5e00 0x7f79d015a130 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f79d00c1e30 tx=0x7f79b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8569acafa41598b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.254+0000 7f79cdd9b640 1 -- 172.21.15.71:0/3938892300 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b80317e0 con 0x7f79d00c5e00 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.254+0000 7f79cdd9b640 1 -- 172.21.15.71:0/3938892300 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b8031940 con 0x7f79d00c5e00 2023-08-25T09:50:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.254+0000 7f79cdd9b640 1 -- 172.21.15.71:0/3938892300 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8031cd0 con 0x7f79d00c5e00 2023-08-25T09:50:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.254+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3938892300 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00c5e00 msgr2=0x7f79d015a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.254+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3938892300 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00c5e00 0x7f79d015a130 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f79d00c1e30 tx=0x7f79b802b880 comp rx=0 tx=0).stop 2023-08-25T09:50:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.255+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3938892300 shutdown_connections 2023-08-25T09:50:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.255+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3938892300 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00c5e00 0x7f79d015a130 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.255+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3938892300 >> v2:172.21.15.164:3300/0 conn(0x7f79d00cadd0 0x7f79d01575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.255+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3938892300 >> 172.21.15.71:0/3938892300 conn(0x7f79d006c8b0 msgr2=0x7f79d006ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.255+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3938892300 shutdown_connections 2023-08-25T09:50:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.255+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3938892300 wait complete. 2023-08-25T09:50:40.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.256+0000 7f79d4f5b640 1 Processor -- start 2023-08-25T09:50:40.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.256+0000 7f79d4f5b640 1 -- start start 2023-08-25T09:50:40.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.257+0000 7f79d4f5b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5e00 0x7f79d00c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.257+0000 7f79d4f5b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 0x7f79d01f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.257+0000 7f79d4f5b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00cb1b0 0x7f79d01f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.257+0000 7f79d4f5b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d0148310 con 0x7f79d00cadd0 2023-08-25T09:50:40.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.257+0000 7f79d4f5b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d0154ca0 con 0x7f79d00cb1b0 2023-08-25T09:50:40.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.257+0000 7f79d4f5b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d00c1680 con 0x7f79d00c5e00 2023-08-25T09:50:40.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 0x7f79d01f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 0x7f79d01f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52588/0 (socket says 172.21.15.71:52588) 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3740612489 learned_addr learned my addr 172.21.15.71:0/3740612489 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79ce59c640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5e00 0x7f79d00c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79ced9d640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00cb1b0 0x7f79d01f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5e00 msgr2=0x7f79d00c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5e00 0x7f79d00c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00cb1b0 msgr2=0x7f79d01f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00cb1b0 0x7f79d01f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.258+0000 7f79c5d9b640 1 -- 172.21.15.71:0/3740612489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b80094d0 con 0x7f79d00cadd0 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.259+0000 7f79c5d9b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 0x7f79d01f1b40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f79b802b8c0 tx=0x7f79b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.259+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8032880 con 0x7f79d00cadd0 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.259+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b8032a00 con 0x7f79d00cadd0 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.259+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8032db0 con 0x7f79d00cadd0 2023-08-25T09:50:40.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.259+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d0148520 con 0x7f79d00cadd0 2023-08-25T09:50:40.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.259+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d0154930 con 0x7f79d00cadd0 2023-08-25T09:50:40.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.260+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d01f1920 con 0x7f79d00cadd0 2023-08-25T09:50:40.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.261+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f79b804e450 con 0x7f79d00cadd0 2023-08-25T09:50:40.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.262+0000 7f79c77fe640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a050 0x7f79b006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.262+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f79b80aab10 con 0x7f79d00cadd0 2023-08-25T09:50:40.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.262+0000 7f79ce59c640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a050 0x7f79b006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.263+0000 7f79ce59c640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a050 0x7f79b006c510 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009a60 tx=0x7f79c0003990 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:40.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.267+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f79b8075f70 con 0x7f79d00cadd0 2023-08-25T09:50:40.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.454+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f79d00c1b80 con 0x7f79d00cadd0 2023-08-25T09:50:40.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.454+0000 7f79c77fe640 1 -- 172.21.15.71:0/3740612489 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) v1 ==== 103+0+11 (secure 0 0 0) 0x7f79b8079590 con 0x7f79d00cadd0 2023-08-25T09:50:40.457 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:40.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a050 msgr2=0x7f79b006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a050 0x7f79b006c510 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009a60 tx=0x7f79c0003990 comp rx=0 tx=0).stop 2023-08-25T09:50:40.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 msgr2=0x7f79d01f1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 0x7f79d01f1b40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f79b802b8c0 tx=0x7f79b80046c0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 shutdown_connections 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79d00cb1b0 0x7f79d01f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f79b006a050 0x7f79b006c510 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79d00cadd0 0x7f79d01f1b40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 --2- 172.21.15.71:0/3740612489 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79d00c5e00 0x7f79d00c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 >> 172.21.15.71:0/3740612489 conn(0x7f79d006c8b0 msgr2=0x7f79d006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:40.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 shutdown_connections 2023-08-25T09:50:40.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.457+0000 7f79d4f5b640 1 -- 172.21.15.71:0/3740612489 wait complete. 2023-08-25T09:50:40.476 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-08-25T09:50:40.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:50:40.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.689+0000 7f09d359e640 1 Processor -- start 2023-08-25T09:50:40.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.689+0000 7f09d359e640 1 -- start start 2023-08-25T09:50:40.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f09d40c72f0 0x7f09d40acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 0x7f09d4155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d359e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40c1060 con 0x7f09d40c76d0 2023-08-25T09:50:40.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d40adf70 con 0x7f09d40c72f0 2023-08-25T09:50:40.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40af730 con 0x7f09d40ad2f0 2023-08-25T09:50:40.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f09d40c72f0 0x7f09d40acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f09d40c72f0 0x7f09d40acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52592/0 (socket says 172.21.15.71:52592) 2023-08-25T09:50:40.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.690+0000 7f09d259c640 1 -- 172.21.15.71:0/92079364 learned_addr learned my addr 172.21.15.71:0/92079364 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:40.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.691+0000 7f09d259c640 1 -- 172.21.15.71:0/92079364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 msgr2=0x7f09d4155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:40.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.691+0000 7f09d259c640 1 --2- 172.21.15.71:0/92079364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 0x7f09d4155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.691+0000 7f09d259c640 1 -- 172.21.15.71:0/92079364 >> v1:172.21.15.164:6789/0 conn(0x7f09d40c76d0 legacy=0x7f09d4152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:40.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.691+0000 7f09d259c640 1 -- 172.21.15.71:0/92079364 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d406b4a0 con 0x7f09d40c72f0 2023-08-25T09:50:40.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.692+0000 7f09d259c640 1 --2- 172.21.15.71:0/92079364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d40acdb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009e50 tx=0x7f09c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9be4c255231b87 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:40.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.692+0000 7f09d159a640 1 -- 172.21.15.71:0/92079364 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c402cb00 con 0x7f09d40c72f0 2023-08-25T09:50:40.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.692+0000 7f09d159a640 1 -- 172.21.15.71:0/92079364 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c402cc60 con 0x7f09d40c72f0 2023-08-25T09:50:40.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.692+0000 7f09d159a640 1 -- 172.21.15.71:0/92079364 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c40343f0 con 0x7f09d40c72f0 2023-08-25T09:50:40.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 -- 172.21.15.71:0/92079364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 msgr2=0x7f09d40acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 --2- 172.21.15.71:0/92079364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d40acdb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009e50 tx=0x7f09c402bf60 comp rx=0 tx=0).stop 2023-08-25T09:50:40.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 -- 172.21.15.71:0/92079364 shutdown_connections 2023-08-25T09:50:40.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 --2- 172.21.15.71:0/92079364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 0x7f09d4155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 --2- 172.21.15.71:0/92079364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d40acdb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 -- 172.21.15.71:0/92079364 >> 172.21.15.71:0/92079364 conn(0x7f09d406c280 msgr2=0x7f09d406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:40.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 -- 172.21.15.71:0/92079364 shutdown_connections 2023-08-25T09:50:40.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.693+0000 7f09d359e640 1 -- 172.21.15.71:0/92079364 wait complete. 2023-08-25T09:50:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.695+0000 7f09d359e640 1 Processor -- start 2023-08-25T09:50:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.695+0000 7f09d359e640 1 -- start start 2023-08-25T09:50:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 0x7f09d41eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d41ebea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d40c76d0 0x7f09d41e1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d41e1ba0 con 0x7f09d40c72f0 2023-08-25T09:50:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4069d30 con 0x7f09d40c76d0 2023-08-25T09:50:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d41e1da0 con 0x7f09d40ad2f0 2023-08-25T09:50:40.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d41ebea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d41ebea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52608/0 (socket says 172.21.15.71:52608) 2023-08-25T09:50:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 -- 172.21.15.71:0/3703780374 learned_addr learned my addr 172.21.15.71:0/3703780374 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 -- 172.21.15.71:0/3703780374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 msgr2=0x7f09d41eff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 0x7f09d41eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 -- 172.21.15.71:0/3703780374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d40c76d0 msgr2=0x7f09d41e1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d40c76d0 0x7f09d41e1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.696+0000 7f09d1d9b640 1 -- 172.21.15.71:0/3703780374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c4009b30 con 0x7f09d40c72f0 2023-08-25T09:50:40.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.697+0000 7f09d1d9b640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d41ebea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b3f0 tx=0x7f09bc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:40.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.697+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011830 con 0x7f09d40c72f0 2023-08-25T09:50:40.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.697+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc011990 con 0x7f09d40c72f0 2023-08-25T09:50:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.697+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011d40 con 0x7f09d40c72f0 2023-08-25T09:50:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.697+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d41e2080 con 0x7f09d40c72f0 2023-08-25T09:50:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.697+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d41ef790 con 0x7f09d40c72f0 2023-08-25T09:50:40.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.698+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d404e420 con 0x7f09d40c72f0 2023-08-25T09:50:40.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.699+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f09bc01a400 con 0x7f09d40c72f0 2023-08-25T09:50:40.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.700+0000 7f09caffd640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f09b406a050 0x7f09b406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:40.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.700+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f09bc088900 con 0x7f09d40c72f0 2023-08-25T09:50:40.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.700+0000 7f09d259c640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f09b406a050 0x7f09b406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:40.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.701+0000 7f09d259c640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f09b406a050 0x7f09b406c510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f09c4035000 tx=0x7f09c402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:40.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.705+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f09bc053b90 con 0x7f09d40c72f0 2023-08-25T09:50:40.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.938+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d406c9e0 con 0x7f09d40c72f0 2023-08-25T09:50:40.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.939+0000 7f09caffd640 1 -- 172.21.15.71:0/3703780374 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f09bc015090 con 0x7f09d40c72f0 2023-08-25T09:50:40.941 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:40.941 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:50:23.760465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:50:40.941 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 17 2023-08-25T09:50:40.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f09b406a050 msgr2=0x7f09b406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f09b406a050 0x7f09b406c510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f09c4035000 tx=0x7f09c402e000 comp rx=0 tx=0).stop 2023-08-25T09:50:40.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 msgr2=0x7f09d41ebea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:40.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d41ebea0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b3f0 tx=0x7f09bc00b8b0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 shutdown_connections 2023-08-25T09:50:40.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f09d40c76d0 0x7f09d41e1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f09b406a050 0x7f09b406c510 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f09d40c72f0 0x7f09d41ebea0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 --2- 172.21.15.71:0/3703780374 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f09d40ad2f0 0x7f09d41eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:40.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 >> 172.21.15.71:0/3703780374 conn(0x7f09d406c280 msgr2=0x7f09d40c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:40.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 shutdown_connections 2023-08-25T09:50:40.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:40.942+0000 7f09d359e640 1 -- 172.21.15.71:0/3703780374 wait complete. 2023-08-25T09:50:40.962 INFO:tasks.mds_thrash:Ready to start thrashing 2023-08-25T09:50:40.963 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-08-25T09:50:40.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:41.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.173+0000 7fbda4bcf640 1 Processor -- start 2023-08-25T09:50:41.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.173+0000 7fbda4bcf640 1 -- start start 2023-08-25T09:50:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.174+0000 7fbda4bcf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbda00c3a40 0x7fbda00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.174+0000 7fbda4bcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda01509d0 0x7fbda0150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.174+0000 7fbda4bcf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda00c1dc0 con 0x7fbda00c3660 2023-08-25T09:50:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.174+0000 7fbda4bcf640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda0157a80 con 0x7fbda00c3a40 2023-08-25T09:50:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.174+0000 7fbda4bcf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda0151360 con 0x7fbda01509d0 2023-08-25T09:50:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbda00c3a40 0x7fbda00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbda00c3a40 0x7fbda00c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52622/0 (socket says 172.21.15.71:52622) 2023-08-25T09:50:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 -- 172.21.15.71:0/1796895092 learned_addr learned my addr 172.21.15.71:0/1796895092 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd9ed9d640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda01509d0 0x7fbda0150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 -- 172.21.15.71:0/1796895092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda01509d0 msgr2=0x7fbda0150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda01509d0 0x7fbda0150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 -- 172.21.15.71:0/1796895092 >> v1:172.21.15.164:6789/0 conn(0x7fbda00c3660 legacy=0x7fbda006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd95d9b640 1 -- 172.21.15.71:0/1796895092 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0156e00 con 0x7fbda00c3a40 2023-08-25T09:50:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.175+0000 7fbd9ed9d640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda01509d0 0x7fbda0150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.176+0000 7fbd95d9b640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3a40 0x7fbda00c1710 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009e50 tx=0x7fbd8802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c994cdc16b22e4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.177+0000 7fbd9dd9b640 1 -- 172.21.15.71:0/1796895092 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8802caf0 con 0x7fbda00c3a40 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.177+0000 7fbd9dd9b640 1 -- 172.21.15.71:0/1796895092 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8802cc50 con 0x7fbda00c3a40 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.177+0000 7fbd9dd9b640 1 -- 172.21.15.71:0/1796895092 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880343f0 con 0x7fbda00c3a40 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.177+0000 7fbda4bcf640 1 -- 172.21.15.71:0/1796895092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3a40 msgr2=0x7fbda00c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.177+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3a40 0x7fbda00c1710 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009e50 tx=0x7fbd8802bf50 comp rx=0 tx=0).stop 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.177+0000 7fbda4bcf640 1 -- 172.21.15.71:0/1796895092 shutdown_connections 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.178+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda01509d0 0x7fbda0150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.178+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/1796895092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3a40 0x7fbda00c1710 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.178+0000 7fbda4bcf640 1 -- 172.21.15.71:0/1796895092 >> 172.21.15.71:0/1796895092 conn(0x7fbda00654f0 msgr2=0x7fbda0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:41.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.178+0000 7fbda4bcf640 1 -- 172.21.15.71:0/1796895092 shutdown_connections 2023-08-25T09:50:41.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.178+0000 7fbda4bcf640 1 -- 172.21.15.71:0/1796895092 wait complete. 2023-08-25T09:50:41.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.179+0000 7fbda4bcf640 1 Processor -- start 2023-08-25T09:50:41.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.179+0000 7fbda4bcf640 1 -- start start 2023-08-25T09:50:41.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.180+0000 7fbda4bcf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 0x7fbda006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.180+0000 7fbda4bcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda00c3a40 0x7fbda0151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.180+0000 7fbda4bcf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbda01509d0 0x7fbda01f1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.180+0000 7fbda4bcf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda014ccb0 con 0x7fbda00c3660 2023-08-25T09:50:41.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.180+0000 7fbda4bcf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda0154470 con 0x7fbda01509d0 2023-08-25T09:50:41.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.180+0000 7fbda4bcf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda0153460 con 0x7fbda00c3a40 2023-08-25T09:50:41.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 0x7fbda006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 0x7fbda006a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52626/0 (socket says 172.21.15.71:52626) 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 -- 172.21.15.71:0/3451551770 learned_addr learned my addr 172.21.15.71:0/3451551770 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd95d9b640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda00c3a40 0x7fbda0151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9ed9d640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbda01509d0 0x7fbda01f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 -- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda00c3a40 msgr2=0x7fbda0151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda00c3a40 0x7fbda0151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 -- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbda01509d0 msgr2=0x7fbda01f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbda01509d0 0x7fbda01f1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 -- 172.21.15.71:0/3451551770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88009b30 con 0x7fbda00c3660 2023-08-25T09:50:41.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.181+0000 7fbd9e59c640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 0x7fbda006a8b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000b420 tx=0x7fbd9000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.182+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90011830 con 0x7fbda00c3660 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.182+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd90011990 con 0x7fbda00c3660 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.182+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90011d40 con 0x7fbda00c3660 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.182+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda014ce90 con 0x7fbda00c3660 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.182+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda0153990 con 0x7fbda00c3660 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.183+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda00c29c0 con 0x7fbda00c3660 2023-08-25T09:50:41.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.184+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbd9001b460 con 0x7fbda00c3660 2023-08-25T09:50:41.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.185+0000 7fbd977fe640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbd8006a050 0x7fbd8006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.185+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fbd90088540 con 0x7fbda00c3660 2023-08-25T09:50:41.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.185+0000 7fbd95d9b640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbd8006a050 0x7fbd8006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.186+0000 7fbd95d9b640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbd8006a050 0x7fbd8006c510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbd88035000 tx=0x7fbd88030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:41.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.189+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbd90053860 con 0x7fbda00c3660 2023-08-25T09:50:41.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.376+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbda0067a80 con 0x7fbd8006a050 2023-08-25T09:50:41.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.408+0000 7fbd977fe640 1 -- 172.21.15.71:0/3451551770 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+389523 (secure 0 0 0) 0x7fbda0067a80 con 0x7fbd8006a050 2023-08-25T09:50:41.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:41.417 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:41.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbd8006a050 msgr2=0x7fbd8006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbd8006a050 0x7fbd8006c510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbd88035000 tx=0x7fbd88030000 comp rx=0 tx=0).stop 2023-08-25T09:50:41.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 msgr2=0x7fbda006a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 0x7fbda006a8b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000b420 tx=0x7fbd9000b8e0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 shutdown_connections 2023-08-25T09:50:41.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbda01509d0 0x7fbda01f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbd8006a050 0x7fbd8006c510 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbda00c3a40 0x7fbda0151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 --2- 172.21.15.71:0/3451551770 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbda00c3660 0x7fbda006a8b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.414+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 >> 172.21.15.71:0/3451551770 conn(0x7fbda00654f0 msgr2=0x7fbda0069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:41.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.415+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 shutdown_connections 2023-08-25T09:50:41.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.415+0000 7fbda4bcf640 1 -- 172.21.15.71:0/3451551770 wait complete. 2023-08-25T09:50:41.434 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-08-25T09:50:40.660103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602391,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":376,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227424,"kb_used_data":2976,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747296,"statfs":{"total":773094113280,"available":772861231104,"internally_reserved":0,"allocated":3047424,"data_stored":2328822,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1760,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004736"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629137+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:50:12.629137+0000","last_peered":"2023-08-25T09:50:12.629137+0000","last_clean":"2023-08-25T09:50:12.629137+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:50:12.629137+0000","last_undegraded":"2023-08-25T09:50:12.629137+0000","last_fullsized":"2023-08-25T09:50:12.629137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:23.094991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"31'13","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.758090+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:50:22.758090+0000","last_peered":"2023-08-25T09:50:22.758090+0000","last_clean":"2023-08-25T09:50:22.758090+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:50:22.758090+0000","last_undegraded":"2023-08-25T09:50:22.758090+0000","last_fullsized":"2023-08-25T09:50:22.758090+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:58:03.201648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554648+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:50:12.554648+0000","last_peered":"2023-08-25T09:50:12.554648+0000","last_clean":"2023-08-25T09:50:12.554648+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:50:12.554648+0000","last_undegraded":"2023-08-25T09:50:12.554648+0000","last_fullsized":"2023-08-25T09:50:12.554648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:26.126307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554093+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:50:12.554093+0000","last_peered":"2023-08-25T09:50:12.554093+0000","last_clean":"2023-08-25T09:50:12.554093+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:50:12.554093+0000","last_undegraded":"2023-08-25T09:50:12.554093+0000","last_fullsized":"2023-08-25T09:50:12.554093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:22.526110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"29'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627113+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:50:12.627113+0000","last_peered":"2023-08-25T09:50:12.627113+0000","last_clean":"2023-08-25T09:50:12.627113+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:50:12.627113+0000","last_undegraded":"2023-08-25T09:50:12.627113+0000","last_fullsized":"2023-08-25T09:50:12.627113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:44:00.313382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626363+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:50:12.626363+0000","last_peered":"2023-08-25T09:50:12.626363+0000","last_clean":"2023-08-25T09:50:12.626363+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:50:12.626363+0000","last_undegraded":"2023-08-25T09:50:12.626363+0000","last_fullsized":"2023-08-25T09:50:12.626363+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:06:14.787284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556659+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:50:12.556659+0000","last_peered":"2023-08-25T09:50:12.556659+0000","last_clean":"2023-08-25T09:50:12.556659+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:50:12.556659+0000","last_undegraded":"2023-08-25T09:50:12.556659+0000","last_fullsized":"2023-08-25T09:50:12.556659+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:45:30.828332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761816+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:50:22.761816+0000","last_peered":"2023-08-25T09:50:22.761816+0000","last_clean":"2023-08-25T09:50:22.761816+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:50:22.761816+0000","last_undegraded":"2023-08-25T09:50:22.761816+0000","last_fullsized":"2023-08-25T09:50:22.761816+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:15:33.280852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558309+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:50:12.558309+0000","last_peered":"2023-08-25T09:50:12.558309+0000","last_clean":"2023-08-25T09:50:12.558309+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:50:12.558309+0000","last_undegraded":"2023-08-25T09:50:12.558309+0000","last_fullsized":"2023-08-25T09:50:12.558309+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:45.240925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626253+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:50:12.626253+0000","last_peered":"2023-08-25T09:50:12.626253+0000","last_clean":"2023-08-25T09:50:12.626253+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:50:12.626253+0000","last_undegraded":"2023-08-25T09:50:12.626253+0000","last_fullsized":"2023-08-25T09:50:12.626253+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:16:41.085653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761058+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:50:22.761058+0000","last_peered":"2023-08-25T09:50:22.761058+0000","last_clean":"2023-08-25T09:50:22.761058+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:50:22.761058+0000","last_undegraded":"2023-08-25T09:50:22.761058+0000","last_fullsized":"2023-08-25T09:50:22.761058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:10:43.593031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556912+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:50:12.556912+0000","last_peered":"2023-08-25T09:50:12.556912+0000","last_clean":"2023-08-25T09:50:12.556912+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:50:12.556912+0000","last_undegraded":"2023-08-25T09:50:12.556912+0000","last_fullsized":"2023-08-25T09:50:12.556912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:41:19.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628183+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:50:12.628183+0000","last_peered":"2023-08-25T09:50:12.628183+0000","last_clean":"2023-08-25T09:50:12.628183+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:50:12.628183+0000","last_undegraded":"2023-08-25T09:50:12.628183+0000","last_fullsized":"2023-08-25T09:50:12.628183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:55:22.425822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552143+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:50:12.552143+0000","last_peered":"2023-08-25T09:50:12.552143+0000","last_clean":"2023-08-25T09:50:12.552143+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:50:12.552143+0000","last_undegraded":"2023-08-25T09:50:12.552143+0000","last_fullsized":"2023-08-25T09:50:12.552143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:56.784267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556859+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:50:12.556859+0000","last_peered":"2023-08-25T09:50:12.556859+0000","last_clean":"2023-08-25T09:50:12.556859+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:50:12.556859+0000","last_undegraded":"2023-08-25T09:50:12.556859+0000","last_fullsized":"2023-08-25T09:50:12.556859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:29:29.962318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557828+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:50:12.557828+0000","last_peered":"2023-08-25T09:50:12.557828+0000","last_clean":"2023-08-25T09:50:12.557828+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:50:12.557828+0000","last_undegraded":"2023-08-25T09:50:12.557828+0000","last_fullsized":"2023-08-25T09:50:12.557828+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:46.007193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.575799+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:50:17.575799+0000","last_peered":"2023-08-25T09:50:17.575799+0000","last_clean":"2023-08-25T09:50:17.575799+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:50:17.575799+0000","last_undegraded":"2023-08-25T09:50:17.575799+0000","last_fullsized":"2023-08-25T09:50:17.575799+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:23:50.467500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628494+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:50:12.628494+0000","last_peered":"2023-08-25T09:50:12.628494+0000","last_clean":"2023-08-25T09:50:12.628494+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:50:12.628494+0000","last_undegraded":"2023-08-25T09:50:12.628494+0000","last_fullsized":"2023-08-25T09:50:12.628494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:28:21.130424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566445+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:50:12.566445+0000","last_peered":"2023-08-25T09:50:12.566445+0000","last_clean":"2023-08-25T09:50:12.566445+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:50:12.566445+0000","last_undegraded":"2023-08-25T09:50:12.566445+0000","last_fullsized":"2023-08-25T09:50:12.566445+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:24.203781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630137+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:50:12.630137+0000","last_peered":"2023-08-25T09:50:12.630137+0000","last_clean":"2023-08-25T09:50:12.630137+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:50:12.630137+0000","last_undegraded":"2023-08-25T09:50:12.630137+0000","last_fullsized":"2023-08-25T09:50:12.630137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:11:08.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551378+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:50:12.551378+0000","last_peered":"2023-08-25T09:50:12.551378+0000","last_clean":"2023-08-25T09:50:12.551378+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:50:12.551378+0000","last_undegraded":"2023-08-25T09:50:12.551378+0000","last_fullsized":"2023-08-25T09:50:12.551378+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:12:56.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552046+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:50:12.552046+0000","last_peered":"2023-08-25T09:50:12.552046+0000","last_clean":"2023-08-25T09:50:12.552046+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:50:12.552046+0000","last_undegraded":"2023-08-25T09:50:12.552046+0000","last_fullsized":"2023-08-25T09:50:12.552046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:30:58.600767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573085+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:50:17.573085+0000","last_peered":"2023-08-25T09:50:17.573085+0000","last_clean":"2023-08-25T09:50:17.573085+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:50:17.573085+0000","last_undegraded":"2023-08-25T09:50:17.573085+0000","last_fullsized":"2023-08-25T09:50:17.573085+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:06.900671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566581+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:50:12.566581+0000","last_peered":"2023-08-25T09:50:12.566581+0000","last_clean":"2023-08-25T09:50:12.566581+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:50:12.566581+0000","last_undegraded":"2023-08-25T09:50:12.566581+0000","last_fullsized":"2023-08-25T09:50:12.566581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:51:56.697715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625495+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:50:12.625495+0000","last_peered":"2023-08-25T09:50:12.625495+0000","last_clean":"2023-08-25T09:50:12.625495+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:50:12.625495+0000","last_undegraded":"2023-08-25T09:50:12.625495+0000","last_fullsized":"2023-08-25T09:50:12.625495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:23:54.271157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:33.134863+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:50:33.134863+0000","last_peered":"2023-08-25T09:50:33.134863+0000","last_clean":"2023-08-25T09:50:33.134863+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:50:33.134863+0000","last_undegraded":"2023-08-25T09:50:33.134863+0000","last_fullsized":"2023-08-25T09:50:33.134863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:19:35.376472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565155+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:50:12.565155+0000","last_peered":"2023-08-25T09:50:12.565155+0000","last_clean":"2023-08-25T09:50:12.565155+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:50:12.565155+0000","last_undegraded":"2023-08-25T09:50:12.565155+0000","last_fullsized":"2023-08-25T09:50:12.565155+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:16:54.785190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626109+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:50:12.626109+0000","last_peered":"2023-08-25T09:50:12.626109+0000","last_clean":"2023-08-25T09:50:12.626109+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:50:12.626109+0000","last_undegraded":"2023-08-25T09:50:12.626109+0000","last_fullsized":"2023-08-25T09:50:12.626109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:00:08.343712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.759325+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:50:22.759325+0000","last_peered":"2023-08-25T09:50:22.759325+0000","last_clean":"2023-08-25T09:50:22.759325+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:50:22.759325+0000","last_undegraded":"2023-08-25T09:50:22.759325+0000","last_fullsized":"2023-08-25T09:50:22.759325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:38:03.473951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625586+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:50:12.625586+0000","last_peered":"2023-08-25T09:50:12.625586+0000","last_clean":"2023-08-25T09:50:12.625586+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:50:12.625586+0000","last_undegraded":"2023-08-25T09:50:12.625586+0000","last_fullsized":"2023-08-25T09:50:12.625586+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:03:07.147111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626969+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:50:12.626969+0000","last_peered":"2023-08-25T09:50:12.626969+0000","last_clean":"2023-08-25T09:50:12.626969+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:50:12.626969+0000","last_undegraded":"2023-08-25T09:50:12.626969+0000","last_fullsized":"2023-08-25T09:50:12.626969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:14:08.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630451+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:50:12.630451+0000","last_peered":"2023-08-25T09:50:12.630451+0000","last_clean":"2023-08-25T09:50:12.630451+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:50:12.630451+0000","last_undegraded":"2023-08-25T09:50:12.630451+0000","last_fullsized":"2023-08-25T09:50:12.630451+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:33.100204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626271+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:50:12.626271+0000","last_peered":"2023-08-25T09:50:12.626271+0000","last_clean":"2023-08-25T09:50:12.626271+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:50:12.626271+0000","last_undegraded":"2023-08-25T09:50:12.626271+0000","last_fullsized":"2023-08-25T09:50:12.626271+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:31:59.929747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"31'16","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:33.155598+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:50:33.155598+0000","last_peered":"2023-08-25T09:50:33.155598+0000","last_clean":"2023-08-25T09:50:33.155598+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:50:33.155598+0000","last_undegraded":"2023-08-25T09:50:33.155598+0000","last_fullsized":"2023-08-25T09:50:33.155598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:59:56.712100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4281,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557216+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:50:12.557216+0000","last_peered":"2023-08-25T09:50:12.557216+0000","last_clean":"2023-08-25T09:50:12.557216+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:50:12.557216+0000","last_undegraded":"2023-08-25T09:50:12.557216+0000","last_fullsized":"2023-08-25T09:50:12.557216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:24:19.739443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629763+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:50:12.629763+0000","last_peered":"2023-08-25T09:50:12.629763+0000","last_clean":"2023-08-25T09:50:12.629763+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:50:12.629763+0000","last_undegraded":"2023-08-25T09:50:12.629763+0000","last_fullsized":"2023-08-25T09:50:12.629763+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:08:23.975702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555540+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:50:12.555540+0000","last_peered":"2023-08-25T09:50:12.555540+0000","last_clean":"2023-08-25T09:50:12.555540+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:50:12.555540+0000","last_undegraded":"2023-08-25T09:50:12.555540+0000","last_fullsized":"2023-08-25T09:50:12.555540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:23:24.270948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566757+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:50:12.566757+0000","last_peered":"2023-08-25T09:50:12.566757+0000","last_clean":"2023-08-25T09:50:12.566757+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:50:12.566757+0000","last_undegraded":"2023-08-25T09:50:12.566757+0000","last_fullsized":"2023-08-25T09:50:12.566757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:10.046784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.576095+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:50:17.576095+0000","last_peered":"2023-08-25T09:50:17.576095+0000","last_clean":"2023-08-25T09:50:17.576095+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:50:17.576095+0000","last_undegraded":"2023-08-25T09:50:17.576095+0000","last_fullsized":"2023-08-25T09:50:17.576095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:12:06.214562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625440+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:50:12.625440+0000","last_peered":"2023-08-25T09:50:12.625440+0000","last_clean":"2023-08-25T09:50:12.625440+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:50:12.625440+0000","last_undegraded":"2023-08-25T09:50:12.625440+0000","last_fullsized":"2023-08-25T09:50:12.625440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:40:36.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625747+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:50:12.625747+0000","last_peered":"2023-08-25T09:50:12.625747+0000","last_clean":"2023-08-25T09:50:12.625747+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:50:12.625747+0000","last_undegraded":"2023-08-25T09:50:12.625747+0000","last_fullsized":"2023-08-25T09:50:12.625747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:39:48.184915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630298+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:50:12.630298+0000","last_peered":"2023-08-25T09:50:12.630298+0000","last_clean":"2023-08-25T09:50:12.630298+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:50:12.630298+0000","last_undegraded":"2023-08-25T09:50:12.630298+0000","last_fullsized":"2023-08-25T09:50:12.630298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:45:35.013224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.759471+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:50:22.759471+0000","last_peered":"2023-08-25T09:50:22.759471+0000","last_clean":"2023-08-25T09:50:22.759471+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:50:22.759471+0000","last_undegraded":"2023-08-25T09:50:22.759471+0000","last_fullsized":"2023-08-25T09:50:22.759471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:55:21.711434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554400+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:50:12.554400+0000","last_peered":"2023-08-25T09:50:12.554400+0000","last_clean":"2023-08-25T09:50:12.554400+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:50:12.554400+0000","last_undegraded":"2023-08-25T09:50:12.554400+0000","last_fullsized":"2023-08-25T09:50:12.554400+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:23:36.823930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551745+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:50:12.551745+0000","last_peered":"2023-08-25T09:50:12.551745+0000","last_clean":"2023-08-25T09:50:12.551745+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:50:12.551745+0000","last_undegraded":"2023-08-25T09:50:12.551745+0000","last_fullsized":"2023-08-25T09:50:12.551745+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:31:55.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627612+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:50:12.627612+0000","last_peered":"2023-08-25T09:50:12.627612+0000","last_clean":"2023-08-25T09:50:12.627612+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:50:12.627612+0000","last_undegraded":"2023-08-25T09:50:12.627612+0000","last_fullsized":"2023-08-25T09:50:12.627612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:44:34.927600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"31'1","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628052+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:50:12.628052+0000","last_peered":"2023-08-25T09:50:12.628052+0000","last_clean":"2023-08-25T09:50:12.628052+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:50:12.628052+0000","last_undegraded":"2023-08-25T09:50:12.628052+0000","last_fullsized":"2023-08-25T09:50:12.628052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:05:32.332141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565443+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:50:12.565443+0000","last_peered":"2023-08-25T09:50:12.565443+0000","last_clean":"2023-08-25T09:50:12.565443+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:50:12.565443+0000","last_undegraded":"2023-08-25T09:50:12.565443+0000","last_fullsized":"2023-08-25T09:50:12.565443+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:15:47.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630669+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:50:12.630669+0000","last_peered":"2023-08-25T09:50:12.630669+0000","last_clean":"2023-08-25T09:50:12.630669+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:50:12.630669+0000","last_undegraded":"2023-08-25T09:50:12.630669+0000","last_fullsized":"2023-08-25T09:50:12.630669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:18:08.404806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628723+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:50:12.628723+0000","last_peered":"2023-08-25T09:50:12.628723+0000","last_clean":"2023-08-25T09:50:12.628723+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:50:12.628723+0000","last_undegraded":"2023-08-25T09:50:12.628723+0000","last_fullsized":"2023-08-25T09:50:12.628723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:17:27.827364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.580091+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:50:17.580091+0000","last_peered":"2023-08-25T09:50:17.580091+0000","last_clean":"2023-08-25T09:50:17.580091+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:50:17.580091+0000","last_undegraded":"2023-08-25T09:50:17.580091+0000","last_fullsized":"2023-08-25T09:50:17.580091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:56:04.945533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551491+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:50:12.551491+0000","last_peered":"2023-08-25T09:50:12.551491+0000","last_clean":"2023-08-25T09:50:12.551491+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:50:12.551491+0000","last_undegraded":"2023-08-25T09:50:12.551491+0000","last_fullsized":"2023-08-25T09:50:12.551491+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:52.676480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627952+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:50:12.627952+0000","last_peered":"2023-08-25T09:50:12.627952+0000","last_clean":"2023-08-25T09:50:12.627952+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:50:12.627952+0000","last_undegraded":"2023-08-25T09:50:12.627952+0000","last_fullsized":"2023-08-25T09:50:12.627952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:42:22.416093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573338+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:50:17.573338+0000","last_peered":"2023-08-25T09:50:17.573338+0000","last_clean":"2023-08-25T09:50:17.573338+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:50:17.573338+0000","last_undegraded":"2023-08-25T09:50:17.573338+0000","last_fullsized":"2023-08-25T09:50:17.573338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:49:49.710740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556160+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:50:12.556160+0000","last_peered":"2023-08-25T09:50:12.556160+0000","last_clean":"2023-08-25T09:50:12.556160+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:50:12.556160+0000","last_undegraded":"2023-08-25T09:50:12.556160+0000","last_fullsized":"2023-08-25T09:50:12.556160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:34.254249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551025+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:50:12.551025+0000","last_peered":"2023-08-25T09:50:12.551025+0000","last_clean":"2023-08-25T09:50:12.551025+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:50:12.551025+0000","last_undegraded":"2023-08-25T09:50:12.551025+0000","last_fullsized":"2023-08-25T09:50:12.551025+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:38:55.415402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629657+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:50:12.629657+0000","last_peered":"2023-08-25T09:50:12.629657+0000","last_clean":"2023-08-25T09:50:12.629657+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:50:12.629657+0000","last_undegraded":"2023-08-25T09:50:12.629657+0000","last_fullsized":"2023-08-25T09:50:12.629657+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:50:25.245004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627351+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:50:12.627351+0000","last_peered":"2023-08-25T09:50:12.627351+0000","last_clean":"2023-08-25T09:50:12.627351+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:50:12.627351+0000","last_undegraded":"2023-08-25T09:50:12.627351+0000","last_fullsized":"2023-08-25T09:50:12.627351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:41:28.016611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557166+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:50:12.557166+0000","last_peered":"2023-08-25T09:50:12.557166+0000","last_clean":"2023-08-25T09:50:12.557166+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:50:12.557166+0000","last_undegraded":"2023-08-25T09:50:12.557166+0000","last_fullsized":"2023-08-25T09:50:12.557166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:09:13.822549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557109+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:50:12.557109+0000","last_peered":"2023-08-25T09:50:12.557109+0000","last_clean":"2023-08-25T09:50:12.557109+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:50:12.557109+0000","last_undegraded":"2023-08-25T09:50:12.557109+0000","last_fullsized":"2023-08-25T09:50:12.557109+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:15:47.257428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551738+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:50:12.551738+0000","last_peered":"2023-08-25T09:50:12.551738+0000","last_clean":"2023-08-25T09:50:12.551738+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:50:12.551738+0000","last_undegraded":"2023-08-25T09:50:12.551738+0000","last_fullsized":"2023-08-25T09:50:12.551738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:22:38.020973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761598+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:50:22.761598+0000","last_peered":"2023-08-25T09:50:22.761598+0000","last_clean":"2023-08-25T09:50:22.761598+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:50:22.761598+0000","last_undegraded":"2023-08-25T09:50:22.761598+0000","last_fullsized":"2023-08-25T09:50:22.761598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:02:36.889545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626347+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:50:12.626347+0000","last_peered":"2023-08-25T09:50:12.626347+0000","last_clean":"2023-08-25T09:50:12.626347+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:50:12.626347+0000","last_undegraded":"2023-08-25T09:50:12.626347+0000","last_fullsized":"2023-08-25T09:50:12.626347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:52:59.975913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558258+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:50:12.558258+0000","last_peered":"2023-08-25T09:50:12.558258+0000","last_clean":"2023-08-25T09:50:12.558258+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:50:12.558258+0000","last_undegraded":"2023-08-25T09:50:12.558258+0000","last_fullsized":"2023-08-25T09:50:12.558258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:11.134847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566106+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:50:12.566106+0000","last_peered":"2023-08-25T09:50:12.566106+0000","last_clean":"2023-08-25T09:50:12.566106+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:50:12.566106+0000","last_undegraded":"2023-08-25T09:50:12.566106+0000","last_fullsized":"2023-08-25T09:50:12.566106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:29:31.536127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761717+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:50:22.761717+0000","last_peered":"2023-08-25T09:50:22.761717+0000","last_clean":"2023-08-25T09:50:22.761717+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:50:22.761717+0000","last_undegraded":"2023-08-25T09:50:22.761717+0000","last_fullsized":"2023-08-25T09:50:22.761717+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:07:02.583931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556866+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:50:12.556866+0000","last_peered":"2023-08-25T09:50:12.556866+0000","last_clean":"2023-08-25T09:50:12.556866+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:50:12.556866+0000","last_undegraded":"2023-08-25T09:50:12.556866+0000","last_fullsized":"2023-08-25T09:50:12.556866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:43:35.431443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":187,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626710+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:50:12.626710+0000","last_peered":"2023-08-25T09:50:12.626710+0000","last_clean":"2023-08-25T09:50:12.626710+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:50:12.626710+0000","last_undegraded":"2023-08-25T09:50:12.626710+0000","last_fullsized":"2023-08-25T09:50:12.626710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:37:25.347377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626465+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:50:12.626465+0000","last_peered":"2023-08-25T09:50:12.626465+0000","last_clean":"2023-08-25T09:50:12.626465+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:50:12.626465+0000","last_undegraded":"2023-08-25T09:50:12.626465+0000","last_fullsized":"2023-08-25T09:50:12.626465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:49:37.046349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629357+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:50:12.629357+0000","last_peered":"2023-08-25T09:50:12.629357+0000","last_clean":"2023-08-25T09:50:12.629357+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:50:12.629357+0000","last_undegraded":"2023-08-25T09:50:12.629357+0000","last_fullsized":"2023-08-25T09:50:12.629357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:29:01.327572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"29'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630079+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:50:12.630079+0000","last_peered":"2023-08-25T09:50:12.630079+0000","last_clean":"2023-08-25T09:50:12.630079+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:50:12.630079+0000","last_undegraded":"2023-08-25T09:50:12.630079+0000","last_fullsized":"2023-08-25T09:50:12.630079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:07:37.714359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.559002+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:50:12.559002+0000","last_peered":"2023-08-25T09:50:12.559002+0000","last_clean":"2023-08-25T09:50:12.559002+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:50:12.559002+0000","last_undegraded":"2023-08-25T09:50:12.559002+0000","last_fullsized":"2023-08-25T09:50:12.559002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:23:29.317977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.559077+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:50:12.559077+0000","last_peered":"2023-08-25T09:50:12.559077+0000","last_clean":"2023-08-25T09:50:12.559077+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:50:12.559077+0000","last_undegraded":"2023-08-25T09:50:12.559077+0000","last_fullsized":"2023-08-25T09:50:12.559077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:09:36.207083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.755917+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:50:22.755917+0000","last_peered":"2023-08-25T09:50:22.755917+0000","last_clean":"2023-08-25T09:50:22.755917+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:50:22.755917+0000","last_undegraded":"2023-08-25T09:50:22.755917+0000","last_fullsized":"2023-08-25T09:50:22.755917+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:35.082983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556600+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:50:12.556600+0000","last_peered":"2023-08-25T09:50:12.556600+0000","last_clean":"2023-08-25T09:50:12.556600+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:50:12.556600+0000","last_undegraded":"2023-08-25T09:50:12.556600+0000","last_fullsized":"2023-08-25T09:50:12.556600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:57:07.092488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558495+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:50:12.558495+0000","last_peered":"2023-08-25T09:50:12.558495+0000","last_clean":"2023-08-25T09:50:12.558495+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:50:12.558495+0000","last_undegraded":"2023-08-25T09:50:12.558495+0000","last_fullsized":"2023-08-25T09:50:12.558495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:49:01.069322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573225+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:50:17.573225+0000","last_peered":"2023-08-25T09:50:17.573225+0000","last_clean":"2023-08-25T09:50:17.573225+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:50:17.573225+0000","last_undegraded":"2023-08-25T09:50:17.573225+0000","last_fullsized":"2023-08-25T09:50:17.573225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:04:42.286097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627074+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:50:12.627074+0000","last_peered":"2023-08-25T09:50:12.627074+0000","last_clean":"2023-08-25T09:50:12.627074+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:50:12.627074+0000","last_undegraded":"2023-08-25T09:50:12.627074+0000","last_fullsized":"2023-08-25T09:50:12.627074+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:10:04.637461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558961+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:50:12.558961+0000","last_peered":"2023-08-25T09:50:12.558961+0000","last_clean":"2023-08-25T09:50:12.558961+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:50:12.558961+0000","last_undegraded":"2023-08-25T09:50:12.558961+0000","last_fullsized":"2023-08-25T09:50:12.558961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:21.839926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"29'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627445+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:50:12.627445+0000","last_peered":"2023-08-25T09:50:12.627445+0000","last_clean":"2023-08-25T09:50:12.627445+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:50:12.627445+0000","last_undegraded":"2023-08-25T09:50:12.627445+0000","last_fullsized":"2023-08-25T09:50:12.627445+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:34:35.939804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557002+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:50:12.557002+0000","last_peered":"2023-08-25T09:50:12.557002+0000","last_clean":"2023-08-25T09:50:12.557002+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:50:12.557002+0000","last_undegraded":"2023-08-25T09:50:12.557002+0000","last_fullsized":"2023-08-25T09:50:12.557002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:40:23.888752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551613+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:50:12.551613+0000","last_peered":"2023-08-25T09:50:12.551613+0000","last_clean":"2023-08-25T09:50:12.551613+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:50:12.551613+0000","last_undegraded":"2023-08-25T09:50:12.551613+0000","last_fullsized":"2023-08-25T09:50:12.551613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:34:14.609357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.582719+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:50:17.582719+0000","last_peered":"2023-08-25T09:50:17.582719+0000","last_clean":"2023-08-25T09:50:17.582719+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:50:17.582719+0000","last_undegraded":"2023-08-25T09:50:17.582719+0000","last_fullsized":"2023-08-25T09:50:17.582719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:18:04.169294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556982+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:50:12.556982+0000","last_peered":"2023-08-25T09:50:12.556982+0000","last_clean":"2023-08-25T09:50:12.556982+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:50:12.556982+0000","last_undegraded":"2023-08-25T09:50:12.556982+0000","last_fullsized":"2023-08-25T09:50:12.556982+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:23:07.786471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.571725+0000","last_change":"2023-08-25T09:50:12.571725+0000","last_active":"2023-08-25T09:50:12.571725+0000","last_peered":"2023-08-25T09:50:12.571725+0000","last_clean":"2023-08-25T09:50:12.571725+0000","last_became_active":"2023-08-25T09:50:12.570960+0000","last_became_peered":"2023-08-25T09:50:12.570960+0000","last_unstale":"2023-08-25T09:50:12.571725+0000","last_undegraded":"2023-08-25T09:50:12.571725+0000","last_fullsized":"2023-08-25T09:50:12.571725+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:36.819051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.570490+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:50:17.570490+0000","last_peered":"2023-08-25T09:50:17.570490+0000","last_clean":"2023-08-25T09:50:17.570490+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:50:17.570490+0000","last_undegraded":"2023-08-25T09:50:17.570490+0000","last_fullsized":"2023-08-25T09:50:17.570490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:21:54.167747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556745+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:50:12.556745+0000","last_peered":"2023-08-25T09:50:12.556745+0000","last_clean":"2023-08-25T09:50:12.556745+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:50:12.556745+0000","last_undegraded":"2023-08-25T09:50:12.556745+0000","last_fullsized":"2023-08-25T09:50:12.556745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:22:52.170219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554991+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:50:12.554991+0000","last_peered":"2023-08-25T09:50:12.554991+0000","last_clean":"2023-08-25T09:50:12.554991+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:50:12.554991+0000","last_undegraded":"2023-08-25T09:50:12.554991+0000","last_fullsized":"2023-08-25T09:50:12.554991+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:54:40.403648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761278+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:50:22.761278+0000","last_peered":"2023-08-25T09:50:22.761278+0000","last_clean":"2023-08-25T09:50:22.761278+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:50:22.761278+0000","last_undegraded":"2023-08-25T09:50:22.761278+0000","last_fullsized":"2023-08-25T09:50:22.761278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:09:51.901728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556688+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:50:12.556688+0000","last_peered":"2023-08-25T09:50:12.556688+0000","last_clean":"2023-08-25T09:50:12.556688+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:50:12.556688+0000","last_undegraded":"2023-08-25T09:50:12.556688+0000","last_fullsized":"2023-08-25T09:50:12.556688+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:59:46.452007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558337+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:50:12.558337+0000","last_peered":"2023-08-25T09:50:12.558337+0000","last_clean":"2023-08-25T09:50:12.558337+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:50:12.558337+0000","last_undegraded":"2023-08-25T09:50:12.558337+0000","last_fullsized":"2023-08-25T09:50:12.558337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:08:31.091599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627499+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:50:12.627499+0000","last_peered":"2023-08-25T09:50:12.627499+0000","last_clean":"2023-08-25T09:50:12.627499+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:50:12.627499+0000","last_undegraded":"2023-08-25T09:50:12.627499+0000","last_fullsized":"2023-08-25T09:50:12.627499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:56:43.404686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626320+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:50:12.626320+0000","last_peered":"2023-08-25T09:50:12.626320+0000","last_clean":"2023-08-25T09:50:12.626320+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:50:12.626320+0000","last_undegraded":"2023-08-25T09:50:12.626320+0000","last_fullsized":"2023-08-25T09:50:12.626320+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:36:12.229666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556039+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:50:12.556039+0000","last_peered":"2023-08-25T09:50:12.556039+0000","last_clean":"2023-08-25T09:50:12.556039+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:50:12.556039+0000","last_undegraded":"2023-08-25T09:50:12.556039+0000","last_fullsized":"2023-08-25T09:50:12.556039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:34:48.602052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'17","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.572678+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:50:17.572678+0000","last_peered":"2023-08-25T09:50:17.572678+0000","last_clean":"2023-08-25T09:50:17.572678+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:50:17.572678+0000","last_undegraded":"2023-08-25T09:50:17.572678+0000","last_fullsized":"2023-08-25T09:50:17.572678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:32.295126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.633719+0000","last_change":"2023-08-25T09:50:12.631949+0000","last_active":"2023-08-25T09:50:12.633719+0000","last_peered":"2023-08-25T09:50:12.633719+0000","last_clean":"2023-08-25T09:50:12.633719+0000","last_became_active":"2023-08-25T09:50:12.630910+0000","last_became_peered":"2023-08-25T09:50:12.630910+0000","last_unstale":"2023-08-25T09:50:12.633719+0000","last_undegraded":"2023-08-25T09:50:12.633719+0000","last_fullsized":"2023-08-25T09:50:12.633719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:36:40.321956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627733+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:50:12.627733+0000","last_peered":"2023-08-25T09:50:12.627733+0000","last_clean":"2023-08-25T09:50:12.627733+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:50:12.627733+0000","last_undegraded":"2023-08-25T09:50:12.627733+0000","last_fullsized":"2023-08-25T09:50:12.627733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:10:04.733907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573176+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:50:17.573176+0000","last_peered":"2023-08-25T09:50:17.573176+0000","last_clean":"2023-08-25T09:50:17.573176+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:50:17.573176+0000","last_undegraded":"2023-08-25T09:50:17.573176+0000","last_fullsized":"2023-08-25T09:50:17.573176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:43:49.838183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552014+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:50:12.552014+0000","last_peered":"2023-08-25T09:50:12.552014+0000","last_clean":"2023-08-25T09:50:12.552014+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:50:12.552014+0000","last_undegraded":"2023-08-25T09:50:12.552014+0000","last_fullsized":"2023-08-25T09:50:12.552014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:55:55.008026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.610262+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:50:17.610262+0000","last_peered":"2023-08-25T09:50:17.610262+0000","last_clean":"2023-08-25T09:50:17.610262+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:50:17.610262+0000","last_undegraded":"2023-08-25T09:50:17.610262+0000","last_fullsized":"2023-08-25T09:50:17.610262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:10.028050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555549+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:50:12.555549+0000","last_peered":"2023-08-25T09:50:12.555549+0000","last_clean":"2023-08-25T09:50:12.555549+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:50:12.555549+0000","last_undegraded":"2023-08-25T09:50:12.555549+0000","last_fullsized":"2023-08-25T09:50:12.555549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:27:10.547485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628374+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:50:12.628374+0000","last_peered":"2023-08-25T09:50:12.628374+0000","last_clean":"2023-08-25T09:50:12.628374+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:50:12.628374+0000","last_undegraded":"2023-08-25T09:50:12.628374+0000","last_fullsized":"2023-08-25T09:50:12.628374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:30:14.626577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"31'9","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761509+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:50:22.761509+0000","last_peered":"2023-08-25T09:50:22.761509+0000","last_clean":"2023-08-25T09:50:22.761509+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:50:22.761509+0000","last_undegraded":"2023-08-25T09:50:22.761509+0000","last_fullsized":"2023-08-25T09:50:22.761509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:30.647173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628001+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:50:12.628001+0000","last_peered":"2023-08-25T09:50:12.628001+0000","last_clean":"2023-08-25T09:50:12.628001+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:50:12.628001+0000","last_undegraded":"2023-08-25T09:50:12.628001+0000","last_fullsized":"2023-08-25T09:50:12.628001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:05.525804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554765+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:50:12.554765+0000","last_peered":"2023-08-25T09:50:12.554765+0000","last_clean":"2023-08-25T09:50:12.554765+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:50:12.554765+0000","last_undegraded":"2023-08-25T09:50:12.554765+0000","last_fullsized":"2023-08-25T09:50:12.554765+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:06:14.805887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.564968+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:50:12.564968+0000","last_peered":"2023-08-25T09:50:12.564968+0000","last_clean":"2023-08-25T09:50:12.564968+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:50:12.564968+0000","last_undegraded":"2023-08-25T09:50:12.564968+0000","last_fullsized":"2023-08-25T09:50:12.564968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:08:13.506185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625740+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:50:12.625740+0000","last_peered":"2023-08-25T09:50:12.625740+0000","last_clean":"2023-08-25T09:50:12.625740+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:50:12.625740+0000","last_undegraded":"2023-08-25T09:50:12.625740+0000","last_fullsized":"2023-08-25T09:50:12.625740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:06:22.528448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558622+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:50:12.558622+0000","last_peered":"2023-08-25T09:50:12.558622+0000","last_clean":"2023-08-25T09:50:12.558622+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:50:12.558622+0000","last_undegraded":"2023-08-25T09:50:12.558622+0000","last_fullsized":"2023-08-25T09:50:12.558622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:59:47.407847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627103+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:50:12.627103+0000","last_peered":"2023-08-25T09:50:12.627103+0000","last_clean":"2023-08-25T09:50:12.627103+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:50:12.627103+0000","last_undegraded":"2023-08-25T09:50:12.627103+0000","last_fullsized":"2023-08-25T09:50:12.627103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:39:13.685181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628326+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:50:12.628326+0000","last_peered":"2023-08-25T09:50:12.628326+0000","last_clean":"2023-08-25T09:50:12.628326+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:50:12.628326+0000","last_undegraded":"2023-08-25T09:50:12.628326+0000","last_fullsized":"2023-08-25T09:50:12.628326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:10:35.667202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555241+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:50:12.555241+0000","last_peered":"2023-08-25T09:50:12.555241+0000","last_clean":"2023-08-25T09:50:12.555241+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:50:12.555241+0000","last_undegraded":"2023-08-25T09:50:12.555241+0000","last_fullsized":"2023-08-25T09:50:12.555241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:58.938045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630384+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:50:12.630384+0000","last_peered":"2023-08-25T09:50:12.630384+0000","last_clean":"2023-08-25T09:50:12.630384+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:50:12.630384+0000","last_undegraded":"2023-08-25T09:50:12.630384+0000","last_fullsized":"2023-08-25T09:50:12.630384+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:01:42.764778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629040+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:50:12.629040+0000","last_peered":"2023-08-25T09:50:12.629040+0000","last_clean":"2023-08-25T09:50:12.629040+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:50:12.629040+0000","last_undegraded":"2023-08-25T09:50:12.629040+0000","last_fullsized":"2023-08-25T09:50:12.629040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:47:27.685717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625993+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:50:12.625993+0000","last_peered":"2023-08-25T09:50:12.625993+0000","last_clean":"2023-08-25T09:50:12.625993+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:50:12.625993+0000","last_undegraded":"2023-08-25T09:50:12.625993+0000","last_fullsized":"2023-08-25T09:50:12.625993+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:42:27.784343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625508+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:50:12.625508+0000","last_peered":"2023-08-25T09:50:12.625508+0000","last_clean":"2023-08-25T09:50:12.625508+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:50:12.625508+0000","last_undegraded":"2023-08-25T09:50:12.625508+0000","last_fullsized":"2023-08-25T09:50:12.625508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:13:44.415314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.550868+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:50:12.550868+0000","last_peered":"2023-08-25T09:50:12.550868+0000","last_clean":"2023-08-25T09:50:12.550868+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:50:12.550868+0000","last_undegraded":"2023-08-25T09:50:12.550868+0000","last_fullsized":"2023-08-25T09:50:12.550868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:33:22.249937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626399+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:50:12.626399+0000","last_peered":"2023-08-25T09:50:12.626399+0000","last_clean":"2023-08-25T09:50:12.626399+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:50:12.626399+0000","last_undegraded":"2023-08-25T09:50:12.626399+0000","last_fullsized":"2023-08-25T09:50:12.626399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:50:48.580274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558258+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:50:12.558258+0000","last_peered":"2023-08-25T09:50:12.558258+0000","last_clean":"2023-08-25T09:50:12.558258+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:50:12.558258+0000","last_undegraded":"2023-08-25T09:50:12.558258+0000","last_fullsized":"2023-08-25T09:50:12.558258+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:01.929500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557718+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:50:12.557718+0000","last_peered":"2023-08-25T09:50:12.557718+0000","last_clean":"2023-08-25T09:50:12.557718+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:50:12.557718+0000","last_undegraded":"2023-08-25T09:50:12.557718+0000","last_fullsized":"2023-08-25T09:50:12.557718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:14:25.545926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558660+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:50:12.558660+0000","last_peered":"2023-08-25T09:50:12.558660+0000","last_clean":"2023-08-25T09:50:12.558660+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:50:12.558660+0000","last_undegraded":"2023-08-25T09:50:12.558660+0000","last_fullsized":"2023-08-25T09:50:12.558660+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:12.817270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551531+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:50:12.551531+0000","last_peered":"2023-08-25T09:50:12.551531+0000","last_clean":"2023-08-25T09:50:12.551531+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:50:12.551531+0000","last_undegraded":"2023-08-25T09:50:12.551531+0000","last_fullsized":"2023-08-25T09:50:12.551531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:21:41.584105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554589+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:50:12.554589+0000","last_peered":"2023-08-25T09:50:12.554589+0000","last_clean":"2023-08-25T09:50:12.554589+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:50:12.554589+0000","last_undegraded":"2023-08-25T09:50:12.554589+0000","last_fullsized":"2023-08-25T09:50:12.554589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:10.183565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628035+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:50:12.628035+0000","last_peered":"2023-08-25T09:50:12.628035+0000","last_clean":"2023-08-25T09:50:12.628035+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:50:12.628035+0000","last_undegraded":"2023-08-25T09:50:12.628035+0000","last_fullsized":"2023-08-25T09:50:12.628035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:18:24.478817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555844+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:50:12.555844+0000","last_peered":"2023-08-25T09:50:12.555844+0000","last_clean":"2023-08-25T09:50:12.555844+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:50:12.555844+0000","last_undegraded":"2023-08-25T09:50:12.555844+0000","last_fullsized":"2023-08-25T09:50:12.555844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:46:47.388440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557103+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:50:12.557103+0000","last_peered":"2023-08-25T09:50:12.557103+0000","last_clean":"2023-08-25T09:50:12.557103+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:50:12.557103+0000","last_undegraded":"2023-08-25T09:50:12.557103+0000","last_fullsized":"2023-08-25T09:50:12.557103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:19.268187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557240+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:50:12.557240+0000","last_peered":"2023-08-25T09:50:12.557240+0000","last_clean":"2023-08-25T09:50:12.557240+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:50:12.557240+0000","last_undegraded":"2023-08-25T09:50:12.557240+0000","last_fullsized":"2023-08-25T09:50:12.557240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:26:06.862900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558086+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:50:12.558086+0000","last_peered":"2023-08-25T09:50:12.558086+0000","last_clean":"2023-08-25T09:50:12.558086+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:50:12.558086+0000","last_undegraded":"2023-08-25T09:50:12.558086+0000","last_fullsized":"2023-08-25T09:50:12.558086+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:21.290722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557015+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:50:12.557015+0000","last_peered":"2023-08-25T09:50:12.557015+0000","last_clean":"2023-08-25T09:50:12.557015+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:50:12.557015+0000","last_undegraded":"2023-08-25T09:50:12.557015+0000","last_fullsized":"2023-08-25T09:50:12.557015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:02:22.581658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551227+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:50:12.551227+0000","last_peered":"2023-08-25T09:50:12.551227+0000","last_clean":"2023-08-25T09:50:12.551227+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:50:12.551227+0000","last_undegraded":"2023-08-25T09:50:12.551227+0000","last_fullsized":"2023-08-25T09:50:12.551227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:15:53.937529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554170+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:50:12.554170+0000","last_peered":"2023-08-25T09:50:12.554170+0000","last_clean":"2023-08-25T09:50:12.554170+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:50:12.554170+0000","last_undegraded":"2023-08-25T09:50:12.554170+0000","last_fullsized":"2023-08-25T09:50:12.554170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:36:20.354285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627168+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:50:12.627168+0000","last_peered":"2023-08-25T09:50:12.627168+0000","last_clean":"2023-08-25T09:50:12.627168+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:50:12.627168+0000","last_undegraded":"2023-08-25T09:50:12.627168+0000","last_fullsized":"2023-08-25T09:50:12.627168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:11.563460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557753+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:50:12.557753+0000","last_peered":"2023-08-25T09:50:12.557753+0000","last_clean":"2023-08-25T09:50:12.557753+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:50:12.557753+0000","last_undegraded":"2023-08-25T09:50:12.557753+0000","last_fullsized":"2023-08-25T09:50:12.557753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:08:46.951943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551870+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:50:12.551870+0000","last_peered":"2023-08-25T09:50:12.551870+0000","last_clean":"2023-08-25T09:50:12.551870+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:50:12.551870+0000","last_undegraded":"2023-08-25T09:50:12.551870+0000","last_fullsized":"2023-08-25T09:50:12.551870+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:58:12.257947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565670+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:50:12.565670+0000","last_peered":"2023-08-25T09:50:12.565670+0000","last_clean":"2023-08-25T09:50:12.565670+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:50:12.565670+0000","last_undegraded":"2023-08-25T09:50:12.565670+0000","last_fullsized":"2023-08-25T09:50:12.565670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:24:09.243873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557849+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:50:12.557849+0000","last_peered":"2023-08-25T09:50:12.557849+0000","last_clean":"2023-08-25T09:50:12.557849+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:50:12.557849+0000","last_undegraded":"2023-08-25T09:50:12.557849+0000","last_fullsized":"2023-08-25T09:50:12.557849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:33:53.261227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556583+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:50:12.556583+0000","last_peered":"2023-08-25T09:50:12.556583+0000","last_clean":"2023-08-25T09:50:12.556583+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:50:12.556583+0000","last_undegraded":"2023-08-25T09:50:12.556583+0000","last_fullsized":"2023-08-25T09:50:12.556583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:06:43.603905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557609+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:50:12.557609+0000","last_peered":"2023-08-25T09:50:12.557609+0000","last_clean":"2023-08-25T09:50:12.557609+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:50:12.557609+0000","last_undegraded":"2023-08-25T09:50:12.557609+0000","last_fullsized":"2023-08-25T09:50:12.557609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:13:32.695366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":12023,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191,"ondisk_log_size":191,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":147818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.4239999999999999}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":2.9159999999999999,"5min":2.9159999999999999,"15min":2.9159999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.4810000000000001}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.86799999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.115},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.278}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":1.083,"5min":1.083,"15min":1.083},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.244}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.825,"15min":1.825},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":7.2990000000000004,"5min":7.2990000000000004,"15min":7.2990000000000004},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.101,"5min":1.101,"15min":1.101},"max":{"1min":4.875,"5min":4.875,"15min":4.875},"last":1.375}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":4.8079999999999998,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":1.331},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.5720000000000001}]}]},{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":446464,"data_stored":737126,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.85699999999999998}]},{"osd":2,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.119},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.282}]},{"osd":3,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":0.997}]},{"osd":4,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.343}]},{"osd":6,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1439999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509460,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":140617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.754},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.1419999999999999}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.411}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.3520000000000001}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.246},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.92000000000000004}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.518},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.0780000000000001}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.474},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.383}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":3.399,"5min":3.399,"15min":3.399},"last":1.03},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.7189999999999999,"5min":2.7189999999999999,"15min":2.7189999999999999},"last":1.3}]}]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":376832,"data_stored":144319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.032},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.4810000000000001}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.26}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.5289999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.355},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.1339999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.611}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.232},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":1.3819999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.5329999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":1.554},{"interface":"front","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":1.242,"5min":1.242,"15min":1.242},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.456}]}]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":144091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.103}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.034},{"interface":"front","average":{"1min":1.304,"5min":1.304,"15min":1.304},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.8290000000000002,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.155},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":1.1679999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.90300000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":140595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.56599999999999995}]},{"osd":3,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.95799999999999996}]},{"osd":5,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.353,"15min":1.353},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.76500000000000001}]},{"osd":7,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.92200000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":140505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.032},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.1719999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.075},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.5590000000000002,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.2689999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":0.90700000000000003}]},{"osd":5,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":1.609}]},{"osd":6,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.3819999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.53},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.135}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":454656,"data_stored":733751,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.96499999999999997}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.0309999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.204},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":1.357}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.325}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.006},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.272}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.232},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.92400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":2878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":7313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":6253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:41.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:41.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.694+0000 7f0ef462d640 1 Processor -- start 2023-08-25T09:50:41.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.695+0000 7f0ef462d640 1 -- start start 2023-08-25T09:50:41.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.696+0000 7f0ef462d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0eec159fc0 0x7f0eec0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.696+0000 7f0ef462d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec06a960 con 0x7f0eec06cc70 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.696+0000 7f0ef462d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec06ac20 con 0x7f0eec1594d0 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef462d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec06af00 con 0x7f0eec159fc0 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef2bca640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0eec159fc0 0x7f0eec0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef1bc8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0eec06cc70 0x7f0eec1598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43610/0 (socket says 172.21.15.71:43610) 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef1bc8640 1 -- 172.21.15.71:0/326674643 learned_addr learned my addr 172.21.15.71:0/326674643 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef2bca640 1 -- 172.21.15.71:0/326674643 >> v1:172.21.15.164:6789/0 conn(0x7f0eec1594d0 legacy=0x7f0eec14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef2bca640 1 -- 172.21.15.71:0/326674643 >> v1:172.21.15.71:6789/0 conn(0x7f0eec06cc70 legacy=0x7f0eec1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef2bca640 1 -- 172.21.15.71:0/326674643 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec0c2e60 con 0x7f0eec159fc0 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.697+0000 7f0ef13c7640 1 -- 172.21.15.71:0/326674643 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623151965 0 0) 0x7f0eec06a960 con 0x7f0eec06cc70 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.698+0000 7f0ef2bca640 1 --2- 172.21.15.71:0/326674643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec159fc0 0x7f0eec0c4c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009d90 tx=0x7f0ee402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e635acb0052989 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.699+0000 7f0ef13c7640 1 -- 172.21.15.71:0/326674643 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee402cd80 con 0x7f0eec159fc0 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.699+0000 7f0ef13c7640 1 -- 172.21.15.71:0/326674643 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee4034440 con 0x7f0eec159fc0 2023-08-25T09:50:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.699+0000 7f0ef13c7640 1 -- 172.21.15.71:0/326674643 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40347d0 con 0x7f0eec159fc0 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.699+0000 7f0ef462d640 1 -- 172.21.15.71:0/326674643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec159fc0 msgr2=0x7f0eec0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.699+0000 7f0ef462d640 1 --2- 172.21.15.71:0/326674643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec159fc0 0x7f0eec0c4c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009d90 tx=0x7f0ee402bfd0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.700+0000 7f0ef462d640 1 -- 172.21.15.71:0/326674643 shutdown_connections 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.700+0000 7f0ef462d640 1 --2- 172.21.15.71:0/326674643 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec159fc0 0x7f0eec0c4c90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.700+0000 7f0ef462d640 1 -- 172.21.15.71:0/326674643 >> 172.21.15.71:0/326674643 conn(0x7f0eec0654f0 msgr2=0x7f0eec065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.700+0000 7f0ef462d640 1 -- 172.21.15.71:0/326674643 shutdown_connections 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.700+0000 7f0ef462d640 1 -- 172.21.15.71:0/326674643 wait complete. 2023-08-25T09:50:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.701+0000 7f0ef462d640 1 Processor -- start 2023-08-25T09:50:41.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.701+0000 7f0ef462d640 1 -- start start 2023-08-25T09:50:41.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.702+0000 7f0ef462d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec06cc70 0x7f0eec14a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.702+0000 7f0ef462d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 0x7f0eec145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.702+0000 7f0ef462d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec14ad20 0x7f0eec06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.702+0000 7f0ef462d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0c5ad0 con 0x7f0eec1594d0 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.702+0000 7f0ef462d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1495e0 con 0x7f0eec14ad20 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.702+0000 7f0ef462d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1ebb10 con 0x7f0eec06cc70 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 0x7f0eec145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 0x7f0eec145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52644/0 (socket says 172.21.15.71:52644) 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 -- 172.21.15.71:0/689147345 learned_addr learned my addr 172.21.15.71:0/689147345 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef2bca640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec14ad20 0x7f0eec06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef23c9640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec06cc70 0x7f0eec14a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 -- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec06cc70 msgr2=0x7f0eec14a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec06cc70 0x7f0eec14a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 -- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec14ad20 msgr2=0x7f0eec06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec14ad20 0x7f0eec06ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 -- 172.21.15.71:0/689147345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4009a70 con 0x7f0eec1594d0 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.703+0000 7f0ef1bc8640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 0x7f0eec145710 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00c4a0 tx=0x7f0edc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.704+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc0074e0 con 0x7f0eec1594d0 2023-08-25T09:50:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.704+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0edc007640 con 0x7f0eec1594d0 2023-08-25T09:50:41.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.704+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0edc004a00 con 0x7f0eec1594d0 2023-08-25T09:50:41.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.704+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec0c5750 con 0x7f0eec1594d0 2023-08-25T09:50:41.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.704+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec149020 con 0x7f0eec1594d0 2023-08-25T09:50:41.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.705+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec1ebcd0 con 0x7f0eec1594d0 2023-08-25T09:50:41.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.706+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0edc004b60 con 0x7f0eec1594d0 2023-08-25T09:50:41.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.707+0000 7f0ee2ffd640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed006a050 0x7f0ed006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:41.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.707+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f0edc087ab0 con 0x7f0eec1594d0 2023-08-25T09:50:41.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.707+0000 7f0ef23c9640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed006a050 0x7f0ed006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:41.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.708+0000 7f0ef23c9640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed006a050 0x7f0ed006c510 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8000f60 tx=0x7f0ee8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:41.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.711+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0edc052e70 con 0x7f0eec1594d0 2023-08-25T09:50:41.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.899+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0eec0c3420 con 0x7f0ed006a050 2023-08-25T09:50:41.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.921+0000 7f0ee2ffd640 1 -- 172.21.15.71:0/689147345 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+389523 (secure 0 0 0) 0x7f0ee8001390 con 0x7f0ed006a050 2023-08-25T09:50:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:41.934 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:41.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed006a050 msgr2=0x7f0ed006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed006a050 0x7f0ed006c510 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8000f60 tx=0x7f0ee8009040 comp rx=0 tx=0).stop 2023-08-25T09:50:41.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 msgr2=0x7f0eec145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:41.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 0x7f0eec145710 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0edc00c4a0 tx=0x7f0edc00c960 comp rx=0 tx=0).stop 2023-08-25T09:50:41.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 shutdown_connections 2023-08-25T09:50:41.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec14ad20 0x7f0eec06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed006a050 0x7f0ed006c510 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1594d0 0x7f0eec145710 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 --2- 172.21.15.71:0/689147345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec06cc70 0x7f0eec14a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:41.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 >> 172.21.15.71:0/689147345 conn(0x7f0eec0654f0 msgr2=0x7f0eec069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:41.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 shutdown_connections 2023-08-25T09:50:41.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:41.929+0000 7f0ef462d640 1 -- 172.21.15.71:0/689147345 wait complete. 2023-08-25T09:50:41.949 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-08-25T09:50:40.660103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602391,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":376,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227424,"kb_used_data":2976,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747296,"statfs":{"total":773094113280,"available":772861231104,"internally_reserved":0,"allocated":3047424,"data_stored":2328822,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1760,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004736"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629137+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:50:12.629137+0000","last_peered":"2023-08-25T09:50:12.629137+0000","last_clean":"2023-08-25T09:50:12.629137+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:50:12.629137+0000","last_undegraded":"2023-08-25T09:50:12.629137+0000","last_fullsized":"2023-08-25T09:50:12.629137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:23.094991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"31'13","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.758090+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:50:22.758090+0000","last_peered":"2023-08-25T09:50:22.758090+0000","last_clean":"2023-08-25T09:50:22.758090+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:50:22.758090+0000","last_undegraded":"2023-08-25T09:50:22.758090+0000","last_fullsized":"2023-08-25T09:50:22.758090+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:58:03.201648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554648+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:50:12.554648+0000","last_peered":"2023-08-25T09:50:12.554648+0000","last_clean":"2023-08-25T09:50:12.554648+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:50:12.554648+0000","last_undegraded":"2023-08-25T09:50:12.554648+0000","last_fullsized":"2023-08-25T09:50:12.554648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:26.126307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554093+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:50:12.554093+0000","last_peered":"2023-08-25T09:50:12.554093+0000","last_clean":"2023-08-25T09:50:12.554093+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:50:12.554093+0000","last_undegraded":"2023-08-25T09:50:12.554093+0000","last_fullsized":"2023-08-25T09:50:12.554093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:22.526110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"29'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627113+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:50:12.627113+0000","last_peered":"2023-08-25T09:50:12.627113+0000","last_clean":"2023-08-25T09:50:12.627113+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:50:12.627113+0000","last_undegraded":"2023-08-25T09:50:12.627113+0000","last_fullsized":"2023-08-25T09:50:12.627113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:44:00.313382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626363+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:50:12.626363+0000","last_peered":"2023-08-25T09:50:12.626363+0000","last_clean":"2023-08-25T09:50:12.626363+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:50:12.626363+0000","last_undegraded":"2023-08-25T09:50:12.626363+0000","last_fullsized":"2023-08-25T09:50:12.626363+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:06:14.787284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556659+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:50:12.556659+0000","last_peered":"2023-08-25T09:50:12.556659+0000","last_clean":"2023-08-25T09:50:12.556659+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:50:12.556659+0000","last_undegraded":"2023-08-25T09:50:12.556659+0000","last_fullsized":"2023-08-25T09:50:12.556659+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:45:30.828332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761816+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:50:22.761816+0000","last_peered":"2023-08-25T09:50:22.761816+0000","last_clean":"2023-08-25T09:50:22.761816+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:50:22.761816+0000","last_undegraded":"2023-08-25T09:50:22.761816+0000","last_fullsized":"2023-08-25T09:50:22.761816+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:15:33.280852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558309+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:50:12.558309+0000","last_peered":"2023-08-25T09:50:12.558309+0000","last_clean":"2023-08-25T09:50:12.558309+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:50:12.558309+0000","last_undegraded":"2023-08-25T09:50:12.558309+0000","last_fullsized":"2023-08-25T09:50:12.558309+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:45.240925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626253+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:50:12.626253+0000","last_peered":"2023-08-25T09:50:12.626253+0000","last_clean":"2023-08-25T09:50:12.626253+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:50:12.626253+0000","last_undegraded":"2023-08-25T09:50:12.626253+0000","last_fullsized":"2023-08-25T09:50:12.626253+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:16:41.085653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761058+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:50:22.761058+0000","last_peered":"2023-08-25T09:50:22.761058+0000","last_clean":"2023-08-25T09:50:22.761058+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:50:22.761058+0000","last_undegraded":"2023-08-25T09:50:22.761058+0000","last_fullsized":"2023-08-25T09:50:22.761058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:10:43.593031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556912+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:50:12.556912+0000","last_peered":"2023-08-25T09:50:12.556912+0000","last_clean":"2023-08-25T09:50:12.556912+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:50:12.556912+0000","last_undegraded":"2023-08-25T09:50:12.556912+0000","last_fullsized":"2023-08-25T09:50:12.556912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:41:19.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628183+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:50:12.628183+0000","last_peered":"2023-08-25T09:50:12.628183+0000","last_clean":"2023-08-25T09:50:12.628183+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:50:12.628183+0000","last_undegraded":"2023-08-25T09:50:12.628183+0000","last_fullsized":"2023-08-25T09:50:12.628183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:55:22.425822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552143+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:50:12.552143+0000","last_peered":"2023-08-25T09:50:12.552143+0000","last_clean":"2023-08-25T09:50:12.552143+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:50:12.552143+0000","last_undegraded":"2023-08-25T09:50:12.552143+0000","last_fullsized":"2023-08-25T09:50:12.552143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:56.784267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556859+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:50:12.556859+0000","last_peered":"2023-08-25T09:50:12.556859+0000","last_clean":"2023-08-25T09:50:12.556859+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:50:12.556859+0000","last_undegraded":"2023-08-25T09:50:12.556859+0000","last_fullsized":"2023-08-25T09:50:12.556859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:29:29.962318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557828+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:50:12.557828+0000","last_peered":"2023-08-25T09:50:12.557828+0000","last_clean":"2023-08-25T09:50:12.557828+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:50:12.557828+0000","last_undegraded":"2023-08-25T09:50:12.557828+0000","last_fullsized":"2023-08-25T09:50:12.557828+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:46.007193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.575799+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:50:17.575799+0000","last_peered":"2023-08-25T09:50:17.575799+0000","last_clean":"2023-08-25T09:50:17.575799+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:50:17.575799+0000","last_undegraded":"2023-08-25T09:50:17.575799+0000","last_fullsized":"2023-08-25T09:50:17.575799+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:23:50.467500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628494+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:50:12.628494+0000","last_peered":"2023-08-25T09:50:12.628494+0000","last_clean":"2023-08-25T09:50:12.628494+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:50:12.628494+0000","last_undegraded":"2023-08-25T09:50:12.628494+0000","last_fullsized":"2023-08-25T09:50:12.628494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:28:21.130424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566445+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:50:12.566445+0000","last_peered":"2023-08-25T09:50:12.566445+0000","last_clean":"2023-08-25T09:50:12.566445+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:50:12.566445+0000","last_undegraded":"2023-08-25T09:50:12.566445+0000","last_fullsized":"2023-08-25T09:50:12.566445+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:24.203781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630137+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:50:12.630137+0000","last_peered":"2023-08-25T09:50:12.630137+0000","last_clean":"2023-08-25T09:50:12.630137+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:50:12.630137+0000","last_undegraded":"2023-08-25T09:50:12.630137+0000","last_fullsized":"2023-08-25T09:50:12.630137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:11:08.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551378+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:50:12.551378+0000","last_peered":"2023-08-25T09:50:12.551378+0000","last_clean":"2023-08-25T09:50:12.551378+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:50:12.551378+0000","last_undegraded":"2023-08-25T09:50:12.551378+0000","last_fullsized":"2023-08-25T09:50:12.551378+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:12:56.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552046+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:50:12.552046+0000","last_peered":"2023-08-25T09:50:12.552046+0000","last_clean":"2023-08-25T09:50:12.552046+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:50:12.552046+0000","last_undegraded":"2023-08-25T09:50:12.552046+0000","last_fullsized":"2023-08-25T09:50:12.552046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:30:58.600767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573085+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:50:17.573085+0000","last_peered":"2023-08-25T09:50:17.573085+0000","last_clean":"2023-08-25T09:50:17.573085+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:50:17.573085+0000","last_undegraded":"2023-08-25T09:50:17.573085+0000","last_fullsized":"2023-08-25T09:50:17.573085+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:06.900671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566581+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:50:12.566581+0000","last_peered":"2023-08-25T09:50:12.566581+0000","last_clean":"2023-08-25T09:50:12.566581+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:50:12.566581+0000","last_undegraded":"2023-08-25T09:50:12.566581+0000","last_fullsized":"2023-08-25T09:50:12.566581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:51:56.697715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625495+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:50:12.625495+0000","last_peered":"2023-08-25T09:50:12.625495+0000","last_clean":"2023-08-25T09:50:12.625495+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:50:12.625495+0000","last_undegraded":"2023-08-25T09:50:12.625495+0000","last_fullsized":"2023-08-25T09:50:12.625495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:23:54.271157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:33.134863+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:50:33.134863+0000","last_peered":"2023-08-25T09:50:33.134863+0000","last_clean":"2023-08-25T09:50:33.134863+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:50:33.134863+0000","last_undegraded":"2023-08-25T09:50:33.134863+0000","last_fullsized":"2023-08-25T09:50:33.134863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:19:35.376472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565155+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:50:12.565155+0000","last_peered":"2023-08-25T09:50:12.565155+0000","last_clean":"2023-08-25T09:50:12.565155+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:50:12.565155+0000","last_undegraded":"2023-08-25T09:50:12.565155+0000","last_fullsized":"2023-08-25T09:50:12.565155+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:16:54.785190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626109+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:50:12.626109+0000","last_peered":"2023-08-25T09:50:12.626109+0000","last_clean":"2023-08-25T09:50:12.626109+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:50:12.626109+0000","last_undegraded":"2023-08-25T09:50:12.626109+0000","last_fullsized":"2023-08-25T09:50:12.626109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:00:08.343712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.759325+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:50:22.759325+0000","last_peered":"2023-08-25T09:50:22.759325+0000","last_clean":"2023-08-25T09:50:22.759325+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:50:22.759325+0000","last_undegraded":"2023-08-25T09:50:22.759325+0000","last_fullsized":"2023-08-25T09:50:22.759325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:38:03.473951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625586+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:50:12.625586+0000","last_peered":"2023-08-25T09:50:12.625586+0000","last_clean":"2023-08-25T09:50:12.625586+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:50:12.625586+0000","last_undegraded":"2023-08-25T09:50:12.625586+0000","last_fullsized":"2023-08-25T09:50:12.625586+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:03:07.147111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626969+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:50:12.626969+0000","last_peered":"2023-08-25T09:50:12.626969+0000","last_clean":"2023-08-25T09:50:12.626969+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:50:12.626969+0000","last_undegraded":"2023-08-25T09:50:12.626969+0000","last_fullsized":"2023-08-25T09:50:12.626969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:14:08.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630451+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:50:12.630451+0000","last_peered":"2023-08-25T09:50:12.630451+0000","last_clean":"2023-08-25T09:50:12.630451+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:50:12.630451+0000","last_undegraded":"2023-08-25T09:50:12.630451+0000","last_fullsized":"2023-08-25T09:50:12.630451+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:33.100204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626271+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:50:12.626271+0000","last_peered":"2023-08-25T09:50:12.626271+0000","last_clean":"2023-08-25T09:50:12.626271+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:50:12.626271+0000","last_undegraded":"2023-08-25T09:50:12.626271+0000","last_fullsized":"2023-08-25T09:50:12.626271+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:31:59.929747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"31'16","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:33.155598+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:50:33.155598+0000","last_peered":"2023-08-25T09:50:33.155598+0000","last_clean":"2023-08-25T09:50:33.155598+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:50:33.155598+0000","last_undegraded":"2023-08-25T09:50:33.155598+0000","last_fullsized":"2023-08-25T09:50:33.155598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:59:56.712100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4281,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557216+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:50:12.557216+0000","last_peered":"2023-08-25T09:50:12.557216+0000","last_clean":"2023-08-25T09:50:12.557216+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:50:12.557216+0000","last_undegraded":"2023-08-25T09:50:12.557216+0000","last_fullsized":"2023-08-25T09:50:12.557216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:24:19.739443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629763+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:50:12.629763+0000","last_peered":"2023-08-25T09:50:12.629763+0000","last_clean":"2023-08-25T09:50:12.629763+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:50:12.629763+0000","last_undegraded":"2023-08-25T09:50:12.629763+0000","last_fullsized":"2023-08-25T09:50:12.629763+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:08:23.975702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555540+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:50:12.555540+0000","last_peered":"2023-08-25T09:50:12.555540+0000","last_clean":"2023-08-25T09:50:12.555540+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:50:12.555540+0000","last_undegraded":"2023-08-25T09:50:12.555540+0000","last_fullsized":"2023-08-25T09:50:12.555540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:23:24.270948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566757+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:50:12.566757+0000","last_peered":"2023-08-25T09:50:12.566757+0000","last_clean":"2023-08-25T09:50:12.566757+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:50:12.566757+0000","last_undegraded":"2023-08-25T09:50:12.566757+0000","last_fullsized":"2023-08-25T09:50:12.566757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:10.046784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.576095+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:50:17.576095+0000","last_peered":"2023-08-25T09:50:17.576095+0000","last_clean":"2023-08-25T09:50:17.576095+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:50:17.576095+0000","last_undegraded":"2023-08-25T09:50:17.576095+0000","last_fullsized":"2023-08-25T09:50:17.576095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:12:06.214562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625440+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:50:12.625440+0000","last_peered":"2023-08-25T09:50:12.625440+0000","last_clean":"2023-08-25T09:50:12.625440+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:50:12.625440+0000","last_undegraded":"2023-08-25T09:50:12.625440+0000","last_fullsized":"2023-08-25T09:50:12.625440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:40:36.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625747+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:50:12.625747+0000","last_peered":"2023-08-25T09:50:12.625747+0000","last_clean":"2023-08-25T09:50:12.625747+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:50:12.625747+0000","last_undegraded":"2023-08-25T09:50:12.625747+0000","last_fullsized":"2023-08-25T09:50:12.625747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:39:48.184915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630298+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:50:12.630298+0000","last_peered":"2023-08-25T09:50:12.630298+0000","last_clean":"2023-08-25T09:50:12.630298+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:50:12.630298+0000","last_undegraded":"2023-08-25T09:50:12.630298+0000","last_fullsized":"2023-08-25T09:50:12.630298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:45:35.013224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.759471+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:50:22.759471+0000","last_peered":"2023-08-25T09:50:22.759471+0000","last_clean":"2023-08-25T09:50:22.759471+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:50:22.759471+0000","last_undegraded":"2023-08-25T09:50:22.759471+0000","last_fullsized":"2023-08-25T09:50:22.759471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:55:21.711434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554400+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:50:12.554400+0000","last_peered":"2023-08-25T09:50:12.554400+0000","last_clean":"2023-08-25T09:50:12.554400+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:50:12.554400+0000","last_undegraded":"2023-08-25T09:50:12.554400+0000","last_fullsized":"2023-08-25T09:50:12.554400+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:23:36.823930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551745+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:50:12.551745+0000","last_peered":"2023-08-25T09:50:12.551745+0000","last_clean":"2023-08-25T09:50:12.551745+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:50:12.551745+0000","last_undegraded":"2023-08-25T09:50:12.551745+0000","last_fullsized":"2023-08-25T09:50:12.551745+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:31:55.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627612+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:50:12.627612+0000","last_peered":"2023-08-25T09:50:12.627612+0000","last_clean":"2023-08-25T09:50:12.627612+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:50:12.627612+0000","last_undegraded":"2023-08-25T09:50:12.627612+0000","last_fullsized":"2023-08-25T09:50:12.627612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:44:34.927600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"31'1","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628052+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:50:12.628052+0000","last_peered":"2023-08-25T09:50:12.628052+0000","last_clean":"2023-08-25T09:50:12.628052+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:50:12.628052+0000","last_undegraded":"2023-08-25T09:50:12.628052+0000","last_fullsized":"2023-08-25T09:50:12.628052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:05:32.332141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565443+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:50:12.565443+0000","last_peered":"2023-08-25T09:50:12.565443+0000","last_clean":"2023-08-25T09:50:12.565443+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:50:12.565443+0000","last_undegraded":"2023-08-25T09:50:12.565443+0000","last_fullsized":"2023-08-25T09:50:12.565443+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:15:47.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630669+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:50:12.630669+0000","last_peered":"2023-08-25T09:50:12.630669+0000","last_clean":"2023-08-25T09:50:12.630669+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:50:12.630669+0000","last_undegraded":"2023-08-25T09:50:12.630669+0000","last_fullsized":"2023-08-25T09:50:12.630669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:18:08.404806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628723+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:50:12.628723+0000","last_peered":"2023-08-25T09:50:12.628723+0000","last_clean":"2023-08-25T09:50:12.628723+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:50:12.628723+0000","last_undegraded":"2023-08-25T09:50:12.628723+0000","last_fullsized":"2023-08-25T09:50:12.628723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:17:27.827364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.580091+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:50:17.580091+0000","last_peered":"2023-08-25T09:50:17.580091+0000","last_clean":"2023-08-25T09:50:17.580091+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:50:17.580091+0000","last_undegraded":"2023-08-25T09:50:17.580091+0000","last_fullsized":"2023-08-25T09:50:17.580091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:56:04.945533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551491+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:50:12.551491+0000","last_peered":"2023-08-25T09:50:12.551491+0000","last_clean":"2023-08-25T09:50:12.551491+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:50:12.551491+0000","last_undegraded":"2023-08-25T09:50:12.551491+0000","last_fullsized":"2023-08-25T09:50:12.551491+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:52.676480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627952+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:50:12.627952+0000","last_peered":"2023-08-25T09:50:12.627952+0000","last_clean":"2023-08-25T09:50:12.627952+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:50:12.627952+0000","last_undegraded":"2023-08-25T09:50:12.627952+0000","last_fullsized":"2023-08-25T09:50:12.627952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:42:22.416093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573338+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:50:17.573338+0000","last_peered":"2023-08-25T09:50:17.573338+0000","last_clean":"2023-08-25T09:50:17.573338+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:50:17.573338+0000","last_undegraded":"2023-08-25T09:50:17.573338+0000","last_fullsized":"2023-08-25T09:50:17.573338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:49:49.710740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556160+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:50:12.556160+0000","last_peered":"2023-08-25T09:50:12.556160+0000","last_clean":"2023-08-25T09:50:12.556160+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:50:12.556160+0000","last_undegraded":"2023-08-25T09:50:12.556160+0000","last_fullsized":"2023-08-25T09:50:12.556160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:34.254249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551025+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:50:12.551025+0000","last_peered":"2023-08-25T09:50:12.551025+0000","last_clean":"2023-08-25T09:50:12.551025+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:50:12.551025+0000","last_undegraded":"2023-08-25T09:50:12.551025+0000","last_fullsized":"2023-08-25T09:50:12.551025+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:38:55.415402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629657+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:50:12.629657+0000","last_peered":"2023-08-25T09:50:12.629657+0000","last_clean":"2023-08-25T09:50:12.629657+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:50:12.629657+0000","last_undegraded":"2023-08-25T09:50:12.629657+0000","last_fullsized":"2023-08-25T09:50:12.629657+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:50:25.245004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627351+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:50:12.627351+0000","last_peered":"2023-08-25T09:50:12.627351+0000","last_clean":"2023-08-25T09:50:12.627351+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:50:12.627351+0000","last_undegraded":"2023-08-25T09:50:12.627351+0000","last_fullsized":"2023-08-25T09:50:12.627351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:41:28.016611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557166+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:50:12.557166+0000","last_peered":"2023-08-25T09:50:12.557166+0000","last_clean":"2023-08-25T09:50:12.557166+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:50:12.557166+0000","last_undegraded":"2023-08-25T09:50:12.557166+0000","last_fullsized":"2023-08-25T09:50:12.557166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:09:13.822549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557109+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:50:12.557109+0000","last_peered":"2023-08-25T09:50:12.557109+0000","last_clean":"2023-08-25T09:50:12.557109+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:50:12.557109+0000","last_undegraded":"2023-08-25T09:50:12.557109+0000","last_fullsized":"2023-08-25T09:50:12.557109+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:15:47.257428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551738+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:50:12.551738+0000","last_peered":"2023-08-25T09:50:12.551738+0000","last_clean":"2023-08-25T09:50:12.551738+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:50:12.551738+0000","last_undegraded":"2023-08-25T09:50:12.551738+0000","last_fullsized":"2023-08-25T09:50:12.551738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:22:38.020973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761598+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:50:22.761598+0000","last_peered":"2023-08-25T09:50:22.761598+0000","last_clean":"2023-08-25T09:50:22.761598+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:50:22.761598+0000","last_undegraded":"2023-08-25T09:50:22.761598+0000","last_fullsized":"2023-08-25T09:50:22.761598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:02:36.889545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626347+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:50:12.626347+0000","last_peered":"2023-08-25T09:50:12.626347+0000","last_clean":"2023-08-25T09:50:12.626347+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:50:12.626347+0000","last_undegraded":"2023-08-25T09:50:12.626347+0000","last_fullsized":"2023-08-25T09:50:12.626347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:52:59.975913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558258+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:50:12.558258+0000","last_peered":"2023-08-25T09:50:12.558258+0000","last_clean":"2023-08-25T09:50:12.558258+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:50:12.558258+0000","last_undegraded":"2023-08-25T09:50:12.558258+0000","last_fullsized":"2023-08-25T09:50:12.558258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:11.134847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566106+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:50:12.566106+0000","last_peered":"2023-08-25T09:50:12.566106+0000","last_clean":"2023-08-25T09:50:12.566106+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:50:12.566106+0000","last_undegraded":"2023-08-25T09:50:12.566106+0000","last_fullsized":"2023-08-25T09:50:12.566106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:29:31.536127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761717+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:50:22.761717+0000","last_peered":"2023-08-25T09:50:22.761717+0000","last_clean":"2023-08-25T09:50:22.761717+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:50:22.761717+0000","last_undegraded":"2023-08-25T09:50:22.761717+0000","last_fullsized":"2023-08-25T09:50:22.761717+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:07:02.583931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556866+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:50:12.556866+0000","last_peered":"2023-08-25T09:50:12.556866+0000","last_clean":"2023-08-25T09:50:12.556866+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:50:12.556866+0000","last_undegraded":"2023-08-25T09:50:12.556866+0000","last_fullsized":"2023-08-25T09:50:12.556866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:43:35.431443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":187,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626710+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:50:12.626710+0000","last_peered":"2023-08-25T09:50:12.626710+0000","last_clean":"2023-08-25T09:50:12.626710+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:50:12.626710+0000","last_undegraded":"2023-08-25T09:50:12.626710+0000","last_fullsized":"2023-08-25T09:50:12.626710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:37:25.347377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626465+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:50:12.626465+0000","last_peered":"2023-08-25T09:50:12.626465+0000","last_clean":"2023-08-25T09:50:12.626465+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:50:12.626465+0000","last_undegraded":"2023-08-25T09:50:12.626465+0000","last_fullsized":"2023-08-25T09:50:12.626465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:49:37.046349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629357+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:50:12.629357+0000","last_peered":"2023-08-25T09:50:12.629357+0000","last_clean":"2023-08-25T09:50:12.629357+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:50:12.629357+0000","last_undegraded":"2023-08-25T09:50:12.629357+0000","last_fullsized":"2023-08-25T09:50:12.629357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:29:01.327572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"29'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630079+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:50:12.630079+0000","last_peered":"2023-08-25T09:50:12.630079+0000","last_clean":"2023-08-25T09:50:12.630079+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:50:12.630079+0000","last_undegraded":"2023-08-25T09:50:12.630079+0000","last_fullsized":"2023-08-25T09:50:12.630079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:07:37.714359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.559002+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:50:12.559002+0000","last_peered":"2023-08-25T09:50:12.559002+0000","last_clean":"2023-08-25T09:50:12.559002+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:50:12.559002+0000","last_undegraded":"2023-08-25T09:50:12.559002+0000","last_fullsized":"2023-08-25T09:50:12.559002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:23:29.317977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.559077+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:50:12.559077+0000","last_peered":"2023-08-25T09:50:12.559077+0000","last_clean":"2023-08-25T09:50:12.559077+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:50:12.559077+0000","last_undegraded":"2023-08-25T09:50:12.559077+0000","last_fullsized":"2023-08-25T09:50:12.559077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:09:36.207083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.755917+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:50:22.755917+0000","last_peered":"2023-08-25T09:50:22.755917+0000","last_clean":"2023-08-25T09:50:22.755917+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:50:22.755917+0000","last_undegraded":"2023-08-25T09:50:22.755917+0000","last_fullsized":"2023-08-25T09:50:22.755917+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:35.082983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556600+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:50:12.556600+0000","last_peered":"2023-08-25T09:50:12.556600+0000","last_clean":"2023-08-25T09:50:12.556600+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:50:12.556600+0000","last_undegraded":"2023-08-25T09:50:12.556600+0000","last_fullsized":"2023-08-25T09:50:12.556600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:57:07.092488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558495+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:50:12.558495+0000","last_peered":"2023-08-25T09:50:12.558495+0000","last_clean":"2023-08-25T09:50:12.558495+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:50:12.558495+0000","last_undegraded":"2023-08-25T09:50:12.558495+0000","last_fullsized":"2023-08-25T09:50:12.558495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:49:01.069322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573225+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:50:17.573225+0000","last_peered":"2023-08-25T09:50:17.573225+0000","last_clean":"2023-08-25T09:50:17.573225+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:50:17.573225+0000","last_undegraded":"2023-08-25T09:50:17.573225+0000","last_fullsized":"2023-08-25T09:50:17.573225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:04:42.286097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627074+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:50:12.627074+0000","last_peered":"2023-08-25T09:50:12.627074+0000","last_clean":"2023-08-25T09:50:12.627074+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:50:12.627074+0000","last_undegraded":"2023-08-25T09:50:12.627074+0000","last_fullsized":"2023-08-25T09:50:12.627074+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:10:04.637461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558961+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:50:12.558961+0000","last_peered":"2023-08-25T09:50:12.558961+0000","last_clean":"2023-08-25T09:50:12.558961+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:50:12.558961+0000","last_undegraded":"2023-08-25T09:50:12.558961+0000","last_fullsized":"2023-08-25T09:50:12.558961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:21.839926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"29'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627445+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:50:12.627445+0000","last_peered":"2023-08-25T09:50:12.627445+0000","last_clean":"2023-08-25T09:50:12.627445+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:50:12.627445+0000","last_undegraded":"2023-08-25T09:50:12.627445+0000","last_fullsized":"2023-08-25T09:50:12.627445+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:34:35.939804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557002+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:50:12.557002+0000","last_peered":"2023-08-25T09:50:12.557002+0000","last_clean":"2023-08-25T09:50:12.557002+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:50:12.557002+0000","last_undegraded":"2023-08-25T09:50:12.557002+0000","last_fullsized":"2023-08-25T09:50:12.557002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:40:23.888752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551613+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:50:12.551613+0000","last_peered":"2023-08-25T09:50:12.551613+0000","last_clean":"2023-08-25T09:50:12.551613+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:50:12.551613+0000","last_undegraded":"2023-08-25T09:50:12.551613+0000","last_fullsized":"2023-08-25T09:50:12.551613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:34:14.609357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.582719+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:50:17.582719+0000","last_peered":"2023-08-25T09:50:17.582719+0000","last_clean":"2023-08-25T09:50:17.582719+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:50:17.582719+0000","last_undegraded":"2023-08-25T09:50:17.582719+0000","last_fullsized":"2023-08-25T09:50:17.582719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:18:04.169294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556982+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:50:12.556982+0000","last_peered":"2023-08-25T09:50:12.556982+0000","last_clean":"2023-08-25T09:50:12.556982+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:50:12.556982+0000","last_undegraded":"2023-08-25T09:50:12.556982+0000","last_fullsized":"2023-08-25T09:50:12.556982+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:23:07.786471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.571725+0000","last_change":"2023-08-25T09:50:12.571725+0000","last_active":"2023-08-25T09:50:12.571725+0000","last_peered":"2023-08-25T09:50:12.571725+0000","last_clean":"2023-08-25T09:50:12.571725+0000","last_became_active":"2023-08-25T09:50:12.570960+0000","last_became_peered":"2023-08-25T09:50:12.570960+0000","last_unstale":"2023-08-25T09:50:12.571725+0000","last_undegraded":"2023-08-25T09:50:12.571725+0000","last_fullsized":"2023-08-25T09:50:12.571725+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:36.819051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.570490+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:50:17.570490+0000","last_peered":"2023-08-25T09:50:17.570490+0000","last_clean":"2023-08-25T09:50:17.570490+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:50:17.570490+0000","last_undegraded":"2023-08-25T09:50:17.570490+0000","last_fullsized":"2023-08-25T09:50:17.570490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:21:54.167747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556745+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:50:12.556745+0000","last_peered":"2023-08-25T09:50:12.556745+0000","last_clean":"2023-08-25T09:50:12.556745+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:50:12.556745+0000","last_undegraded":"2023-08-25T09:50:12.556745+0000","last_fullsized":"2023-08-25T09:50:12.556745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:22:52.170219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554991+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:50:12.554991+0000","last_peered":"2023-08-25T09:50:12.554991+0000","last_clean":"2023-08-25T09:50:12.554991+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:50:12.554991+0000","last_undegraded":"2023-08-25T09:50:12.554991+0000","last_fullsized":"2023-08-25T09:50:12.554991+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:54:40.403648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761278+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:50:22.761278+0000","last_peered":"2023-08-25T09:50:22.761278+0000","last_clean":"2023-08-25T09:50:22.761278+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:50:22.761278+0000","last_undegraded":"2023-08-25T09:50:22.761278+0000","last_fullsized":"2023-08-25T09:50:22.761278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:09:51.901728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556688+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:50:12.556688+0000","last_peered":"2023-08-25T09:50:12.556688+0000","last_clean":"2023-08-25T09:50:12.556688+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:50:12.556688+0000","last_undegraded":"2023-08-25T09:50:12.556688+0000","last_fullsized":"2023-08-25T09:50:12.556688+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:59:46.452007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558337+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:50:12.558337+0000","last_peered":"2023-08-25T09:50:12.558337+0000","last_clean":"2023-08-25T09:50:12.558337+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:50:12.558337+0000","last_undegraded":"2023-08-25T09:50:12.558337+0000","last_fullsized":"2023-08-25T09:50:12.558337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:08:31.091599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627499+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:50:12.627499+0000","last_peered":"2023-08-25T09:50:12.627499+0000","last_clean":"2023-08-25T09:50:12.627499+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:50:12.627499+0000","last_undegraded":"2023-08-25T09:50:12.627499+0000","last_fullsized":"2023-08-25T09:50:12.627499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:56:43.404686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626320+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:50:12.626320+0000","last_peered":"2023-08-25T09:50:12.626320+0000","last_clean":"2023-08-25T09:50:12.626320+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:50:12.626320+0000","last_undegraded":"2023-08-25T09:50:12.626320+0000","last_fullsized":"2023-08-25T09:50:12.626320+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:36:12.229666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556039+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:50:12.556039+0000","last_peered":"2023-08-25T09:50:12.556039+0000","last_clean":"2023-08-25T09:50:12.556039+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:50:12.556039+0000","last_undegraded":"2023-08-25T09:50:12.556039+0000","last_fullsized":"2023-08-25T09:50:12.556039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:34:48.602052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'17","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.572678+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:50:17.572678+0000","last_peered":"2023-08-25T09:50:17.572678+0000","last_clean":"2023-08-25T09:50:17.572678+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:50:17.572678+0000","last_undegraded":"2023-08-25T09:50:17.572678+0000","last_fullsized":"2023-08-25T09:50:17.572678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:32.295126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.633719+0000","last_change":"2023-08-25T09:50:12.631949+0000","last_active":"2023-08-25T09:50:12.633719+0000","last_peered":"2023-08-25T09:50:12.633719+0000","last_clean":"2023-08-25T09:50:12.633719+0000","last_became_active":"2023-08-25T09:50:12.630910+0000","last_became_peered":"2023-08-25T09:50:12.630910+0000","last_unstale":"2023-08-25T09:50:12.633719+0000","last_undegraded":"2023-08-25T09:50:12.633719+0000","last_fullsized":"2023-08-25T09:50:12.633719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:36:40.321956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627733+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:50:12.627733+0000","last_peered":"2023-08-25T09:50:12.627733+0000","last_clean":"2023-08-25T09:50:12.627733+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:50:12.627733+0000","last_undegraded":"2023-08-25T09:50:12.627733+0000","last_fullsized":"2023-08-25T09:50:12.627733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:10:04.733907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573176+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:50:17.573176+0000","last_peered":"2023-08-25T09:50:17.573176+0000","last_clean":"2023-08-25T09:50:17.573176+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:50:17.573176+0000","last_undegraded":"2023-08-25T09:50:17.573176+0000","last_fullsized":"2023-08-25T09:50:17.573176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:43:49.838183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552014+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:50:12.552014+0000","last_peered":"2023-08-25T09:50:12.552014+0000","last_clean":"2023-08-25T09:50:12.552014+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:50:12.552014+0000","last_undegraded":"2023-08-25T09:50:12.552014+0000","last_fullsized":"2023-08-25T09:50:12.552014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:55:55.008026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.610262+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:50:17.610262+0000","last_peered":"2023-08-25T09:50:17.610262+0000","last_clean":"2023-08-25T09:50:17.610262+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:50:17.610262+0000","last_undegraded":"2023-08-25T09:50:17.610262+0000","last_fullsized":"2023-08-25T09:50:17.610262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:10.028050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555549+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:50:12.555549+0000","last_peered":"2023-08-25T09:50:12.555549+0000","last_clean":"2023-08-25T09:50:12.555549+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:50:12.555549+0000","last_undegraded":"2023-08-25T09:50:12.555549+0000","last_fullsized":"2023-08-25T09:50:12.555549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:27:10.547485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628374+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:50:12.628374+0000","last_peered":"2023-08-25T09:50:12.628374+0000","last_clean":"2023-08-25T09:50:12.628374+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:50:12.628374+0000","last_undegraded":"2023-08-25T09:50:12.628374+0000","last_fullsized":"2023-08-25T09:50:12.628374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:30:14.626577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"31'9","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761509+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:50:22.761509+0000","last_peered":"2023-08-25T09:50:22.761509+0000","last_clean":"2023-08-25T09:50:22.761509+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:50:22.761509+0000","last_undegraded":"2023-08-25T09:50:22.761509+0000","last_fullsized":"2023-08-25T09:50:22.761509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:30.647173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628001+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:50:12.628001+0000","last_peered":"2023-08-25T09:50:12.628001+0000","last_clean":"2023-08-25T09:50:12.628001+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:50:12.628001+0000","last_undegraded":"2023-08-25T09:50:12.628001+0000","last_fullsized":"2023-08-25T09:50:12.628001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:05.525804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554765+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:50:12.554765+0000","last_peered":"2023-08-25T09:50:12.554765+0000","last_clean":"2023-08-25T09:50:12.554765+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:50:12.554765+0000","last_undegraded":"2023-08-25T09:50:12.554765+0000","last_fullsized":"2023-08-25T09:50:12.554765+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:06:14.805887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.564968+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:50:12.564968+0000","last_peered":"2023-08-25T09:50:12.564968+0000","last_clean":"2023-08-25T09:50:12.564968+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:50:12.564968+0000","last_undegraded":"2023-08-25T09:50:12.564968+0000","last_fullsized":"2023-08-25T09:50:12.564968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:08:13.506185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625740+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:50:12.625740+0000","last_peered":"2023-08-25T09:50:12.625740+0000","last_clean":"2023-08-25T09:50:12.625740+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:50:12.625740+0000","last_undegraded":"2023-08-25T09:50:12.625740+0000","last_fullsized":"2023-08-25T09:50:12.625740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:06:22.528448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558622+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:50:12.558622+0000","last_peered":"2023-08-25T09:50:12.558622+0000","last_clean":"2023-08-25T09:50:12.558622+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:50:12.558622+0000","last_undegraded":"2023-08-25T09:50:12.558622+0000","last_fullsized":"2023-08-25T09:50:12.558622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:59:47.407847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627103+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:50:12.627103+0000","last_peered":"2023-08-25T09:50:12.627103+0000","last_clean":"2023-08-25T09:50:12.627103+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:50:12.627103+0000","last_undegraded":"2023-08-25T09:50:12.627103+0000","last_fullsized":"2023-08-25T09:50:12.627103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:39:13.685181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628326+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:50:12.628326+0000","last_peered":"2023-08-25T09:50:12.628326+0000","last_clean":"2023-08-25T09:50:12.628326+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:50:12.628326+0000","last_undegraded":"2023-08-25T09:50:12.628326+0000","last_fullsized":"2023-08-25T09:50:12.628326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:10:35.667202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555241+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:50:12.555241+0000","last_peered":"2023-08-25T09:50:12.555241+0000","last_clean":"2023-08-25T09:50:12.555241+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:50:12.555241+0000","last_undegraded":"2023-08-25T09:50:12.555241+0000","last_fullsized":"2023-08-25T09:50:12.555241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:58.938045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630384+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:50:12.630384+0000","last_peered":"2023-08-25T09:50:12.630384+0000","last_clean":"2023-08-25T09:50:12.630384+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:50:12.630384+0000","last_undegraded":"2023-08-25T09:50:12.630384+0000","last_fullsized":"2023-08-25T09:50:12.630384+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:01:42.764778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629040+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:50:12.629040+0000","last_peered":"2023-08-25T09:50:12.629040+0000","last_clean":"2023-08-25T09:50:12.629040+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:50:12.629040+0000","last_undegraded":"2023-08-25T09:50:12.629040+0000","last_fullsized":"2023-08-25T09:50:12.629040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:47:27.685717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625993+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:50:12.625993+0000","last_peered":"2023-08-25T09:50:12.625993+0000","last_clean":"2023-08-25T09:50:12.625993+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:50:12.625993+0000","last_undegraded":"2023-08-25T09:50:12.625993+0000","last_fullsized":"2023-08-25T09:50:12.625993+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:42:27.784343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625508+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:50:12.625508+0000","last_peered":"2023-08-25T09:50:12.625508+0000","last_clean":"2023-08-25T09:50:12.625508+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:50:12.625508+0000","last_undegraded":"2023-08-25T09:50:12.625508+0000","last_fullsized":"2023-08-25T09:50:12.625508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:13:44.415314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.550868+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:50:12.550868+0000","last_peered":"2023-08-25T09:50:12.550868+0000","last_clean":"2023-08-25T09:50:12.550868+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:50:12.550868+0000","last_undegraded":"2023-08-25T09:50:12.550868+0000","last_fullsized":"2023-08-25T09:50:12.550868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:33:22.249937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626399+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:50:12.626399+0000","last_peered":"2023-08-25T09:50:12.626399+0000","last_clean":"2023-08-25T09:50:12.626399+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:50:12.626399+0000","last_undegraded":"2023-08-25T09:50:12.626399+0000","last_fullsized":"2023-08-25T09:50:12.626399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:50:48.580274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558258+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:50:12.558258+0000","last_peered":"2023-08-25T09:50:12.558258+0000","last_clean":"2023-08-25T09:50:12.558258+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:50:12.558258+0000","last_undegraded":"2023-08-25T09:50:12.558258+0000","last_fullsized":"2023-08-25T09:50:12.558258+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:01.929500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557718+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:50:12.557718+0000","last_peered":"2023-08-25T09:50:12.557718+0000","last_clean":"2023-08-25T09:50:12.557718+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:50:12.557718+0000","last_undegraded":"2023-08-25T09:50:12.557718+0000","last_fullsized":"2023-08-25T09:50:12.557718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:14:25.545926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558660+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:50:12.558660+0000","last_peered":"2023-08-25T09:50:12.558660+0000","last_clean":"2023-08-25T09:50:12.558660+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:50:12.558660+0000","last_undegraded":"2023-08-25T09:50:12.558660+0000","last_fullsized":"2023-08-25T09:50:12.558660+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:12.817270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551531+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:50:12.551531+0000","last_peered":"2023-08-25T09:50:12.551531+0000","last_clean":"2023-08-25T09:50:12.551531+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:50:12.551531+0000","last_undegraded":"2023-08-25T09:50:12.551531+0000","last_fullsized":"2023-08-25T09:50:12.551531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:21:41.584105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554589+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:50:12.554589+0000","last_peered":"2023-08-25T09:50:12.554589+0000","last_clean":"2023-08-25T09:50:12.554589+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:50:12.554589+0000","last_undegraded":"2023-08-25T09:50:12.554589+0000","last_fullsized":"2023-08-25T09:50:12.554589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:10.183565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628035+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:50:12.628035+0000","last_peered":"2023-08-25T09:50:12.628035+0000","last_clean":"2023-08-25T09:50:12.628035+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:50:12.628035+0000","last_undegraded":"2023-08-25T09:50:12.628035+0000","last_fullsized":"2023-08-25T09:50:12.628035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:18:24.478817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555844+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:50:12.555844+0000","last_peered":"2023-08-25T09:50:12.555844+0000","last_clean":"2023-08-25T09:50:12.555844+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:50:12.555844+0000","last_undegraded":"2023-08-25T09:50:12.555844+0000","last_fullsized":"2023-08-25T09:50:12.555844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:46:47.388440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557103+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:50:12.557103+0000","last_peered":"2023-08-25T09:50:12.557103+0000","last_clean":"2023-08-25T09:50:12.557103+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:50:12.557103+0000","last_undegraded":"2023-08-25T09:50:12.557103+0000","last_fullsized":"2023-08-25T09:50:12.557103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:19.268187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557240+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:50:12.557240+0000","last_peered":"2023-08-25T09:50:12.557240+0000","last_clean":"2023-08-25T09:50:12.557240+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:50:12.557240+0000","last_undegraded":"2023-08-25T09:50:12.557240+0000","last_fullsized":"2023-08-25T09:50:12.557240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:26:06.862900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558086+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:50:12.558086+0000","last_peered":"2023-08-25T09:50:12.558086+0000","last_clean":"2023-08-25T09:50:12.558086+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:50:12.558086+0000","last_undegraded":"2023-08-25T09:50:12.558086+0000","last_fullsized":"2023-08-25T09:50:12.558086+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:21.290722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557015+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:50:12.557015+0000","last_peered":"2023-08-25T09:50:12.557015+0000","last_clean":"2023-08-25T09:50:12.557015+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:50:12.557015+0000","last_undegraded":"2023-08-25T09:50:12.557015+0000","last_fullsized":"2023-08-25T09:50:12.557015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:02:22.581658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551227+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:50:12.551227+0000","last_peered":"2023-08-25T09:50:12.551227+0000","last_clean":"2023-08-25T09:50:12.551227+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:50:12.551227+0000","last_undegraded":"2023-08-25T09:50:12.551227+0000","last_fullsized":"2023-08-25T09:50:12.551227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:15:53.937529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554170+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:50:12.554170+0000","last_peered":"2023-08-25T09:50:12.554170+0000","last_clean":"2023-08-25T09:50:12.554170+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:50:12.554170+0000","last_undegraded":"2023-08-25T09:50:12.554170+0000","last_fullsized":"2023-08-25T09:50:12.554170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:36:20.354285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627168+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:50:12.627168+0000","last_peered":"2023-08-25T09:50:12.627168+0000","last_clean":"2023-08-25T09:50:12.627168+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:50:12.627168+0000","last_undegraded":"2023-08-25T09:50:12.627168+0000","last_fullsized":"2023-08-25T09:50:12.627168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:11.563460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557753+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:50:12.557753+0000","last_peered":"2023-08-25T09:50:12.557753+0000","last_clean":"2023-08-25T09:50:12.557753+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:50:12.557753+0000","last_undegraded":"2023-08-25T09:50:12.557753+0000","last_fullsized":"2023-08-25T09:50:12.557753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:08:46.951943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551870+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:50:12.551870+0000","last_peered":"2023-08-25T09:50:12.551870+0000","last_clean":"2023-08-25T09:50:12.551870+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:50:12.551870+0000","last_undegraded":"2023-08-25T09:50:12.551870+0000","last_fullsized":"2023-08-25T09:50:12.551870+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:58:12.257947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565670+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:50:12.565670+0000","last_peered":"2023-08-25T09:50:12.565670+0000","last_clean":"2023-08-25T09:50:12.565670+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:50:12.565670+0000","last_undegraded":"2023-08-25T09:50:12.565670+0000","last_fullsized":"2023-08-25T09:50:12.565670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:24:09.243873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557849+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:50:12.557849+0000","last_peered":"2023-08-25T09:50:12.557849+0000","last_clean":"2023-08-25T09:50:12.557849+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:50:12.557849+0000","last_undegraded":"2023-08-25T09:50:12.557849+0000","last_fullsized":"2023-08-25T09:50:12.557849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:33:53.261227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556583+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:50:12.556583+0000","last_peered":"2023-08-25T09:50:12.556583+0000","last_clean":"2023-08-25T09:50:12.556583+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:50:12.556583+0000","last_undegraded":"2023-08-25T09:50:12.556583+0000","last_fullsized":"2023-08-25T09:50:12.556583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:06:43.603905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557609+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:50:12.557609+0000","last_peered":"2023-08-25T09:50:12.557609+0000","last_clean":"2023-08-25T09:50:12.557609+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:50:12.557609+0000","last_undegraded":"2023-08-25T09:50:12.557609+0000","last_fullsized":"2023-08-25T09:50:12.557609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:13:32.695366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":12023,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191,"ondisk_log_size":191,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":147818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.4239999999999999}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":2.9159999999999999,"5min":2.9159999999999999,"15min":2.9159999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.4810000000000001}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.86799999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.115},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.278}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":1.083,"5min":1.083,"15min":1.083},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.244}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.825,"15min":1.825},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":7.2990000000000004,"5min":7.2990000000000004,"15min":7.2990000000000004},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.101,"5min":1.101,"15min":1.101},"max":{"1min":4.875,"5min":4.875,"15min":4.875},"last":1.375}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":4.8079999999999998,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":1.331},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.5720000000000001}]}]},{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":446464,"data_stored":737126,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.85699999999999998}]},{"osd":2,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.119},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.282}]},{"osd":3,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":0.997}]},{"osd":4,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.343}]},{"osd":6,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1439999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509460,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":140617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.754},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.1419999999999999}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.411}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.3520000000000001}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.246},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.92000000000000004}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.518},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.0780000000000001}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.474},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.383}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":3.399,"5min":3.399,"15min":3.399},"last":1.03},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.7189999999999999,"5min":2.7189999999999999,"15min":2.7189999999999999},"last":1.3}]}]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":376832,"data_stored":144319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.032},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.4810000000000001}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.26}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.5289999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.355},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.1339999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.611}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.232},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":1.3819999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.5329999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":1.554},{"interface":"front","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":1.242,"5min":1.242,"15min":1.242},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.456}]}]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":144091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.103}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.034},{"interface":"front","average":{"1min":1.304,"5min":1.304,"15min":1.304},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.8290000000000002,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.155},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":1.1679999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.90300000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":140595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.56599999999999995}]},{"osd":3,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.95799999999999996}]},{"osd":5,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.353,"15min":1.353},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.76500000000000001}]},{"osd":7,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.92200000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":140505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.032},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.1719999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.075},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.5590000000000002,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.2689999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":0.90700000000000003}]},{"osd":5,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":1.609}]},{"osd":6,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.3819999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.53},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.135}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":454656,"data_stored":733751,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.96499999999999997}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.0309999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.204},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":1.357}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.325}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.006},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.272}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.232},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.92400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":2878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":7313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":6253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:41.957 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-08-25T09:50:41.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T09:50:42.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.209+0000 7f32a888d640 1 Processor -- start 2023-08-25T09:50:42.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.209+0000 7f32a888d640 1 -- start start 2023-08-25T09:50:42.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a888d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f32a40cb570 0x7f32a40ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a888d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40b1af0 con 0x7f32a415cd00 2023-08-25T09:50:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a888d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a40b0fb0 con 0x7f32a40cb190 2023-08-25T09:50:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a888d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32a40c98f0 con 0x7f32a40cb570 2023-08-25T09:50:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a359e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f32a415cd00 0x7f32a4158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43614/0 (socket says 172.21.15.71:43614) 2023-08-25T09:50:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a359e640 1 -- 172.21.15.71:0/1054950124 learned_addr learned my addr 172.21.15.71:0/1054950124 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f329bfff640 1 --2- 172.21.15.71:0/1054950124 >> v2:172.21.15.164:3300/0 conn(0x7f32a40cb570 0x7f32a40ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903037053 0 0) 0x7f32a40b1af0 con 0x7f32a415cd00 2023-08-25T09:50:42.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.211+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3290003520 con 0x7f32a415cd00 2023-08-25T09:50:42.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1346404132 0 0) 0x7f3290003520 con 0x7f32a415cd00 2023-08-25T09:50:42.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 >> v2:172.21.15.164:3300/0 conn(0x7f32a40cb570 msgr2=0x7f32a40ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 --2- 172.21.15.71:0/1054950124 >> v2:172.21.15.164:3300/0 conn(0x7f32a40cb570 0x7f32a40ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 >> v1:172.21.15.164:6789/0 conn(0x7f32a40cb190 legacy=0x7f32a40c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a415d130 con 0x7f32a415cd00 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f329c0019f0 con 0x7f32a415cd00 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 >> v1:172.21.15.71:6789/0 conn(0x7f32a415cd00 legacy=0x7f32a4158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32900039e0 0x7f3290003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32900047c0 0x7f3290004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3290005180 0x7f329000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40c98f0 con 0x7f3290005180 2023-08-25T09:50:42.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329000c090 con 0x7f32900039e0 2023-08-25T09:50:42.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329000c250 con 0x7f32900047c0 2023-08-25T09:50:42.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.212+0000 7f32a359e640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3290005180 0x7f329000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a2d9d640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32900039e0 0x7f3290003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a359e640 1 -- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32900047c0 msgr2=0x7f3290004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:42.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a359e640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32900047c0 0x7f3290004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a359e640 1 -- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32900039e0 msgr2=0x7f3290003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:42.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a359e640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32900039e0 0x7f3290003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a359e640 1 -- 172.21.15.71:0/1054950124 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3290003520 con 0x7f3290005180 2023-08-25T09:50:42.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.213+0000 7f32a2d9d640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32900039e0 0x7f3290003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:42.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a359e640 1 -- 172.21.15.71:0/1054950124 reap_dead start 2023-08-25T09:50:42.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a359e640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3290005180 0x7f329000b980 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f329c00a800 tx=0x7f32900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b36dbdcd0856ff server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:42.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c034630 con 0x7f3290005180 2023-08-25T09:50:42.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329c02f070 con 0x7f3290005180 2023-08-25T09:50:42.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a259c640 1 -- 172.21.15.71:0/1054950124 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329c034a10 con 0x7f3290005180 2023-08-25T09:50:42.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a888d640 1 -- 172.21.15.71:0/1054950124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3290005180 msgr2=0x7f329000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.214+0000 7f32a888d640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3290005180 0x7f329000b980 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f329c00a800 tx=0x7f32900057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.215+0000 7f32a888d640 1 -- 172.21.15.71:0/1054950124 shutdown_connections 2023-08-25T09:50:42.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.215+0000 7f32a888d640 1 --2- 172.21.15.71:0/1054950124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3290005180 0x7f329000b980 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.215+0000 7f32a888d640 1 -- 172.21.15.71:0/1054950124 >> 172.21.15.71:0/1054950124 conn(0x7f32a406ae20 msgr2=0x7f32a406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:42.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.215+0000 7f32a888d640 1 -- 172.21.15.71:0/1054950124 shutdown_connections 2023-08-25T09:50:42.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.215+0000 7f32a888d640 1 -- 172.21.15.71:0/1054950124 wait complete. 2023-08-25T09:50:42.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.216+0000 7f32a888d640 1 Processor -- start 2023-08-25T09:50:42.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.217+0000 7f32a888d640 1 -- start start 2023-08-25T09:50:42.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a888d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3290005180 0x7f32a4147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a888d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 0x7f32a4154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a888d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32a4155220 0x7f32a4155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a888d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a40cf360 con 0x7f32a414cff0 2023-08-25T09:50:42.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a888d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a414f500 con 0x7f3290005180 2023-08-25T09:50:42.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a888d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32a414e930 con 0x7f32a4155220 2023-08-25T09:50:42.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 0x7f32a4154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 0x7f32a4154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52660/0 (socket says 172.21.15.71:52660) 2023-08-25T09:50:42.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 -- 172.21.15.71:0/2218461659 learned_addr learned my addr 172.21.15.71:0/2218461659 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:42.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 -- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32a4155220 msgr2=0x7f32a4155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32a4155220 0x7f32a4155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f32a2d9d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3290005180 0x7f32a4147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 -- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3290005180 msgr2=0x7f32a4147260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3290005180 0x7f32a4147260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.218+0000 7f329bfff640 1 -- 172.21.15.71:0/2218461659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a414ebd0 con 0x7f32a414cff0 2023-08-25T09:50:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.219+0000 7f329bfff640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 0x7f32a4154ce0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f328c00e7c0 tx=0x7f328c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:42.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.219+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c00eed0 con 0x7f32a414cff0 2023-08-25T09:50:42.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.219+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c00c8c0 con 0x7f32a414cff0 2023-08-25T09:50:42.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.219+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c00cc70 con 0x7f32a414cff0 2023-08-25T09:50:42.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.219+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a414eeb0 con 0x7f32a414cff0 2023-08-25T09:50:42.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.219+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a41477a0 con 0x7f32a414cff0 2023-08-25T09:50:42.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.220+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a4072710 con 0x7f32a414cff0 2023-08-25T09:50:42.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.221+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f328c013470 con 0x7f32a414cff0 2023-08-25T09:50:42.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.222+0000 7f329b7fe640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f328406a050 0x7f328406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.222+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f328c0889d0 con 0x7f32a414cff0 2023-08-25T09:50:42.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.222+0000 7f32a2d9d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f328406a050 0x7f328406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.223+0000 7f32a2d9d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f328406a050 0x7f328406c510 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f32940015b0 tx=0x7f3294009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:42.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.226+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f328c053d00 con 0x7f32a414cff0 2023-08-25T09:50:42.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.413+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32a41edc70 con 0x7f328406a050 2023-08-25T09:50:42.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.425+0000 7f329b7fe640 1 -- 172.21.15.71:0/2218461659 <== mgr.4107 v2:172.21.15.71:6832/786313351 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+389523 (secure 0 0 0) 0x7f32a41edc70 con 0x7f328406a050 2023-08-25T09:50:42.431 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:42.432 INFO:teuthology.orchestra.run.smithi071.stderr:dumped all 2023-08-25T09:50:42.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f328406a050 msgr2=0x7f328406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f328406a050 0x7f328406c510 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f32940015b0 tx=0x7f3294009040 comp rx=0 tx=0).stop 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 msgr2=0x7f32a4154ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 0x7f32a4154ce0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f328c00e7c0 tx=0x7f328c00ec80 comp rx=0 tx=0).stop 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 shutdown_connections 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f328406a050 0x7f328406c510 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32a4155220 0x7f32a4155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32a414cff0 0x7f32a4154ce0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 --2- 172.21.15.71:0/2218461659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3290005180 0x7f32a4147260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.431+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 >> 172.21.15.71:0/2218461659 conn(0x7f32a406ae20 msgr2=0x7f32a415afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:42.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.432+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 shutdown_connections 2023-08-25T09:50:42.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.432+0000 7f32a888d640 1 -- 172.21.15.71:0/2218461659 wait complete. 2023-08-25T09:50:42.453 INFO:teuthology.orchestra.run.smithi071.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-08-25T09:50:40.660103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602391,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":376,"num_write_kb":2491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":308,"ondisk_log_size":308,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227424,"kb_used_data":2976,"kb_used_omap":13,"kb_used_meta":224370,"kb_avail":754747296,"statfs":{"total":773094113280,"available":772861231104,"internally_reserved":0,"allocated":3047424,"data_stored":2328822,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":14248,"internal_metadata":229754968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1760,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004736"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629137+0000","last_change":"2023-08-25T09:49:38.677254+0000","last_active":"2023-08-25T09:50:12.629137+0000","last_peered":"2023-08-25T09:50:12.629137+0000","last_clean":"2023-08-25T09:50:12.629137+0000","last_became_active":"2023-08-25T09:49:38.676839+0000","last_became_peered":"2023-08-25T09:49:38.676839+0000","last_unstale":"2023-08-25T09:50:12.629137+0000","last_undegraded":"2023-08-25T09:50:12.629137+0000","last_fullsized":"2023-08-25T09:50:12.629137+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:56:23.094991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"31'13","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.758090+0000","last_change":"2023-08-25T09:49:33.608012+0000","last_active":"2023-08-25T09:50:22.758090+0000","last_peered":"2023-08-25T09:50:22.758090+0000","last_clean":"2023-08-25T09:50:22.758090+0000","last_became_active":"2023-08-25T09:49:33.607737+0000","last_became_peered":"2023-08-25T09:49:33.607737+0000","last_unstale":"2023-08-25T09:50:22.758090+0000","last_undegraded":"2023-08-25T09:50:22.758090+0000","last_fullsized":"2023-08-25T09:50:22.758090+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:58:03.201648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554648+0000","last_change":"2023-08-25T09:49:34.618889+0000","last_active":"2023-08-25T09:50:12.554648+0000","last_peered":"2023-08-25T09:50:12.554648+0000","last_clean":"2023-08-25T09:50:12.554648+0000","last_became_active":"2023-08-25T09:49:34.618570+0000","last_became_peered":"2023-08-25T09:49:34.618570+0000","last_unstale":"2023-08-25T09:50:12.554648+0000","last_undegraded":"2023-08-25T09:50:12.554648+0000","last_fullsized":"2023-08-25T09:50:12.554648+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:24:26.126307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554093+0000","last_change":"2023-08-25T09:49:38.670003+0000","last_active":"2023-08-25T09:50:12.554093+0000","last_peered":"2023-08-25T09:50:12.554093+0000","last_clean":"2023-08-25T09:50:12.554093+0000","last_became_active":"2023-08-25T09:49:38.669431+0000","last_became_peered":"2023-08-25T09:49:38.669431+0000","last_unstale":"2023-08-25T09:50:12.554093+0000","last_undegraded":"2023-08-25T09:50:12.554093+0000","last_fullsized":"2023-08-25T09:50:12.554093+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:02:22.526110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"29'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627113+0000","last_change":"2023-08-25T09:49:33.611687+0000","last_active":"2023-08-25T09:50:12.627113+0000","last_peered":"2023-08-25T09:50:12.627113+0000","last_clean":"2023-08-25T09:50:12.627113+0000","last_became_active":"2023-08-25T09:49:33.611353+0000","last_became_peered":"2023-08-25T09:49:33.611353+0000","last_unstale":"2023-08-25T09:50:12.627113+0000","last_undegraded":"2023-08-25T09:50:12.627113+0000","last_fullsized":"2023-08-25T09:50:12.627113+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:44:00.313382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626363+0000","last_change":"2023-08-25T09:49:34.609052+0000","last_active":"2023-08-25T09:50:12.626363+0000","last_peered":"2023-08-25T09:50:12.626363+0000","last_clean":"2023-08-25T09:50:12.626363+0000","last_became_active":"2023-08-25T09:49:34.608549+0000","last_became_peered":"2023-08-25T09:49:34.608549+0000","last_unstale":"2023-08-25T09:50:12.626363+0000","last_undegraded":"2023-08-25T09:50:12.626363+0000","last_fullsized":"2023-08-25T09:50:12.626363+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:06:14.787284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556659+0000","last_change":"2023-08-25T09:49:38.683469+0000","last_active":"2023-08-25T09:50:12.556659+0000","last_peered":"2023-08-25T09:50:12.556659+0000","last_clean":"2023-08-25T09:50:12.556659+0000","last_became_active":"2023-08-25T09:49:38.683100+0000","last_became_peered":"2023-08-25T09:49:38.683100+0000","last_unstale":"2023-08-25T09:50:12.556659+0000","last_undegraded":"2023-08-25T09:50:12.556659+0000","last_fullsized":"2023-08-25T09:50:12.556659+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:45:30.828332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761816+0000","last_change":"2023-08-25T09:49:33.615442+0000","last_active":"2023-08-25T09:50:22.761816+0000","last_peered":"2023-08-25T09:50:22.761816+0000","last_clean":"2023-08-25T09:50:22.761816+0000","last_became_active":"2023-08-25T09:49:33.615148+0000","last_became_peered":"2023-08-25T09:49:33.615148+0000","last_unstale":"2023-08-25T09:50:22.761816+0000","last_undegraded":"2023-08-25T09:50:22.761816+0000","last_fullsized":"2023-08-25T09:50:22.761816+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:15:33.280852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558309+0000","last_change":"2023-08-25T09:49:34.607640+0000","last_active":"2023-08-25T09:50:12.558309+0000","last_peered":"2023-08-25T09:50:12.558309+0000","last_clean":"2023-08-25T09:50:12.558309+0000","last_became_active":"2023-08-25T09:49:34.607331+0000","last_became_peered":"2023-08-25T09:49:34.607331+0000","last_unstale":"2023-08-25T09:50:12.558309+0000","last_undegraded":"2023-08-25T09:50:12.558309+0000","last_fullsized":"2023-08-25T09:50:12.558309+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:20:45.240925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626253+0000","last_change":"2023-08-25T09:49:38.689276+0000","last_active":"2023-08-25T09:50:12.626253+0000","last_peered":"2023-08-25T09:50:12.626253+0000","last_clean":"2023-08-25T09:50:12.626253+0000","last_became_active":"2023-08-25T09:49:38.688852+0000","last_became_peered":"2023-08-25T09:49:38.688852+0000","last_unstale":"2023-08-25T09:50:12.626253+0000","last_undegraded":"2023-08-25T09:50:12.626253+0000","last_fullsized":"2023-08-25T09:50:12.626253+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:16:41.085653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761058+0000","last_change":"2023-08-25T09:49:33.615410+0000","last_active":"2023-08-25T09:50:22.761058+0000","last_peered":"2023-08-25T09:50:22.761058+0000","last_clean":"2023-08-25T09:50:22.761058+0000","last_became_active":"2023-08-25T09:49:33.614807+0000","last_became_peered":"2023-08-25T09:49:33.614807+0000","last_unstale":"2023-08-25T09:50:22.761058+0000","last_undegraded":"2023-08-25T09:50:22.761058+0000","last_fullsized":"2023-08-25T09:50:22.761058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:10:43.593031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556912+0000","last_change":"2023-08-25T09:49:34.609115+0000","last_active":"2023-08-25T09:50:12.556912+0000","last_peered":"2023-08-25T09:50:12.556912+0000","last_clean":"2023-08-25T09:50:12.556912+0000","last_became_active":"2023-08-25T09:49:34.608626+0000","last_became_peered":"2023-08-25T09:49:34.608626+0000","last_unstale":"2023-08-25T09:50:12.556912+0000","last_undegraded":"2023-08-25T09:50:12.556912+0000","last_fullsized":"2023-08-25T09:50:12.556912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:41:19.475348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628183+0000","last_change":"2023-08-25T09:49:38.682868+0000","last_active":"2023-08-25T09:50:12.628183+0000","last_peered":"2023-08-25T09:50:12.628183+0000","last_clean":"2023-08-25T09:50:12.628183+0000","last_became_active":"2023-08-25T09:49:38.682470+0000","last_became_peered":"2023-08-25T09:49:38.682470+0000","last_unstale":"2023-08-25T09:50:12.628183+0000","last_undegraded":"2023-08-25T09:50:12.628183+0000","last_fullsized":"2023-08-25T09:50:12.628183+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:55:22.425822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552143+0000","last_change":"2023-08-25T09:49:33.614270+0000","last_active":"2023-08-25T09:50:12.552143+0000","last_peered":"2023-08-25T09:50:12.552143+0000","last_clean":"2023-08-25T09:50:12.552143+0000","last_became_active":"2023-08-25T09:49:33.613952+0000","last_became_peered":"2023-08-25T09:49:33.613952+0000","last_unstale":"2023-08-25T09:50:12.552143+0000","last_undegraded":"2023-08-25T09:50:12.552143+0000","last_fullsized":"2023-08-25T09:50:12.552143+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T09:52:56.784267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1062,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556859+0000","last_change":"2023-08-25T09:49:34.625525+0000","last_active":"2023-08-25T09:50:12.556859+0000","last_peered":"2023-08-25T09:50:12.556859+0000","last_clean":"2023-08-25T09:50:12.556859+0000","last_became_active":"2023-08-25T09:49:34.625047+0000","last_became_peered":"2023-08-25T09:49:34.625047+0000","last_unstale":"2023-08-25T09:50:12.556859+0000","last_undegraded":"2023-08-25T09:50:12.556859+0000","last_fullsized":"2023-08-25T09:50:12.556859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:29:29.962318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557828+0000","last_change":"2023-08-25T09:49:38.674303+0000","last_active":"2023-08-25T09:50:12.557828+0000","last_peered":"2023-08-25T09:50:12.557828+0000","last_clean":"2023-08-25T09:50:12.557828+0000","last_became_active":"2023-08-25T09:49:38.673900+0000","last_became_peered":"2023-08-25T09:49:38.673900+0000","last_unstale":"2023-08-25T09:50:12.557828+0000","last_undegraded":"2023-08-25T09:50:12.557828+0000","last_fullsized":"2023-08-25T09:50:12.557828+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:24:46.007193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.575799+0000","last_change":"2023-08-25T09:49:33.611940+0000","last_active":"2023-08-25T09:50:17.575799+0000","last_peered":"2023-08-25T09:50:17.575799+0000","last_clean":"2023-08-25T09:50:17.575799+0000","last_became_active":"2023-08-25T09:49:33.611325+0000","last_became_peered":"2023-08-25T09:49:33.611325+0000","last_unstale":"2023-08-25T09:50:17.575799+0000","last_undegraded":"2023-08-25T09:50:17.575799+0000","last_fullsized":"2023-08-25T09:50:17.575799+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:23:50.467500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628494+0000","last_change":"2023-08-25T09:49:34.624815+0000","last_active":"2023-08-25T09:50:12.628494+0000","last_peered":"2023-08-25T09:50:12.628494+0000","last_clean":"2023-08-25T09:50:12.628494+0000","last_became_active":"2023-08-25T09:49:34.624520+0000","last_became_peered":"2023-08-25T09:49:34.624520+0000","last_unstale":"2023-08-25T09:50:12.628494+0000","last_undegraded":"2023-08-25T09:50:12.628494+0000","last_fullsized":"2023-08-25T09:50:12.628494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:28:21.130424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566445+0000","last_change":"2023-08-25T09:49:38.682365+0000","last_active":"2023-08-25T09:50:12.566445+0000","last_peered":"2023-08-25T09:50:12.566445+0000","last_clean":"2023-08-25T09:50:12.566445+0000","last_became_active":"2023-08-25T09:49:38.678323+0000","last_became_peered":"2023-08-25T09:49:38.678323+0000","last_unstale":"2023-08-25T09:50:12.566445+0000","last_undegraded":"2023-08-25T09:50:12.566445+0000","last_fullsized":"2023-08-25T09:50:12.566445+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:16:24.203781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630137+0000","last_change":"2023-08-25T09:49:33.610138+0000","last_active":"2023-08-25T09:50:12.630137+0000","last_peered":"2023-08-25T09:50:12.630137+0000","last_clean":"2023-08-25T09:50:12.630137+0000","last_became_active":"2023-08-25T09:49:33.609801+0000","last_became_peered":"2023-08-25T09:49:33.609801+0000","last_unstale":"2023-08-25T09:50:12.630137+0000","last_undegraded":"2023-08-25T09:50:12.630137+0000","last_fullsized":"2023-08-25T09:50:12.630137+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:11:08.732481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551378+0000","last_change":"2023-08-25T09:49:34.621009+0000","last_active":"2023-08-25T09:50:12.551378+0000","last_peered":"2023-08-25T09:50:12.551378+0000","last_clean":"2023-08-25T09:50:12.551378+0000","last_became_active":"2023-08-25T09:49:34.620677+0000","last_became_peered":"2023-08-25T09:49:34.620677+0000","last_unstale":"2023-08-25T09:50:12.551378+0000","last_undegraded":"2023-08-25T09:50:12.551378+0000","last_fullsized":"2023-08-25T09:50:12.551378+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:12:56.940938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552046+0000","last_change":"2023-08-25T09:49:38.672877+0000","last_active":"2023-08-25T09:50:12.552046+0000","last_peered":"2023-08-25T09:50:12.552046+0000","last_clean":"2023-08-25T09:50:12.552046+0000","last_became_active":"2023-08-25T09:49:38.672500+0000","last_became_peered":"2023-08-25T09:49:38.672500+0000","last_unstale":"2023-08-25T09:50:12.552046+0000","last_undegraded":"2023-08-25T09:50:12.552046+0000","last_fullsized":"2023-08-25T09:50:12.552046+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:30:58.600767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573085+0000","last_change":"2023-08-25T09:49:33.611580+0000","last_active":"2023-08-25T09:50:17.573085+0000","last_peered":"2023-08-25T09:50:17.573085+0000","last_clean":"2023-08-25T09:50:17.573085+0000","last_became_active":"2023-08-25T09:49:33.608371+0000","last_became_peered":"2023-08-25T09:49:33.608371+0000","last_unstale":"2023-08-25T09:50:17.573085+0000","last_undegraded":"2023-08-25T09:50:17.573085+0000","last_fullsized":"2023-08-25T09:50:17.573085+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:44:06.900671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566581+0000","last_change":"2023-08-25T09:49:34.616417+0000","last_active":"2023-08-25T09:50:12.566581+0000","last_peered":"2023-08-25T09:50:12.566581+0000","last_clean":"2023-08-25T09:50:12.566581+0000","last_became_active":"2023-08-25T09:49:34.616123+0000","last_became_peered":"2023-08-25T09:49:34.616123+0000","last_unstale":"2023-08-25T09:50:12.566581+0000","last_undegraded":"2023-08-25T09:50:12.566581+0000","last_fullsized":"2023-08-25T09:50:12.566581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:51:56.697715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625495+0000","last_change":"2023-08-25T09:49:34.630503+0000","last_active":"2023-08-25T09:50:12.625495+0000","last_peered":"2023-08-25T09:50:12.625495+0000","last_clean":"2023-08-25T09:50:12.625495+0000","last_became_active":"2023-08-25T09:49:34.630217+0000","last_became_peered":"2023-08-25T09:49:34.630217+0000","last_unstale":"2023-08-25T09:50:12.625495+0000","last_undegraded":"2023-08-25T09:50:12.625495+0000","last_fullsized":"2023-08-25T09:50:12.625495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:23:54.271157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:33.134863+0000","last_change":"2023-08-25T09:49:33.602273+0000","last_active":"2023-08-25T09:50:33.134863+0000","last_peered":"2023-08-25T09:50:33.134863+0000","last_clean":"2023-08-25T09:50:33.134863+0000","last_became_active":"2023-08-25T09:49:33.601882+0000","last_became_peered":"2023-08-25T09:49:33.601882+0000","last_unstale":"2023-08-25T09:50:33.134863+0000","last_undegraded":"2023-08-25T09:50:33.134863+0000","last_fullsized":"2023-08-25T09:50:33.134863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:19:35.376472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565155+0000","last_change":"2023-08-25T09:49:38.681823+0000","last_active":"2023-08-25T09:50:12.565155+0000","last_peered":"2023-08-25T09:50:12.565155+0000","last_clean":"2023-08-25T09:50:12.565155+0000","last_became_active":"2023-08-25T09:49:38.678352+0000","last_became_peered":"2023-08-25T09:49:38.678352+0000","last_unstale":"2023-08-25T09:50:12.565155+0000","last_undegraded":"2023-08-25T09:50:12.565155+0000","last_fullsized":"2023-08-25T09:50:12.565155+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:16:54.785190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626109+0000","last_change":"2023-08-25T09:49:34.624156+0000","last_active":"2023-08-25T09:50:12.626109+0000","last_peered":"2023-08-25T09:50:12.626109+0000","last_clean":"2023-08-25T09:50:12.626109+0000","last_became_active":"2023-08-25T09:49:34.622883+0000","last_became_peered":"2023-08-25T09:49:34.622883+0000","last_unstale":"2023-08-25T09:50:12.626109+0000","last_undegraded":"2023-08-25T09:50:12.626109+0000","last_fullsized":"2023-08-25T09:50:12.626109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:00:08.343712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.759325+0000","last_change":"2023-08-25T09:49:33.613210+0000","last_active":"2023-08-25T09:50:22.759325+0000","last_peered":"2023-08-25T09:50:22.759325+0000","last_clean":"2023-08-25T09:50:22.759325+0000","last_became_active":"2023-08-25T09:49:33.606448+0000","last_became_peered":"2023-08-25T09:49:33.606448+0000","last_unstale":"2023-08-25T09:50:22.759325+0000","last_undegraded":"2023-08-25T09:50:22.759325+0000","last_fullsized":"2023-08-25T09:50:22.759325+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:38:03.473951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625586+0000","last_change":"2023-08-25T09:49:38.684022+0000","last_active":"2023-08-25T09:50:12.625586+0000","last_peered":"2023-08-25T09:50:12.625586+0000","last_clean":"2023-08-25T09:50:12.625586+0000","last_became_active":"2023-08-25T09:49:38.683599+0000","last_became_peered":"2023-08-25T09:49:38.683599+0000","last_unstale":"2023-08-25T09:50:12.625586+0000","last_undegraded":"2023-08-25T09:50:12.625586+0000","last_fullsized":"2023-08-25T09:50:12.625586+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:03:07.147111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626969+0000","last_change":"2023-08-25T09:49:33.614274+0000","last_active":"2023-08-25T09:50:12.626969+0000","last_peered":"2023-08-25T09:50:12.626969+0000","last_clean":"2023-08-25T09:50:12.626969+0000","last_became_active":"2023-08-25T09:49:33.614010+0000","last_became_peered":"2023-08-25T09:49:33.614010+0000","last_unstale":"2023-08-25T09:50:12.626969+0000","last_undegraded":"2023-08-25T09:50:12.626969+0000","last_fullsized":"2023-08-25T09:50:12.626969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:14:08.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630451+0000","last_change":"2023-08-25T09:49:34.619960+0000","last_active":"2023-08-25T09:50:12.630451+0000","last_peered":"2023-08-25T09:50:12.630451+0000","last_clean":"2023-08-25T09:50:12.630451+0000","last_became_active":"2023-08-25T09:49:34.619651+0000","last_became_peered":"2023-08-25T09:49:34.619651+0000","last_unstale":"2023-08-25T09:50:12.630451+0000","last_undegraded":"2023-08-25T09:50:12.630451+0000","last_fullsized":"2023-08-25T09:50:12.630451+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:01:33.100204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626271+0000","last_change":"2023-08-25T09:49:38.671601+0000","last_active":"2023-08-25T09:50:12.626271+0000","last_peered":"2023-08-25T09:50:12.626271+0000","last_clean":"2023-08-25T09:50:12.626271+0000","last_became_active":"2023-08-25T09:49:38.671163+0000","last_became_peered":"2023-08-25T09:49:38.671163+0000","last_unstale":"2023-08-25T09:50:12.626271+0000","last_undegraded":"2023-08-25T09:50:12.626271+0000","last_fullsized":"2023-08-25T09:50:12.626271+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:31:59.929747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"31'16","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:33.155598+0000","last_change":"2023-08-25T09:49:33.606462+0000","last_active":"2023-08-25T09:50:33.155598+0000","last_peered":"2023-08-25T09:50:33.155598+0000","last_clean":"2023-08-25T09:50:33.155598+0000","last_became_active":"2023-08-25T09:49:33.606123+0000","last_became_peered":"2023-08-25T09:49:33.606123+0000","last_unstale":"2023-08-25T09:50:33.155598+0000","last_undegraded":"2023-08-25T09:50:33.155598+0000","last_fullsized":"2023-08-25T09:50:33.155598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:59:56.712100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4281,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557216+0000","last_change":"2023-08-25T09:49:34.617923+0000","last_active":"2023-08-25T09:50:12.557216+0000","last_peered":"2023-08-25T09:50:12.557216+0000","last_clean":"2023-08-25T09:50:12.557216+0000","last_became_active":"2023-08-25T09:49:34.617551+0000","last_became_peered":"2023-08-25T09:49:34.617551+0000","last_unstale":"2023-08-25T09:50:12.557216+0000","last_undegraded":"2023-08-25T09:50:12.557216+0000","last_fullsized":"2023-08-25T09:50:12.557216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:24:19.739443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629763+0000","last_change":"2023-08-25T09:49:38.677306+0000","last_active":"2023-08-25T09:50:12.629763+0000","last_peered":"2023-08-25T09:50:12.629763+0000","last_clean":"2023-08-25T09:50:12.629763+0000","last_became_active":"2023-08-25T09:49:38.676927+0000","last_became_peered":"2023-08-25T09:49:38.676927+0000","last_unstale":"2023-08-25T09:50:12.629763+0000","last_undegraded":"2023-08-25T09:50:12.629763+0000","last_fullsized":"2023-08-25T09:50:12.629763+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:08:23.975702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555540+0000","last_change":"2023-08-25T09:49:34.603439+0000","last_active":"2023-08-25T09:50:12.555540+0000","last_peered":"2023-08-25T09:50:12.555540+0000","last_clean":"2023-08-25T09:50:12.555540+0000","last_became_active":"2023-08-25T09:49:34.602130+0000","last_became_peered":"2023-08-25T09:49:34.602130+0000","last_unstale":"2023-08-25T09:50:12.555540+0000","last_undegraded":"2023-08-25T09:50:12.555540+0000","last_fullsized":"2023-08-25T09:50:12.555540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:23:24.270948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566757+0000","last_change":"2023-08-25T09:49:27.537419+0000","last_active":"2023-08-25T09:50:12.566757+0000","last_peered":"2023-08-25T09:50:12.566757+0000","last_clean":"2023-08-25T09:50:12.566757+0000","last_became_active":"2023-08-25T09:49:27.536593+0000","last_became_peered":"2023-08-25T09:49:27.536593+0000","last_unstale":"2023-08-25T09:50:12.566757+0000","last_undegraded":"2023-08-25T09:50:12.566757+0000","last_fullsized":"2023-08-25T09:50:12.566757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:18:10.046784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.576095+0000","last_change":"2023-08-25T09:49:33.605352+0000","last_active":"2023-08-25T09:50:17.576095+0000","last_peered":"2023-08-25T09:50:17.576095+0000","last_clean":"2023-08-25T09:50:17.576095+0000","last_became_active":"2023-08-25T09:49:33.605027+0000","last_became_peered":"2023-08-25T09:49:33.605027+0000","last_unstale":"2023-08-25T09:50:17.576095+0000","last_undegraded":"2023-08-25T09:50:17.576095+0000","last_fullsized":"2023-08-25T09:50:17.576095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:12:06.214562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625440+0000","last_change":"2023-08-25T09:49:38.670076+0000","last_active":"2023-08-25T09:50:12.625440+0000","last_peered":"2023-08-25T09:50:12.625440+0000","last_clean":"2023-08-25T09:50:12.625440+0000","last_became_active":"2023-08-25T09:49:38.669634+0000","last_became_peered":"2023-08-25T09:49:38.669634+0000","last_unstale":"2023-08-25T09:50:12.625440+0000","last_undegraded":"2023-08-25T09:50:12.625440+0000","last_fullsized":"2023-08-25T09:50:12.625440+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:40:36.040907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625747+0000","last_change":"2023-08-25T09:49:34.624231+0000","last_active":"2023-08-25T09:50:12.625747+0000","last_peered":"2023-08-25T09:50:12.625747+0000","last_clean":"2023-08-25T09:50:12.625747+0000","last_became_active":"2023-08-25T09:49:34.623105+0000","last_became_peered":"2023-08-25T09:49:34.623105+0000","last_unstale":"2023-08-25T09:50:12.625747+0000","last_undegraded":"2023-08-25T09:50:12.625747+0000","last_fullsized":"2023-08-25T09:50:12.625747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:39:48.184915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630298+0000","last_change":"2023-08-25T09:49:27.534603+0000","last_active":"2023-08-25T09:50:12.630298+0000","last_peered":"2023-08-25T09:50:12.630298+0000","last_clean":"2023-08-25T09:50:12.630298+0000","last_became_active":"2023-08-25T09:49:27.534058+0000","last_became_peered":"2023-08-25T09:49:27.534058+0000","last_unstale":"2023-08-25T09:50:12.630298+0000","last_undegraded":"2023-08-25T09:50:12.630298+0000","last_fullsized":"2023-08-25T09:50:12.630298+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:45:35.013224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.759471+0000","last_change":"2023-08-25T09:49:33.612851+0000","last_active":"2023-08-25T09:50:22.759471+0000","last_peered":"2023-08-25T09:50:22.759471+0000","last_clean":"2023-08-25T09:50:22.759471+0000","last_became_active":"2023-08-25T09:49:33.612419+0000","last_became_peered":"2023-08-25T09:49:33.612419+0000","last_unstale":"2023-08-25T09:50:22.759471+0000","last_undegraded":"2023-08-25T09:50:22.759471+0000","last_fullsized":"2023-08-25T09:50:22.759471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:55:21.711434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554400+0000","last_change":"2023-08-25T09:49:38.684060+0000","last_active":"2023-08-25T09:50:12.554400+0000","last_peered":"2023-08-25T09:50:12.554400+0000","last_clean":"2023-08-25T09:50:12.554400+0000","last_became_active":"2023-08-25T09:49:38.683247+0000","last_became_peered":"2023-08-25T09:49:38.683247+0000","last_unstale":"2023-08-25T09:50:12.554400+0000","last_undegraded":"2023-08-25T09:50:12.554400+0000","last_fullsized":"2023-08-25T09:50:12.554400+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:23:36.823930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551745+0000","last_change":"2023-08-25T09:49:34.622428+0000","last_active":"2023-08-25T09:50:12.551745+0000","last_peered":"2023-08-25T09:50:12.551745+0000","last_clean":"2023-08-25T09:50:12.551745+0000","last_became_active":"2023-08-25T09:49:34.622148+0000","last_became_peered":"2023-08-25T09:49:34.622148+0000","last_unstale":"2023-08-25T09:50:12.551745+0000","last_undegraded":"2023-08-25T09:50:12.551745+0000","last_fullsized":"2023-08-25T09:50:12.551745+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:31:55.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627612+0000","last_change":"2023-08-25T09:49:27.536027+0000","last_active":"2023-08-25T09:50:12.627612+0000","last_peered":"2023-08-25T09:50:12.627612+0000","last_clean":"2023-08-25T09:50:12.627612+0000","last_became_active":"2023-08-25T09:49:27.535351+0000","last_became_peered":"2023-08-25T09:49:27.535351+0000","last_unstale":"2023-08-25T09:50:12.627612+0000","last_undegraded":"2023-08-25T09:50:12.627612+0000","last_fullsized":"2023-08-25T09:50:12.627612+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:44:34.927600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"31'1","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628052+0000","last_change":"2023-08-25T09:49:33.604458+0000","last_active":"2023-08-25T09:50:12.628052+0000","last_peered":"2023-08-25T09:50:12.628052+0000","last_clean":"2023-08-25T09:50:12.628052+0000","last_became_active":"2023-08-25T09:49:33.604094+0000","last_became_peered":"2023-08-25T09:49:33.604094+0000","last_unstale":"2023-08-25T09:50:12.628052+0000","last_undegraded":"2023-08-25T09:50:12.628052+0000","last_fullsized":"2023-08-25T09:50:12.628052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:05:32.332141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565443+0000","last_change":"2023-08-25T09:49:38.682339+0000","last_active":"2023-08-25T09:50:12.565443+0000","last_peered":"2023-08-25T09:50:12.565443+0000","last_clean":"2023-08-25T09:50:12.565443+0000","last_became_active":"2023-08-25T09:49:38.680504+0000","last_became_peered":"2023-08-25T09:49:38.680504+0000","last_unstale":"2023-08-25T09:50:12.565443+0000","last_undegraded":"2023-08-25T09:50:12.565443+0000","last_fullsized":"2023-08-25T09:50:12.565443+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:15:47.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630669+0000","last_change":"2023-08-25T09:49:34.626211+0000","last_active":"2023-08-25T09:50:12.630669+0000","last_peered":"2023-08-25T09:50:12.630669+0000","last_clean":"2023-08-25T09:50:12.630669+0000","last_became_active":"2023-08-25T09:49:34.625938+0000","last_became_peered":"2023-08-25T09:49:34.625938+0000","last_unstale":"2023-08-25T09:50:12.630669+0000","last_undegraded":"2023-08-25T09:50:12.630669+0000","last_fullsized":"2023-08-25T09:50:12.630669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:18:08.404806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628723+0000","last_change":"2023-08-25T09:49:27.527437+0000","last_active":"2023-08-25T09:50:12.628723+0000","last_peered":"2023-08-25T09:50:12.628723+0000","last_clean":"2023-08-25T09:50:12.628723+0000","last_became_active":"2023-08-25T09:49:27.527003+0000","last_became_peered":"2023-08-25T09:49:27.527003+0000","last_unstale":"2023-08-25T09:50:12.628723+0000","last_undegraded":"2023-08-25T09:50:12.628723+0000","last_fullsized":"2023-08-25T09:50:12.628723+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:17:27.827364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.580091+0000","last_change":"2023-08-25T09:49:33.614084+0000","last_active":"2023-08-25T09:50:17.580091+0000","last_peered":"2023-08-25T09:50:17.580091+0000","last_clean":"2023-08-25T09:50:17.580091+0000","last_became_active":"2023-08-25T09:49:33.613723+0000","last_became_peered":"2023-08-25T09:49:33.613723+0000","last_unstale":"2023-08-25T09:50:17.580091+0000","last_undegraded":"2023-08-25T09:50:17.580091+0000","last_fullsized":"2023-08-25T09:50:17.580091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:56:04.945533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551491+0000","last_change":"2023-08-25T09:49:38.675033+0000","last_active":"2023-08-25T09:50:12.551491+0000","last_peered":"2023-08-25T09:50:12.551491+0000","last_clean":"2023-08-25T09:50:12.551491+0000","last_became_active":"2023-08-25T09:49:38.674586+0000","last_became_peered":"2023-08-25T09:49:38.674586+0000","last_unstale":"2023-08-25T09:50:12.551491+0000","last_undegraded":"2023-08-25T09:50:12.551491+0000","last_fullsized":"2023-08-25T09:50:12.551491+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:34:52.676480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627952+0000","last_change":"2023-08-25T09:49:34.622579+0000","last_active":"2023-08-25T09:50:12.627952+0000","last_peered":"2023-08-25T09:50:12.627952+0000","last_clean":"2023-08-25T09:50:12.627952+0000","last_became_active":"2023-08-25T09:49:34.621877+0000","last_became_peered":"2023-08-25T09:49:34.621877+0000","last_unstale":"2023-08-25T09:50:12.627952+0000","last_undegraded":"2023-08-25T09:50:12.627952+0000","last_fullsized":"2023-08-25T09:50:12.627952+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:42:22.416093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573338+0000","last_change":"2023-08-25T09:49:33.606990+0000","last_active":"2023-08-25T09:50:17.573338+0000","last_peered":"2023-08-25T09:50:17.573338+0000","last_clean":"2023-08-25T09:50:17.573338+0000","last_became_active":"2023-08-25T09:49:33.606664+0000","last_became_peered":"2023-08-25T09:49:33.606664+0000","last_unstale":"2023-08-25T09:50:17.573338+0000","last_undegraded":"2023-08-25T09:50:17.573338+0000","last_fullsized":"2023-08-25T09:50:17.573338+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:49:49.710740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556160+0000","last_change":"2023-08-25T09:49:27.528477+0000","last_active":"2023-08-25T09:50:12.556160+0000","last_peered":"2023-08-25T09:50:12.556160+0000","last_clean":"2023-08-25T09:50:12.556160+0000","last_became_active":"2023-08-25T09:49:27.526572+0000","last_became_peered":"2023-08-25T09:49:27.526572+0000","last_unstale":"2023-08-25T09:50:12.556160+0000","last_undegraded":"2023-08-25T09:50:12.556160+0000","last_fullsized":"2023-08-25T09:50:12.556160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:37:34.254249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551025+0000","last_change":"2023-08-25T09:49:38.675566+0000","last_active":"2023-08-25T09:50:12.551025+0000","last_peered":"2023-08-25T09:50:12.551025+0000","last_clean":"2023-08-25T09:50:12.551025+0000","last_became_active":"2023-08-25T09:49:38.673982+0000","last_became_peered":"2023-08-25T09:49:38.673982+0000","last_unstale":"2023-08-25T09:50:12.551025+0000","last_undegraded":"2023-08-25T09:50:12.551025+0000","last_fullsized":"2023-08-25T09:50:12.551025+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:38:55.415402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629657+0000","last_change":"2023-08-25T09:49:34.621176+0000","last_active":"2023-08-25T09:50:12.629657+0000","last_peered":"2023-08-25T09:50:12.629657+0000","last_clean":"2023-08-25T09:50:12.629657+0000","last_became_active":"2023-08-25T09:49:34.620190+0000","last_became_peered":"2023-08-25T09:49:34.620190+0000","last_unstale":"2023-08-25T09:50:12.629657+0000","last_undegraded":"2023-08-25T09:50:12.629657+0000","last_fullsized":"2023-08-25T09:50:12.629657+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:50:25.245004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627351+0000","last_change":"2023-08-25T09:49:33.602276+0000","last_active":"2023-08-25T09:50:12.627351+0000","last_peered":"2023-08-25T09:50:12.627351+0000","last_clean":"2023-08-25T09:50:12.627351+0000","last_became_active":"2023-08-25T09:49:33.601897+0000","last_became_peered":"2023-08-25T09:49:33.601897+0000","last_unstale":"2023-08-25T09:50:12.627351+0000","last_undegraded":"2023-08-25T09:50:12.627351+0000","last_fullsized":"2023-08-25T09:50:12.627351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:41:28.016611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557166+0000","last_change":"2023-08-25T09:49:27.684749+0000","last_active":"2023-08-25T09:50:12.557166+0000","last_peered":"2023-08-25T09:50:12.557166+0000","last_clean":"2023-08-25T09:50:12.557166+0000","last_became_active":"2023-08-25T09:49:27.684074+0000","last_became_peered":"2023-08-25T09:49:27.684074+0000","last_unstale":"2023-08-25T09:50:12.557166+0000","last_undegraded":"2023-08-25T09:50:12.557166+0000","last_fullsized":"2023-08-25T09:50:12.557166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:09:13.822549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557109+0000","last_change":"2023-08-25T09:49:38.677626+0000","last_active":"2023-08-25T09:50:12.557109+0000","last_peered":"2023-08-25T09:50:12.557109+0000","last_clean":"2023-08-25T09:50:12.557109+0000","last_became_active":"2023-08-25T09:49:38.677186+0000","last_became_peered":"2023-08-25T09:49:38.677186+0000","last_unstale":"2023-08-25T09:50:12.557109+0000","last_undegraded":"2023-08-25T09:50:12.557109+0000","last_fullsized":"2023-08-25T09:50:12.557109+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:15:47.257428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551738+0000","last_change":"2023-08-25T09:49:34.622361+0000","last_active":"2023-08-25T09:50:12.551738+0000","last_peered":"2023-08-25T09:50:12.551738+0000","last_clean":"2023-08-25T09:50:12.551738+0000","last_became_active":"2023-08-25T09:49:34.621880+0000","last_became_peered":"2023-08-25T09:49:34.621880+0000","last_unstale":"2023-08-25T09:50:12.551738+0000","last_undegraded":"2023-08-25T09:50:12.551738+0000","last_fullsized":"2023-08-25T09:50:12.551738+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:22:38.020973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761598+0000","last_change":"2023-08-25T09:49:33.614511+0000","last_active":"2023-08-25T09:50:22.761598+0000","last_peered":"2023-08-25T09:50:22.761598+0000","last_clean":"2023-08-25T09:50:22.761598+0000","last_became_active":"2023-08-25T09:49:33.613473+0000","last_became_peered":"2023-08-25T09:49:33.613473+0000","last_unstale":"2023-08-25T09:50:22.761598+0000","last_undegraded":"2023-08-25T09:50:22.761598+0000","last_fullsized":"2023-08-25T09:50:22.761598+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:02:36.889545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626347+0000","last_change":"2023-08-25T09:49:27.533106+0000","last_active":"2023-08-25T09:50:12.626347+0000","last_peered":"2023-08-25T09:50:12.626347+0000","last_clean":"2023-08-25T09:50:12.626347+0000","last_became_active":"2023-08-25T09:49:27.532238+0000","last_became_peered":"2023-08-25T09:49:27.532238+0000","last_unstale":"2023-08-25T09:50:12.626347+0000","last_undegraded":"2023-08-25T09:50:12.626347+0000","last_fullsized":"2023-08-25T09:50:12.626347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:52:59.975913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558258+0000","last_change":"2023-08-25T09:49:38.682190+0000","last_active":"2023-08-25T09:50:12.558258+0000","last_peered":"2023-08-25T09:50:12.558258+0000","last_clean":"2023-08-25T09:50:12.558258+0000","last_became_active":"2023-08-25T09:49:38.679317+0000","last_became_peered":"2023-08-25T09:49:38.679317+0000","last_unstale":"2023-08-25T09:50:12.558258+0000","last_undegraded":"2023-08-25T09:50:12.558258+0000","last_fullsized":"2023-08-25T09:50:12.558258+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:11.134847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.566106+0000","last_change":"2023-08-25T09:49:34.619590+0000","last_active":"2023-08-25T09:50:12.566106+0000","last_peered":"2023-08-25T09:50:12.566106+0000","last_clean":"2023-08-25T09:50:12.566106+0000","last_became_active":"2023-08-25T09:49:34.619244+0000","last_became_peered":"2023-08-25T09:49:34.619244+0000","last_unstale":"2023-08-25T09:50:12.566106+0000","last_undegraded":"2023-08-25T09:50:12.566106+0000","last_fullsized":"2023-08-25T09:50:12.566106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:29:31.536127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761717+0000","last_change":"2023-08-25T09:49:33.614225+0000","last_active":"2023-08-25T09:50:22.761717+0000","last_peered":"2023-08-25T09:50:22.761717+0000","last_clean":"2023-08-25T09:50:22.761717+0000","last_became_active":"2023-08-25T09:49:33.613737+0000","last_became_peered":"2023-08-25T09:49:33.613737+0000","last_unstale":"2023-08-25T09:50:22.761717+0000","last_undegraded":"2023-08-25T09:50:22.761717+0000","last_fullsized":"2023-08-25T09:50:22.761717+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:07:02.583931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556866+0000","last_change":"2023-08-25T09:49:27.527484+0000","last_active":"2023-08-25T09:50:12.556866+0000","last_peered":"2023-08-25T09:50:12.556866+0000","last_clean":"2023-08-25T09:50:12.556866+0000","last_became_active":"2023-08-25T09:49:27.526326+0000","last_became_peered":"2023-08-25T09:49:27.526326+0000","last_unstale":"2023-08-25T09:50:12.556866+0000","last_undegraded":"2023-08-25T09:50:12.556866+0000","last_fullsized":"2023-08-25T09:50:12.556866+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:26.497044+0000","last_clean_scrub_stamp":"2023-08-25T09:49:26.497044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:43:35.431443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":187,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626710+0000","last_change":"2023-08-25T09:49:20.431906+0000","last_active":"2023-08-25T09:50:12.626710+0000","last_peered":"2023-08-25T09:50:12.626710+0000","last_clean":"2023-08-25T09:50:12.626710+0000","last_became_active":"2023-08-25T09:49:20.429819+0000","last_became_peered":"2023-08-25T09:49:20.429819+0000","last_unstale":"2023-08-25T09:50:12.626710+0000","last_undegraded":"2023-08-25T09:50:12.626710+0000","last_fullsized":"2023-08-25T09:50:12.626710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:19.277436+0000","last_clean_scrub_stamp":"2023-08-25T09:49:19.277436+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:37:25.347377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626465+0000","last_change":"2023-08-25T09:49:38.689251+0000","last_active":"2023-08-25T09:50:12.626465+0000","last_peered":"2023-08-25T09:50:12.626465+0000","last_clean":"2023-08-25T09:50:12.626465+0000","last_became_active":"2023-08-25T09:49:38.688874+0000","last_became_peered":"2023-08-25T09:49:38.688874+0000","last_unstale":"2023-08-25T09:50:12.626465+0000","last_undegraded":"2023-08-25T09:50:12.626465+0000","last_fullsized":"2023-08-25T09:50:12.626465+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:49:37.046349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629357+0000","last_change":"2023-08-25T09:49:34.621533+0000","last_active":"2023-08-25T09:50:12.629357+0000","last_peered":"2023-08-25T09:50:12.629357+0000","last_clean":"2023-08-25T09:50:12.629357+0000","last_became_active":"2023-08-25T09:49:34.620883+0000","last_became_peered":"2023-08-25T09:49:34.620883+0000","last_unstale":"2023-08-25T09:50:12.629357+0000","last_undegraded":"2023-08-25T09:50:12.629357+0000","last_fullsized":"2023-08-25T09:50:12.629357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:29:01.327572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"29'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630079+0000","last_change":"2023-08-25T09:49:33.599956+0000","last_active":"2023-08-25T09:50:12.630079+0000","last_peered":"2023-08-25T09:50:12.630079+0000","last_clean":"2023-08-25T09:50:12.630079+0000","last_became_active":"2023-08-25T09:49:33.599707+0000","last_became_peered":"2023-08-25T09:49:33.599707+0000","last_unstale":"2023-08-25T09:50:12.630079+0000","last_undegraded":"2023-08-25T09:50:12.630079+0000","last_fullsized":"2023-08-25T09:50:12.630079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:07:37.714359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.559002+0000","last_change":"2023-08-25T09:49:38.683077+0000","last_active":"2023-08-25T09:50:12.559002+0000","last_peered":"2023-08-25T09:50:12.559002+0000","last_clean":"2023-08-25T09:50:12.559002+0000","last_became_active":"2023-08-25T09:49:38.682685+0000","last_became_peered":"2023-08-25T09:49:38.682685+0000","last_unstale":"2023-08-25T09:50:12.559002+0000","last_undegraded":"2023-08-25T09:50:12.559002+0000","last_fullsized":"2023-08-25T09:50:12.559002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:23:29.317977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.559077+0000","last_change":"2023-08-25T09:49:34.618531+0000","last_active":"2023-08-25T09:50:12.559077+0000","last_peered":"2023-08-25T09:50:12.559077+0000","last_clean":"2023-08-25T09:50:12.559077+0000","last_became_active":"2023-08-25T09:49:34.618203+0000","last_became_peered":"2023-08-25T09:49:34.618203+0000","last_unstale":"2023-08-25T09:50:12.559077+0000","last_undegraded":"2023-08-25T09:50:12.559077+0000","last_fullsized":"2023-08-25T09:50:12.559077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:09:36.207083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"31'7","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.755917+0000","last_change":"2023-08-25T09:49:33.613000+0000","last_active":"2023-08-25T09:50:22.755917+0000","last_peered":"2023-08-25T09:50:22.755917+0000","last_clean":"2023-08-25T09:50:22.755917+0000","last_became_active":"2023-08-25T09:49:33.612120+0000","last_became_peered":"2023-08-25T09:49:33.612120+0000","last_unstale":"2023-08-25T09:50:22.755917+0000","last_undegraded":"2023-08-25T09:50:22.755917+0000","last_fullsized":"2023-08-25T09:50:22.755917+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:54:35.082983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556600+0000","last_change":"2023-08-25T09:49:38.682448+0000","last_active":"2023-08-25T09:50:12.556600+0000","last_peered":"2023-08-25T09:50:12.556600+0000","last_clean":"2023-08-25T09:50:12.556600+0000","last_became_active":"2023-08-25T09:49:38.679927+0000","last_became_peered":"2023-08-25T09:49:38.679927+0000","last_unstale":"2023-08-25T09:50:12.556600+0000","last_undegraded":"2023-08-25T09:50:12.556600+0000","last_fullsized":"2023-08-25T09:50:12.556600+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:57:07.092488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558495+0000","last_change":"2023-08-25T09:49:34.618258+0000","last_active":"2023-08-25T09:50:12.558495+0000","last_peered":"2023-08-25T09:50:12.558495+0000","last_clean":"2023-08-25T09:50:12.558495+0000","last_became_active":"2023-08-25T09:49:34.617932+0000","last_became_peered":"2023-08-25T09:49:34.617932+0000","last_unstale":"2023-08-25T09:50:12.558495+0000","last_undegraded":"2023-08-25T09:50:12.558495+0000","last_fullsized":"2023-08-25T09:50:12.558495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:49:01.069322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573225+0000","last_change":"2023-08-25T09:49:33.614411+0000","last_active":"2023-08-25T09:50:17.573225+0000","last_peered":"2023-08-25T09:50:17.573225+0000","last_clean":"2023-08-25T09:50:17.573225+0000","last_became_active":"2023-08-25T09:49:33.613864+0000","last_became_peered":"2023-08-25T09:49:33.613864+0000","last_unstale":"2023-08-25T09:50:17.573225+0000","last_undegraded":"2023-08-25T09:50:17.573225+0000","last_fullsized":"2023-08-25T09:50:17.573225+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:04:42.286097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627074+0000","last_change":"2023-08-25T09:49:38.683990+0000","last_active":"2023-08-25T09:50:12.627074+0000","last_peered":"2023-08-25T09:50:12.627074+0000","last_clean":"2023-08-25T09:50:12.627074+0000","last_became_active":"2023-08-25T09:49:38.683542+0000","last_became_peered":"2023-08-25T09:49:38.683542+0000","last_unstale":"2023-08-25T09:50:12.627074+0000","last_undegraded":"2023-08-25T09:50:12.627074+0000","last_fullsized":"2023-08-25T09:50:12.627074+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:10:04.637461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558961+0000","last_change":"2023-08-25T09:49:34.609513+0000","last_active":"2023-08-25T09:50:12.558961+0000","last_peered":"2023-08-25T09:50:12.558961+0000","last_clean":"2023-08-25T09:50:12.558961+0000","last_became_active":"2023-08-25T09:49:34.609148+0000","last_became_peered":"2023-08-25T09:49:34.609148+0000","last_unstale":"2023-08-25T09:50:12.558961+0000","last_undegraded":"2023-08-25T09:50:12.558961+0000","last_fullsized":"2023-08-25T09:50:12.558961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:08:21.839926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"29'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627445+0000","last_change":"2023-08-25T09:49:33.608361+0000","last_active":"2023-08-25T09:50:12.627445+0000","last_peered":"2023-08-25T09:50:12.627445+0000","last_clean":"2023-08-25T09:50:12.627445+0000","last_became_active":"2023-08-25T09:49:33.608012+0000","last_became_peered":"2023-08-25T09:49:33.608012+0000","last_unstale":"2023-08-25T09:50:12.627445+0000","last_undegraded":"2023-08-25T09:50:12.627445+0000","last_fullsized":"2023-08-25T09:50:12.627445+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:34:35.939804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557002+0000","last_change":"2023-08-25T09:49:38.679711+0000","last_active":"2023-08-25T09:50:12.557002+0000","last_peered":"2023-08-25T09:50:12.557002+0000","last_clean":"2023-08-25T09:50:12.557002+0000","last_became_active":"2023-08-25T09:49:38.679224+0000","last_became_peered":"2023-08-25T09:49:38.679224+0000","last_unstale":"2023-08-25T09:50:12.557002+0000","last_undegraded":"2023-08-25T09:50:12.557002+0000","last_fullsized":"2023-08-25T09:50:12.557002+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:40:23.888752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551613+0000","last_change":"2023-08-25T09:49:34.622254+0000","last_active":"2023-08-25T09:50:12.551613+0000","last_peered":"2023-08-25T09:50:12.551613+0000","last_clean":"2023-08-25T09:50:12.551613+0000","last_became_active":"2023-08-25T09:49:34.621726+0000","last_became_peered":"2023-08-25T09:49:34.621726+0000","last_unstale":"2023-08-25T09:50:12.551613+0000","last_undegraded":"2023-08-25T09:50:12.551613+0000","last_fullsized":"2023-08-25T09:50:12.551613+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:34:14.609357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.582719+0000","last_change":"2023-08-25T09:49:33.613708+0000","last_active":"2023-08-25T09:50:17.582719+0000","last_peered":"2023-08-25T09:50:17.582719+0000","last_clean":"2023-08-25T09:50:17.582719+0000","last_became_active":"2023-08-25T09:49:33.613419+0000","last_became_peered":"2023-08-25T09:49:33.613419+0000","last_unstale":"2023-08-25T09:50:17.582719+0000","last_undegraded":"2023-08-25T09:50:17.582719+0000","last_fullsized":"2023-08-25T09:50:17.582719+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:18:04.169294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1610,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556982+0000","last_change":"2023-08-25T09:49:38.678729+0000","last_active":"2023-08-25T09:50:12.556982+0000","last_peered":"2023-08-25T09:50:12.556982+0000","last_clean":"2023-08-25T09:50:12.556982+0000","last_became_active":"2023-08-25T09:49:38.678273+0000","last_became_peered":"2023-08-25T09:49:38.678273+0000","last_unstale":"2023-08-25T09:50:12.556982+0000","last_undegraded":"2023-08-25T09:50:12.556982+0000","last_fullsized":"2023-08-25T09:50:12.556982+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:23:07.786471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.571725+0000","last_change":"2023-08-25T09:50:12.571725+0000","last_active":"2023-08-25T09:50:12.571725+0000","last_peered":"2023-08-25T09:50:12.571725+0000","last_clean":"2023-08-25T09:50:12.571725+0000","last_became_active":"2023-08-25T09:50:12.570960+0000","last_became_peered":"2023-08-25T09:50:12.570960+0000","last_unstale":"2023-08-25T09:50:12.571725+0000","last_undegraded":"2023-08-25T09:50:12.571725+0000","last_fullsized":"2023-08-25T09:50:12.571725+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:36.819051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"31'3","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.570490+0000","last_change":"2023-08-25T09:49:33.620878+0000","last_active":"2023-08-25T09:50:17.570490+0000","last_peered":"2023-08-25T09:50:17.570490+0000","last_clean":"2023-08-25T09:50:17.570490+0000","last_became_active":"2023-08-25T09:49:33.620554+0000","last_became_peered":"2023-08-25T09:49:33.620554+0000","last_unstale":"2023-08-25T09:50:17.570490+0000","last_undegraded":"2023-08-25T09:50:17.570490+0000","last_fullsized":"2023-08-25T09:50:17.570490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:21:54.167747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556745+0000","last_change":"2023-08-25T09:49:38.671606+0000","last_active":"2023-08-25T09:50:12.556745+0000","last_peered":"2023-08-25T09:50:12.556745+0000","last_clean":"2023-08-25T09:50:12.556745+0000","last_became_active":"2023-08-25T09:49:38.671180+0000","last_became_peered":"2023-08-25T09:49:38.671180+0000","last_unstale":"2023-08-25T09:50:12.556745+0000","last_undegraded":"2023-08-25T09:50:12.556745+0000","last_fullsized":"2023-08-25T09:50:12.556745+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:22:52.170219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554991+0000","last_change":"2023-08-25T09:49:34.621362+0000","last_active":"2023-08-25T09:50:12.554991+0000","last_peered":"2023-08-25T09:50:12.554991+0000","last_clean":"2023-08-25T09:50:12.554991+0000","last_became_active":"2023-08-25T09:49:34.621045+0000","last_became_peered":"2023-08-25T09:49:34.621045+0000","last_unstale":"2023-08-25T09:50:12.554991+0000","last_undegraded":"2023-08-25T09:50:12.554991+0000","last_fullsized":"2023-08-25T09:50:12.554991+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:54:40.403648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761278+0000","last_change":"2023-08-25T09:49:33.610813+0000","last_active":"2023-08-25T09:50:22.761278+0000","last_peered":"2023-08-25T09:50:22.761278+0000","last_clean":"2023-08-25T09:50:22.761278+0000","last_became_active":"2023-08-25T09:49:33.610476+0000","last_became_peered":"2023-08-25T09:49:33.610476+0000","last_unstale":"2023-08-25T09:50:22.761278+0000","last_undegraded":"2023-08-25T09:50:22.761278+0000","last_fullsized":"2023-08-25T09:50:22.761278+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:09:51.901728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556688+0000","last_change":"2023-08-25T09:49:38.666844+0000","last_active":"2023-08-25T09:50:12.556688+0000","last_peered":"2023-08-25T09:50:12.556688+0000","last_clean":"2023-08-25T09:50:12.556688+0000","last_became_active":"2023-08-25T09:49:38.666415+0000","last_became_peered":"2023-08-25T09:49:38.666415+0000","last_unstale":"2023-08-25T09:50:12.556688+0000","last_undegraded":"2023-08-25T09:50:12.556688+0000","last_fullsized":"2023-08-25T09:50:12.556688+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:59:46.452007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558337+0000","last_change":"2023-08-25T09:49:34.616834+0000","last_active":"2023-08-25T09:50:12.558337+0000","last_peered":"2023-08-25T09:50:12.558337+0000","last_clean":"2023-08-25T09:50:12.558337+0000","last_became_active":"2023-08-25T09:49:34.616452+0000","last_became_peered":"2023-08-25T09:49:34.616452+0000","last_unstale":"2023-08-25T09:50:12.558337+0000","last_undegraded":"2023-08-25T09:50:12.558337+0000","last_fullsized":"2023-08-25T09:50:12.558337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:08:31.091599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627499+0000","last_change":"2023-08-25T09:49:33.610733+0000","last_active":"2023-08-25T09:50:12.627499+0000","last_peered":"2023-08-25T09:50:12.627499+0000","last_clean":"2023-08-25T09:50:12.627499+0000","last_became_active":"2023-08-25T09:49:33.610410+0000","last_became_peered":"2023-08-25T09:49:33.610410+0000","last_unstale":"2023-08-25T09:50:12.627499+0000","last_undegraded":"2023-08-25T09:50:12.627499+0000","last_fullsized":"2023-08-25T09:50:12.627499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:56:43.404686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626320+0000","last_change":"2023-08-25T09:49:38.690646+0000","last_active":"2023-08-25T09:50:12.626320+0000","last_peered":"2023-08-25T09:50:12.626320+0000","last_clean":"2023-08-25T09:50:12.626320+0000","last_became_active":"2023-08-25T09:49:38.690151+0000","last_became_peered":"2023-08-25T09:49:38.690151+0000","last_unstale":"2023-08-25T09:50:12.626320+0000","last_undegraded":"2023-08-25T09:50:12.626320+0000","last_fullsized":"2023-08-25T09:50:12.626320+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:36:12.229666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556039+0000","last_change":"2023-08-25T09:49:38.679888+0000","last_active":"2023-08-25T09:50:12.556039+0000","last_peered":"2023-08-25T09:50:12.556039+0000","last_clean":"2023-08-25T09:50:12.556039+0000","last_became_active":"2023-08-25T09:49:38.679451+0000","last_became_peered":"2023-08-25T09:49:38.679451+0000","last_unstale":"2023-08-25T09:50:12.556039+0000","last_undegraded":"2023-08-25T09:50:12.556039+0000","last_fullsized":"2023-08-25T09:50:12.556039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:34:48.602052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"31'17","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.572678+0000","last_change":"2023-08-25T09:49:33.610235+0000","last_active":"2023-08-25T09:50:17.572678+0000","last_peered":"2023-08-25T09:50:17.572678+0000","last_clean":"2023-08-25T09:50:17.572678+0000","last_became_active":"2023-08-25T09:49:33.609519+0000","last_became_peered":"2023-08-25T09:49:33.609519+0000","last_unstale":"2023-08-25T09:50:17.572678+0000","last_undegraded":"2023-08-25T09:50:17.572678+0000","last_fullsized":"2023-08-25T09:50:17.572678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:17:32.295126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.633719+0000","last_change":"2023-08-25T09:50:12.631949+0000","last_active":"2023-08-25T09:50:12.633719+0000","last_peered":"2023-08-25T09:50:12.633719+0000","last_clean":"2023-08-25T09:50:12.633719+0000","last_became_active":"2023-08-25T09:50:12.630910+0000","last_became_peered":"2023-08-25T09:50:12.630910+0000","last_unstale":"2023-08-25T09:50:12.633719+0000","last_undegraded":"2023-08-25T09:50:12.633719+0000","last_fullsized":"2023-08-25T09:50:12.633719+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:36:40.321956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627733+0000","last_change":"2023-08-25T09:49:38.673265+0000","last_active":"2023-08-25T09:50:12.627733+0000","last_peered":"2023-08-25T09:50:12.627733+0000","last_clean":"2023-08-25T09:50:12.627733+0000","last_became_active":"2023-08-25T09:49:38.672941+0000","last_became_peered":"2023-08-25T09:49:38.672941+0000","last_unstale":"2023-08-25T09:50:12.627733+0000","last_undegraded":"2023-08-25T09:50:12.627733+0000","last_fullsized":"2023-08-25T09:50:12.627733+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:10:04.733907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"31'6","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.573176+0000","last_change":"2023-08-25T09:49:33.613796+0000","last_active":"2023-08-25T09:50:17.573176+0000","last_peered":"2023-08-25T09:50:17.573176+0000","last_clean":"2023-08-25T09:50:17.573176+0000","last_became_active":"2023-08-25T09:49:33.613466+0000","last_became_peered":"2023-08-25T09:49:33.613466+0000","last_unstale":"2023-08-25T09:50:17.573176+0000","last_undegraded":"2023-08-25T09:50:17.573176+0000","last_fullsized":"2023-08-25T09:50:17.573176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:43:49.838183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1028,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.552014+0000","last_change":"2023-08-25T09:49:34.632531+0000","last_active":"2023-08-25T09:50:12.552014+0000","last_peered":"2023-08-25T09:50:12.552014+0000","last_clean":"2023-08-25T09:50:12.552014+0000","last_became_active":"2023-08-25T09:49:34.632163+0000","last_became_peered":"2023-08-25T09:49:34.632163+0000","last_unstale":"2023-08-25T09:50:12.552014+0000","last_undegraded":"2023-08-25T09:50:12.552014+0000","last_fullsized":"2023-08-25T09:50:12.552014+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:55:55.008026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"31'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:17.610262+0000","last_change":"2023-08-25T09:49:33.603438+0000","last_active":"2023-08-25T09:50:17.610262+0000","last_peered":"2023-08-25T09:50:17.610262+0000","last_clean":"2023-08-25T09:50:17.610262+0000","last_became_active":"2023-08-25T09:49:33.603119+0000","last_became_peered":"2023-08-25T09:49:33.603119+0000","last_unstale":"2023-08-25T09:50:17.610262+0000","last_undegraded":"2023-08-25T09:50:17.610262+0000","last_fullsized":"2023-08-25T09:50:17.610262+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:00:10.028050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555549+0000","last_change":"2023-08-25T09:49:34.616567+0000","last_active":"2023-08-25T09:50:12.555549+0000","last_peered":"2023-08-25T09:50:12.555549+0000","last_clean":"2023-08-25T09:50:12.555549+0000","last_became_active":"2023-08-25T09:49:34.616245+0000","last_became_peered":"2023-08-25T09:49:34.616245+0000","last_unstale":"2023-08-25T09:50:12.555549+0000","last_undegraded":"2023-08-25T09:50:12.555549+0000","last_fullsized":"2023-08-25T09:50:12.555549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:27:10.547485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628374+0000","last_change":"2023-08-25T09:49:38.677091+0000","last_active":"2023-08-25T09:50:12.628374+0000","last_peered":"2023-08-25T09:50:12.628374+0000","last_clean":"2023-08-25T09:50:12.628374+0000","last_became_active":"2023-08-25T09:49:38.676680+0000","last_became_peered":"2023-08-25T09:49:38.676680+0000","last_unstale":"2023-08-25T09:50:12.628374+0000","last_undegraded":"2023-08-25T09:50:12.628374+0000","last_fullsized":"2023-08-25T09:50:12.628374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:30:14.626577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"31'9","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:22.761509+0000","last_change":"2023-08-25T09:49:33.613707+0000","last_active":"2023-08-25T09:50:22.761509+0000","last_peered":"2023-08-25T09:50:22.761509+0000","last_clean":"2023-08-25T09:50:22.761509+0000","last_became_active":"2023-08-25T09:49:33.613390+0000","last_became_peered":"2023-08-25T09:49:33.613390+0000","last_unstale":"2023-08-25T09:50:22.761509+0000","last_undegraded":"2023-08-25T09:50:22.761509+0000","last_fullsized":"2023-08-25T09:50:22.761509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:32.542647+0000","last_clean_scrub_stamp":"2023-08-25T09:49:32.542647+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:25:30.647173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628001+0000","last_change":"2023-08-25T09:49:34.631065+0000","last_active":"2023-08-25T09:50:12.628001+0000","last_peered":"2023-08-25T09:50:12.628001+0000","last_clean":"2023-08-25T09:50:12.628001+0000","last_became_active":"2023-08-25T09:49:34.630739+0000","last_became_peered":"2023-08-25T09:49:34.630739+0000","last_unstale":"2023-08-25T09:50:12.628001+0000","last_undegraded":"2023-08-25T09:50:12.628001+0000","last_fullsized":"2023-08-25T09:50:12.628001+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:24:05.525804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554765+0000","last_change":"2023-08-25T09:49:38.672119+0000","last_active":"2023-08-25T09:50:12.554765+0000","last_peered":"2023-08-25T09:50:12.554765+0000","last_clean":"2023-08-25T09:50:12.554765+0000","last_became_active":"2023-08-25T09:49:38.671710+0000","last_became_peered":"2023-08-25T09:49:38.671710+0000","last_unstale":"2023-08-25T09:50:12.554765+0000","last_undegraded":"2023-08-25T09:50:12.554765+0000","last_fullsized":"2023-08-25T09:50:12.554765+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:37.617978+0000","last_clean_scrub_stamp":"2023-08-25T09:49:37.617978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:06:14.805887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.564968+0000","last_change":"2023-08-25T09:49:34.617482+0000","last_active":"2023-08-25T09:50:12.564968+0000","last_peered":"2023-08-25T09:50:12.564968+0000","last_clean":"2023-08-25T09:50:12.564968+0000","last_became_active":"2023-08-25T09:49:34.617162+0000","last_became_peered":"2023-08-25T09:49:34.617162+0000","last_unstale":"2023-08-25T09:50:12.564968+0000","last_undegraded":"2023-08-25T09:50:12.564968+0000","last_fullsized":"2023-08-25T09:50:12.564968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:08:13.506185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625740+0000","last_change":"2023-08-25T09:49:34.625020+0000","last_active":"2023-08-25T09:50:12.625740+0000","last_peered":"2023-08-25T09:50:12.625740+0000","last_clean":"2023-08-25T09:50:12.625740+0000","last_became_active":"2023-08-25T09:49:34.624374+0000","last_became_peered":"2023-08-25T09:49:34.624374+0000","last_unstale":"2023-08-25T09:50:12.625740+0000","last_undegraded":"2023-08-25T09:50:12.625740+0000","last_fullsized":"2023-08-25T09:50:12.625740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:06:22.528448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558622+0000","last_change":"2023-08-25T09:49:34.614594+0000","last_active":"2023-08-25T09:50:12.558622+0000","last_peered":"2023-08-25T09:50:12.558622+0000","last_clean":"2023-08-25T09:50:12.558622+0000","last_became_active":"2023-08-25T09:49:34.613845+0000","last_became_peered":"2023-08-25T09:49:34.613845+0000","last_unstale":"2023-08-25T09:50:12.558622+0000","last_undegraded":"2023-08-25T09:50:12.558622+0000","last_fullsized":"2023-08-25T09:50:12.558622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:59:47.407847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627103+0000","last_change":"2023-08-25T09:49:34.623638+0000","last_active":"2023-08-25T09:50:12.627103+0000","last_peered":"2023-08-25T09:50:12.627103+0000","last_clean":"2023-08-25T09:50:12.627103+0000","last_became_active":"2023-08-25T09:49:34.623379+0000","last_became_peered":"2023-08-25T09:49:34.623379+0000","last_unstale":"2023-08-25T09:50:12.627103+0000","last_undegraded":"2023-08-25T09:50:12.627103+0000","last_fullsized":"2023-08-25T09:50:12.627103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:39:13.685181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628326+0000","last_change":"2023-08-25T09:49:34.622324+0000","last_active":"2023-08-25T09:50:12.628326+0000","last_peered":"2023-08-25T09:50:12.628326+0000","last_clean":"2023-08-25T09:50:12.628326+0000","last_became_active":"2023-08-25T09:49:34.621482+0000","last_became_peered":"2023-08-25T09:49:34.621482+0000","last_unstale":"2023-08-25T09:50:12.628326+0000","last_undegraded":"2023-08-25T09:50:12.628326+0000","last_fullsized":"2023-08-25T09:50:12.628326+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:10:35.667202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555241+0000","last_change":"2023-08-25T09:49:34.616448+0000","last_active":"2023-08-25T09:50:12.555241+0000","last_peered":"2023-08-25T09:50:12.555241+0000","last_clean":"2023-08-25T09:50:12.555241+0000","last_became_active":"2023-08-25T09:49:34.616028+0000","last_became_peered":"2023-08-25T09:49:34.616028+0000","last_unstale":"2023-08-25T09:50:12.555241+0000","last_undegraded":"2023-08-25T09:50:12.555241+0000","last_fullsized":"2023-08-25T09:50:12.555241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:41:58.938045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.630384+0000","last_change":"2023-08-25T09:49:34.619812+0000","last_active":"2023-08-25T09:50:12.630384+0000","last_peered":"2023-08-25T09:50:12.630384+0000","last_clean":"2023-08-25T09:50:12.630384+0000","last_became_active":"2023-08-25T09:49:34.619465+0000","last_became_peered":"2023-08-25T09:49:34.619465+0000","last_unstale":"2023-08-25T09:50:12.630384+0000","last_undegraded":"2023-08-25T09:50:12.630384+0000","last_fullsized":"2023-08-25T09:50:12.630384+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T11:01:42.764778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.629040+0000","last_change":"2023-08-25T09:49:34.621379+0000","last_active":"2023-08-25T09:50:12.629040+0000","last_peered":"2023-08-25T09:50:12.629040+0000","last_clean":"2023-08-25T09:50:12.629040+0000","last_became_active":"2023-08-25T09:49:34.620519+0000","last_became_peered":"2023-08-25T09:49:34.620519+0000","last_unstale":"2023-08-25T09:50:12.629040+0000","last_undegraded":"2023-08-25T09:50:12.629040+0000","last_fullsized":"2023-08-25T09:50:12.629040+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:47:27.685717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625993+0000","last_change":"2023-08-25T09:49:34.626125+0000","last_active":"2023-08-25T09:50:12.625993+0000","last_peered":"2023-08-25T09:50:12.625993+0000","last_clean":"2023-08-25T09:50:12.625993+0000","last_became_active":"2023-08-25T09:49:34.625766+0000","last_became_peered":"2023-08-25T09:49:34.625766+0000","last_unstale":"2023-08-25T09:50:12.625993+0000","last_undegraded":"2023-08-25T09:50:12.625993+0000","last_fullsized":"2023-08-25T09:50:12.625993+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:42:27.784343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.625508+0000","last_change":"2023-08-25T09:49:34.615257+0000","last_active":"2023-08-25T09:50:12.625508+0000","last_peered":"2023-08-25T09:50:12.625508+0000","last_clean":"2023-08-25T09:50:12.625508+0000","last_became_active":"2023-08-25T09:49:34.614855+0000","last_became_peered":"2023-08-25T09:49:34.614855+0000","last_unstale":"2023-08-25T09:50:12.625508+0000","last_undegraded":"2023-08-25T09:50:12.625508+0000","last_fullsized":"2023-08-25T09:50:12.625508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:13:44.415314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.550868+0000","last_change":"2023-08-25T09:49:34.623185+0000","last_active":"2023-08-25T09:50:12.550868+0000","last_peered":"2023-08-25T09:50:12.550868+0000","last_clean":"2023-08-25T09:50:12.550868+0000","last_became_active":"2023-08-25T09:49:34.622850+0000","last_became_peered":"2023-08-25T09:49:34.622850+0000","last_unstale":"2023-08-25T09:50:12.550868+0000","last_undegraded":"2023-08-25T09:50:12.550868+0000","last_fullsized":"2023-08-25T09:50:12.550868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:33:22.249937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.626399+0000","last_change":"2023-08-25T09:49:34.621937+0000","last_active":"2023-08-25T09:50:12.626399+0000","last_peered":"2023-08-25T09:50:12.626399+0000","last_clean":"2023-08-25T09:50:12.626399+0000","last_became_active":"2023-08-25T09:49:34.620455+0000","last_became_peered":"2023-08-25T09:49:34.620455+0000","last_unstale":"2023-08-25T09:50:12.626399+0000","last_undegraded":"2023-08-25T09:50:12.626399+0000","last_fullsized":"2023-08-25T09:50:12.626399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:50:48.580274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558258+0000","last_change":"2023-08-25T09:49:34.619574+0000","last_active":"2023-08-25T09:50:12.558258+0000","last_peered":"2023-08-25T09:50:12.558258+0000","last_clean":"2023-08-25T09:50:12.558258+0000","last_became_active":"2023-08-25T09:49:34.619270+0000","last_became_peered":"2023-08-25T09:49:34.619270+0000","last_unstale":"2023-08-25T09:50:12.558258+0000","last_undegraded":"2023-08-25T09:50:12.558258+0000","last_fullsized":"2023-08-25T09:50:12.558258+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T15:54:01.929500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557718+0000","last_change":"2023-08-25T09:49:34.616688+0000","last_active":"2023-08-25T09:50:12.557718+0000","last_peered":"2023-08-25T09:50:12.557718+0000","last_clean":"2023-08-25T09:50:12.557718+0000","last_became_active":"2023-08-25T09:49:34.616367+0000","last_became_peered":"2023-08-25T09:49:34.616367+0000","last_unstale":"2023-08-25T09:50:12.557718+0000","last_undegraded":"2023-08-25T09:50:12.557718+0000","last_fullsized":"2023-08-25T09:50:12.557718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:14:25.545926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558660+0000","last_change":"2023-08-25T09:49:34.606447+0000","last_active":"2023-08-25T09:50:12.558660+0000","last_peered":"2023-08-25T09:50:12.558660+0000","last_clean":"2023-08-25T09:50:12.558660+0000","last_became_active":"2023-08-25T09:49:34.606094+0000","last_became_peered":"2023-08-25T09:49:34.606094+0000","last_unstale":"2023-08-25T09:50:12.558660+0000","last_undegraded":"2023-08-25T09:50:12.558660+0000","last_fullsized":"2023-08-25T09:50:12.558660+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T10:31:12.817270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551531+0000","last_change":"2023-08-25T09:49:34.632875+0000","last_active":"2023-08-25T09:50:12.551531+0000","last_peered":"2023-08-25T09:50:12.551531+0000","last_clean":"2023-08-25T09:50:12.551531+0000","last_became_active":"2023-08-25T09:49:34.632591+0000","last_became_peered":"2023-08-25T09:49:34.632591+0000","last_unstale":"2023-08-25T09:50:12.551531+0000","last_undegraded":"2023-08-25T09:50:12.551531+0000","last_fullsized":"2023-08-25T09:50:12.551531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:21:41.584105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554589+0000","last_change":"2023-08-25T09:49:34.615829+0000","last_active":"2023-08-25T09:50:12.554589+0000","last_peered":"2023-08-25T09:50:12.554589+0000","last_clean":"2023-08-25T09:50:12.554589+0000","last_became_active":"2023-08-25T09:49:34.615342+0000","last_became_peered":"2023-08-25T09:49:34.615342+0000","last_unstale":"2023-08-25T09:50:12.554589+0000","last_undegraded":"2023-08-25T09:50:12.554589+0000","last_fullsized":"2023-08-25T09:50:12.554589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:33:10.183565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.628035+0000","last_change":"2023-08-25T09:49:34.627950+0000","last_active":"2023-08-25T09:50:12.628035+0000","last_peered":"2023-08-25T09:50:12.628035+0000","last_clean":"2023-08-25T09:50:12.628035+0000","last_became_active":"2023-08-25T09:49:34.627623+0000","last_became_peered":"2023-08-25T09:49:34.627623+0000","last_unstale":"2023-08-25T09:50:12.628035+0000","last_undegraded":"2023-08-25T09:50:12.628035+0000","last_fullsized":"2023-08-25T09:50:12.628035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:18:24.478817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.555844+0000","last_change":"2023-08-25T09:49:34.602198+0000","last_active":"2023-08-25T09:50:12.555844+0000","last_peered":"2023-08-25T09:50:12.555844+0000","last_clean":"2023-08-25T09:50:12.555844+0000","last_became_active":"2023-08-25T09:49:34.601575+0000","last_became_peered":"2023-08-25T09:49:34.601575+0000","last_unstale":"2023-08-25T09:50:12.555844+0000","last_undegraded":"2023-08-25T09:50:12.555844+0000","last_fullsized":"2023-08-25T09:50:12.555844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:46:47.388440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557103+0000","last_change":"2023-08-25T09:49:34.609678+0000","last_active":"2023-08-25T09:50:12.557103+0000","last_peered":"2023-08-25T09:50:12.557103+0000","last_clean":"2023-08-25T09:50:12.557103+0000","last_became_active":"2023-08-25T09:49:34.609299+0000","last_became_peered":"2023-08-25T09:49:34.609299+0000","last_unstale":"2023-08-25T09:50:12.557103+0000","last_undegraded":"2023-08-25T09:50:12.557103+0000","last_fullsized":"2023-08-25T09:50:12.557103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T12:14:19.268187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557240+0000","last_change":"2023-08-25T09:49:34.607002+0000","last_active":"2023-08-25T09:50:12.557240+0000","last_peered":"2023-08-25T09:50:12.557240+0000","last_clean":"2023-08-25T09:50:12.557240+0000","last_became_active":"2023-08-25T09:49:34.606611+0000","last_became_peered":"2023-08-25T09:49:34.606611+0000","last_unstale":"2023-08-25T09:50:12.557240+0000","last_undegraded":"2023-08-25T09:50:12.557240+0000","last_fullsized":"2023-08-25T09:50:12.557240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:26:06.862900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.558086+0000","last_change":"2023-08-25T09:49:34.623517+0000","last_active":"2023-08-25T09:50:12.558086+0000","last_peered":"2023-08-25T09:50:12.558086+0000","last_clean":"2023-08-25T09:50:12.558086+0000","last_became_active":"2023-08-25T09:49:34.623002+0000","last_became_peered":"2023-08-25T09:49:34.623002+0000","last_unstale":"2023-08-25T09:50:12.558086+0000","last_undegraded":"2023-08-25T09:50:12.558086+0000","last_fullsized":"2023-08-25T09:50:12.558086+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T20:57:21.290722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557015+0000","last_change":"2023-08-25T09:49:34.614367+0000","last_active":"2023-08-25T09:50:12.557015+0000","last_peered":"2023-08-25T09:50:12.557015+0000","last_clean":"2023-08-25T09:50:12.557015+0000","last_became_active":"2023-08-25T09:49:34.614003+0000","last_became_peered":"2023-08-25T09:49:34.614003+0000","last_unstale":"2023-08-25T09:50:12.557015+0000","last_undegraded":"2023-08-25T09:50:12.557015+0000","last_fullsized":"2023-08-25T09:50:12.557015+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:02:22.581658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551227+0000","last_change":"2023-08-25T09:49:34.632874+0000","last_active":"2023-08-25T09:50:12.551227+0000","last_peered":"2023-08-25T09:50:12.551227+0000","last_clean":"2023-08-25T09:50:12.551227+0000","last_became_active":"2023-08-25T09:49:34.632569+0000","last_became_peered":"2023-08-25T09:49:34.632569+0000","last_unstale":"2023-08-25T09:50:12.551227+0000","last_undegraded":"2023-08-25T09:50:12.551227+0000","last_fullsized":"2023-08-25T09:50:12.551227+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:15:53.937529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.554170+0000","last_change":"2023-08-25T09:49:34.599950+0000","last_active":"2023-08-25T09:50:12.554170+0000","last_peered":"2023-08-25T09:50:12.554170+0000","last_clean":"2023-08-25T09:50:12.554170+0000","last_became_active":"2023-08-25T09:49:34.599588+0000","last_became_peered":"2023-08-25T09:49:34.599588+0000","last_unstale":"2023-08-25T09:50:12.554170+0000","last_undegraded":"2023-08-25T09:50:12.554170+0000","last_fullsized":"2023-08-25T09:50:12.554170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T17:36:20.354285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.627168+0000","last_change":"2023-08-25T09:49:34.627988+0000","last_active":"2023-08-25T09:50:12.627168+0000","last_peered":"2023-08-25T09:50:12.627168+0000","last_clean":"2023-08-25T09:50:12.627168+0000","last_became_active":"2023-08-25T09:49:34.627648+0000","last_became_peered":"2023-08-25T09:49:34.627648+0000","last_unstale":"2023-08-25T09:50:12.627168+0000","last_undegraded":"2023-08-25T09:50:12.627168+0000","last_fullsized":"2023-08-25T09:50:12.627168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T21:47:11.563460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557753+0000","last_change":"2023-08-25T09:49:34.614796+0000","last_active":"2023-08-25T09:50:12.557753+0000","last_peered":"2023-08-25T09:50:12.557753+0000","last_clean":"2023-08-25T09:50:12.557753+0000","last_became_active":"2023-08-25T09:49:34.614262+0000","last_became_peered":"2023-08-25T09:49:34.614262+0000","last_unstale":"2023-08-25T09:50:12.557753+0000","last_undegraded":"2023-08-25T09:50:12.557753+0000","last_fullsized":"2023-08-25T09:50:12.557753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T19:08:46.951943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.551870+0000","last_change":"2023-08-25T09:49:34.632511+0000","last_active":"2023-08-25T09:50:12.551870+0000","last_peered":"2023-08-25T09:50:12.551870+0000","last_clean":"2023-08-25T09:50:12.551870+0000","last_became_active":"2023-08-25T09:49:34.632210+0000","last_became_peered":"2023-08-25T09:49:34.632210+0000","last_unstale":"2023-08-25T09:50:12.551870+0000","last_undegraded":"2023-08-25T09:50:12.551870+0000","last_fullsized":"2023-08-25T09:50:12.551870+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T14:58:12.257947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.565670+0000","last_change":"2023-08-25T09:49:34.616778+0000","last_active":"2023-08-25T09:50:12.565670+0000","last_peered":"2023-08-25T09:50:12.565670+0000","last_clean":"2023-08-25T09:50:12.565670+0000","last_became_active":"2023-08-25T09:49:34.616494+0000","last_became_peered":"2023-08-25T09:49:34.616494+0000","last_unstale":"2023-08-25T09:50:12.565670+0000","last_undegraded":"2023-08-25T09:50:12.565670+0000","last_fullsized":"2023-08-25T09:50:12.565670+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T13:24:09.243873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557849+0000","last_change":"2023-08-25T09:49:34.614574+0000","last_active":"2023-08-25T09:50:12.557849+0000","last_peered":"2023-08-25T09:50:12.557849+0000","last_clean":"2023-08-25T09:50:12.557849+0000","last_became_active":"2023-08-25T09:49:34.614152+0000","last_became_peered":"2023-08-25T09:49:34.614152+0000","last_unstale":"2023-08-25T09:50:12.557849+0000","last_undegraded":"2023-08-25T09:50:12.557849+0000","last_fullsized":"2023-08-25T09:50:12.557849+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:33:53.261227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.556583+0000","last_change":"2023-08-25T09:49:34.621972+0000","last_active":"2023-08-25T09:50:12.556583+0000","last_peered":"2023-08-25T09:50:12.556583+0000","last_clean":"2023-08-25T09:50:12.556583+0000","last_became_active":"2023-08-25T09:49:34.621544+0000","last_became_peered":"2023-08-25T09:49:34.621544+0000","last_unstale":"2023-08-25T09:50:12.556583+0000","last_undegraded":"2023-08-25T09:50:12.556583+0000","last_fullsized":"2023-08-25T09:50:12.556583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T16:06:43.603905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-08-25T09:50:12.557609+0000","last_change":"2023-08-25T09:49:34.609056+0000","last_active":"2023-08-25T09:50:12.557609+0000","last_peered":"2023-08-25T09:50:12.557609+0000","last_clean":"2023-08-25T09:50:12.557609+0000","last_became_active":"2023-08-25T09:49:34.608655+0000","last_became_peered":"2023-08-25T09:49:34.608655+0000","last_unstale":"2023-08-25T09:50:12.557609+0000","last_undegraded":"2023-08-25T09:50:12.557609+0000","last_fullsized":"2023-08-25T09:50:12.557609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-25T09:49:33.559230+0000","last_clean_scrub_stamp":"2023-08-25T09:49:33.559230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-26T18:13:32.695366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":12023,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":217,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":24046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":191,"ondisk_log_size":191,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64910,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":147818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.4239999999999999}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.373,"15min":1.373},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":2.9159999999999999,"5min":2.9159999999999999,"15min":2.9159999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.4810000000000001}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.86799999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.641,"5min":1.641,"15min":1.641},"last":1.115},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.278}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.1019999999999999,"5min":3.1019999999999999,"15min":3.1019999999999999},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":1.083,"5min":1.083,"15min":1.083},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.244}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.825,"15min":1.825},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":7.2990000000000004,"5min":7.2990000000000004,"15min":7.2990000000000004},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"min":{"1min":1.101,"5min":1.101,"15min":1.101},"max":{"1min":4.875,"5min":4.875,"15min":4.875},"last":1.375}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.613,"15min":1.613},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":4.8079999999999998,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":1.331},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"min":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":1.5720000000000001}]}]},{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":446464,"data_stored":737126,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":0.85699999999999998}]},{"osd":2,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0249999999999999,"5min":2.0249999999999999,"15min":2.0249999999999999},"last":1.119},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":1.282}]},{"osd":3,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.256,"15min":1.256},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":0.997}]},{"osd":4,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":2.3220000000000001},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.3730000000000002,"5min":2.3730000000000002,"15min":2.3730000000000002},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":1.343}]},{"osd":6,"last update":"Fri Aug 25 09:50:23 2023","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4689999999999999,"5min":2.4689999999999999,"15min":2.4689999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.1439999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509460,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":140617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.754},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.1419999999999999}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.411}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":1.3520000000000001}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.246},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.92000000000000004}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.518},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.0780000000000001}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":1.474},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.383}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":3.399,"5min":3.399,"15min":3.399},"last":1.03},{"interface":"front","average":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"max":{"1min":2.7189999999999999,"5min":2.7189999999999999,"15min":2.7189999999999999},"last":1.3}]}]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":376832,"data_stored":144319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.032},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":1.4810000000000001}]},{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.26}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.222,"15min":1.222},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":1.8520000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":1.5289999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.355},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.1339999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.611}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":1.5329999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.232},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"min":{"1min":1.046,"5min":1.046,"15min":1.046},"max":{"1min":2.609,"5min":2.609,"15min":2.609},"last":1.3819999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.6919999999999999,"15min":1.6919999999999999},"min":{"1min":1.085,"5min":1.085,"15min":1.085},"max":{"1min":2.5329999999999999,"5min":2.5329999999999999,"15min":2.5329999999999999},"last":1.554},{"interface":"front","average":{"1min":1.726,"5min":1.726,"15min":1.726},"min":{"1min":1.242,"5min":1.242,"15min":1.242},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.456}]}]},{"osd":3,"up_from":14,"seq":60129542162,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":144091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Aug 25 09:50:18 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":1.8169999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.5499999999999998,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":1.103}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.034},{"interface":"front","average":{"1min":1.304,"5min":1.304,"15min":1.304},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.8290000000000002,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.155},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":1.1679999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.48,"5min":1.48,"15min":1.48},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.3959999999999999,"5min":2.3959999999999999,"15min":2.3959999999999999},"last":0.90300000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":140595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.843,"5min":1.843,"15min":1.843},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.095,"15min":1.095},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":0.56599999999999995}]},{"osd":3,"last update":"Fri Aug 25 09:50:17 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":0.67500000000000004}]},{"osd":4,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.2629999999999999,"5min":2.2629999999999999,"15min":2.2629999999999999},"last":0.95799999999999996}]},{"osd":5,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.353,"15min":1.353},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":2.113,"5min":2.113,"15min":2.113},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.76500000000000001}]},{"osd":7,"last update":"Fri Aug 25 09:50:22 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.92200000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":140505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.032},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":1.1719999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.075},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":3.5590000000000002,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.2689999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":3.8109999999999999,"5min":3.8109999999999999,"15min":3.8109999999999999},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.724,"5min":1.724,"15min":1.724},"last":0.90700000000000003}]},{"osd":5,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4239999999999999,"5min":2.4239999999999999,"15min":2.4239999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":3.7120000000000002},"last":1.609}]},{"osd":6,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.28,"15min":1.28},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.859,"5min":1.859,"15min":1.859},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.3819999999999999}]},{"osd":7,"last update":"Fri Aug 25 09:50:19 2023","interfaces":[{"interface":"back","average":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":1.53},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.135}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":454656,"data_stored":733751,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1781,"internal_metadata":31914251},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.96499999999999997}]},{"osd":2,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.6799999999999999,"5min":1.6799999999999999,"15min":1.6799999999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.0309999999999999}]},{"osd":4,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.105,"5min":2.105,"15min":2.105},"last":1.204},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":1.357}]},{"osd":5,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.512,"15min":1.512},"min":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.325}]},{"osd":6,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.006},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":2.5009999999999999,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.272}]},{"osd":7,"last update":"Fri Aug 25 09:50:20 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.51,"15min":1.51},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":2.778,"5min":2.778,"15min":2.778},"last":1.232},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":0.92400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32455,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":2878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":3586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":7313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":6253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-25T09:50:42.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:42.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.720+0000 7f6173f22640 1 Processor -- start 2023-08-25T09:50:42.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.721+0000 7f6173f22640 1 -- start start 2023-08-25T09:50:42.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.722+0000 7f6173f22640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f616c1510e0 0x7f616c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.722+0000 7f6173f22640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0c1dc0 con 0x7f616c0c3660 2023-08-25T09:50:42.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.722+0000 7f6173f22640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c156e00 con 0x7f616c0c3a40 2023-08-25T09:50:42.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.722+0000 7f6173f22640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c157a80 con 0x7f616c1510e0 2023-08-25T09:50:42.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.722+0000 7f6171cbe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f616c0c3660 0x7f616c06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43616/0 (socket says 172.21.15.71:43616) 2023-08-25T09:50:42.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.722+0000 7f6171cbe640 1 -- 172.21.15.71:0/1276854478 learned_addr learned my addr 172.21.15.71:0/1276854478 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:42.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.723+0000 7f61724bf640 1 --2- 172.21.15.71:0/1276854478 >> v2:172.21.15.71:3300/0 conn(0x7f616c1510e0 0x7f616c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.723+0000 7f6170cbc640 1 -- 172.21.15.71:0/1276854478 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050634109 0 0) 0x7f616c0c1dc0 con 0x7f616c0c3660 2023-08-25T09:50:42.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.723+0000 7f6170cbc640 1 -- 172.21.15.71:0/1276854478 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6158003520 con 0x7f616c0c3660 2023-08-25T09:50:42.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.723+0000 7f61724bf640 1 -- 172.21.15.71:0/1276854478 >> v1:172.21.15.164:6789/0 conn(0x7f616c0c3a40 legacy=0x7f616c1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.723+0000 7f61724bf640 1 -- 172.21.15.71:0/1276854478 >> v1:172.21.15.71:6789/0 conn(0x7f616c0c3660 legacy=0x7f616c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.723+0000 7f61724bf640 1 -- 172.21.15.71:0/1276854478 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c1570e0 con 0x7f616c1510e0 2023-08-25T09:50:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.724+0000 7f61724bf640 1 --2- 172.21.15.71:0/1276854478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c0c1710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6164009d90 tx=0x7f616402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79164051635a1082 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6170cbc640 1 -- 172.21.15.71:0/1276854478 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616402cd80 con 0x7f616c1510e0 2023-08-25T09:50:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6170cbc640 1 -- 172.21.15.71:0/1276854478 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6164034440 con 0x7f616c1510e0 2023-08-25T09:50:42.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6170cbc640 1 -- 172.21.15.71:0/1276854478 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640347d0 con 0x7f616c1510e0 2023-08-25T09:50:42.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 -- 172.21.15.71:0/1276854478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 msgr2=0x7f616c0c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 --2- 172.21.15.71:0/1276854478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c0c1710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6164009d90 tx=0x7f616402bfd0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 -- 172.21.15.71:0/1276854478 shutdown_connections 2023-08-25T09:50:42.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 --2- 172.21.15.71:0/1276854478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c0c1710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 -- 172.21.15.71:0/1276854478 >> 172.21.15.71:0/1276854478 conn(0x7f616c0654f0 msgr2=0x7f616c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:42.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 -- 172.21.15.71:0/1276854478 shutdown_connections 2023-08-25T09:50:42.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.725+0000 7f6173f22640 1 -- 172.21.15.71:0/1276854478 wait complete. 2023-08-25T09:50:42.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.727+0000 7f6173f22640 1 Processor -- start 2023-08-25T09:50:42.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.727+0000 7f6173f22640 1 -- start start 2023-08-25T09:50:42.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.728+0000 7f6173f22640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f616c0c3660 0x7f616c156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.728+0000 7f6173f22640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f616c0c3a40 0x7f616c158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.728+0000 7f6173f22640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.728+0000 7f6173f22640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c155df0 con 0x7f616c1510e0 2023-08-25T09:50:42.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.728+0000 7f6173f22640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c06a8f0 con 0x7f616c0c3a40 2023-08-25T09:50:42.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.728+0000 7f6173f22640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c06abd0 con 0x7f616c0c3660 2023-08-25T09:50:42.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52680/0 (socket says 172.21.15.71:52680) 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 -- 172.21.15.71:0/2053260034 learned_addr learned my addr 172.21.15.71:0/2053260034 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f6171cbe640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f616c0c3660 0x7f616c156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61714bd640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f616c0c3a40 0x7f616c158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 -- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f616c0c3660 msgr2=0x7f616c156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f616c0c3660 0x7f616c156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 -- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f616c0c3a40 msgr2=0x7f616c158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f616c0c3a40 0x7f616c158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 -- 172.21.15.71:0/2053260034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009a70 con 0x7f616c1510e0 2023-08-25T09:50:42.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.729+0000 7f61724bf640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c145fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6164009d90 tx=0x7f6164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.730+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164040650 con 0x7f616c1510e0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.730+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61640407b0 con 0x7f616c1510e0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.730+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164040b60 con 0x7f616c1510e0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.730+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c1554c0 con 0x7f616c1510e0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.730+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c155a80 con 0x7f616c1510e0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.731+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c155c40 con 0x7f616c1510e0 2023-08-25T09:50:42.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.732+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6164040cc0 con 0x7f616c1510e0 2023-08-25T09:50:42.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.733+0000 7f61627fc640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f615006a050 0x7f615006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:42.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.733+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f61640ab670 con 0x7f616c1510e0 2023-08-25T09:50:42.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.733+0000 7f6171cbe640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f615006a050 0x7f615006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:42.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.734+0000 7f6171cbe640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f615006a050 0x7f615006c510 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f61680011e0 tx=0x7f6168009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:42.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.737+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6164076b40 con 0x7f616c1510e0 2023-08-25T09:50:42.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.930+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f616c157390 con 0x7f616c1510e0 2023-08-25T09:50:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.933+0000 7f61627fc640 1 -- 172.21.15.71:0/2053260034 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7f6164009450 con 0x7f616c1510e0 2023-08-25T09:50:42.935 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:42.935 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:42.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.935+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f615006a050 msgr2=0x7f615006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.935+0000 7f6173f22640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f615006a050 0x7f615006c510 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f61680011e0 tx=0x7f6168009040 comp rx=0 tx=0).stop 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.935+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 msgr2=0x7f616c145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c145fb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6164009d90 tx=0x7f6164004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 shutdown_connections 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f616c0c3a40 0x7f616c158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f615006a050 0x7f615006c510 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f616c1510e0 0x7f616c145fb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 --2- 172.21.15.71:0/2053260034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f616c0c3660 0x7f616c156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 >> 172.21.15.71:0/2053260034 conn(0x7f616c0654f0 msgr2=0x7f616c0c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 shutdown_connections 2023-08-25T09:50:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:42.936+0000 7f6173f22640 1 -- 172.21.15.71:0/2053260034 wait complete. 2023-08-25T09:50:42.957 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T09:50:42.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T09:50:43.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.174+0000 7f18fdc14640 1 Processor -- start 2023-08-25T09:50:43.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.174+0000 7f18fdc14640 1 -- start start 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18fdc14640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f18f80c5f60 0x7f18f8155c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18fdc14640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18f80c6340 0x7f18f80c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18fdc14640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f8069850 0x7f18f814f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18fdc14640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f80c5760 con 0x7f18f80c6340 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18fdc14640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f81561a0 con 0x7f18f80c5f60 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18fdc14640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f806fa50 con 0x7f18f8069850 2023-08-25T09:50:43.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18f6ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18f80c6340 0x7f18f80c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18f6ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18f80c6340 0x7f18f80c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52682/0 (socket says 172.21.15.71:52682) 2023-08-25T09:50:43.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18f6ffd640 1 -- 172.21.15.71:0/4257228190 learned_addr learned my addr 172.21.15.71:0/4257228190 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:43.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18f77fe640 1 --2- 172.21.15.71:0/4257228190 >> v2:172.21.15.164:3300/0 conn(0x7f18f80c5f60 0x7f18f8155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.176+0000 7f18f7fff640 1 --2- 172.21.15.71:0/4257228190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f8069850 0x7f18f814f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.177+0000 7f18f6ffd640 1 -- 172.21.15.71:0/4257228190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f8069850 msgr2=0x7f18f814f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:43.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.177+0000 7f18f6ffd640 1 --2- 172.21.15.71:0/4257228190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f8069850 0x7f18f814f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.177+0000 7f18f6ffd640 1 -- 172.21.15.71:0/4257228190 >> v2:172.21.15.164:3300/0 conn(0x7f18f80c5f60 msgr2=0x7f18f8155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.177+0000 7f18f6ffd640 1 --2- 172.21.15.71:0/4257228190 >> v2:172.21.15.164:3300/0 conn(0x7f18f80c5f60 0x7f18f8155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.177+0000 7f18f6ffd640 1 -- 172.21.15.71:0/4257228190 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f806fee0 con 0x7f18f80c6340 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.177+0000 7f18f77fe640 1 --2- 172.21.15.71:0/4257228190 >> v2:172.21.15.164:3300/0 conn(0x7f18f80c5f60 0x7f18f8155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.178+0000 7f18f6ffd640 1 --2- 172.21.15.71:0/4257228190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c6340 0x7f18f80c4e60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18f8071e00 tx=0x7f18e002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e622811a2f4782 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.178+0000 7f18f67fc640 1 -- 172.21.15.71:0/4257228190 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0030850 con 0x7f18f80c6340 2023-08-25T09:50:43.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.178+0000 7f18f67fc640 1 -- 172.21.15.71:0/4257228190 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e00309b0 con 0x7f18f80c6340 2023-08-25T09:50:43.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.178+0000 7f18f67fc640 1 -- 172.21.15.71:0/4257228190 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0030d40 con 0x7f18f80c6340 2023-08-25T09:50:43.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 -- 172.21.15.71:0/4257228190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c6340 msgr2=0x7f18f80c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 --2- 172.21.15.71:0/4257228190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c6340 0x7f18f80c4e60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18f8071e00 tx=0x7f18e002b990 comp rx=0 tx=0).stop 2023-08-25T09:50:43.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 -- 172.21.15.71:0/4257228190 shutdown_connections 2023-08-25T09:50:43.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 --2- 172.21.15.71:0/4257228190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f8069850 0x7f18f814f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 --2- 172.21.15.71:0/4257228190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c6340 0x7f18f80c4e60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 --2- 172.21.15.71:0/4257228190 >> v2:172.21.15.164:3300/0 conn(0x7f18f80c5f60 0x7f18f8155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 -- 172.21.15.71:0/4257228190 >> 172.21.15.71:0/4257228190 conn(0x7f18f80654f0 msgr2=0x7f18f8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 -- 172.21.15.71:0/4257228190 shutdown_connections 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.179+0000 7f18fdc14640 1 -- 172.21.15.71:0/4257228190 wait complete. 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.180+0000 7f18fdc14640 1 Processor -- start 2023-08-25T09:50:43.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.181+0000 7f18fdc14640 1 -- start start 2023-08-25T09:50:43.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.181+0000 7f18fdc14640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f8069850 0x7f18f81ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.181+0000 7f18fdc14640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 0x7f18f8149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18fdc14640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f80c6340 0x7f18f814bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18fdc14640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8149590 con 0x7f18f80c5f60 2023-08-25T09:50:43.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18fdc14640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f81dec70 con 0x7f18f8069850 2023-08-25T09:50:43.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18fdc14640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f81e03c0 con 0x7f18f80c6340 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 0x7f18f8149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 0x7f18f8149aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52690/0 (socket says 172.21.15.71:52690) 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 -- 172.21.15.71:0/1875807377 learned_addr learned my addr 172.21.15.71:0/1875807377 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f7fff640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f80c6340 0x7f18f814bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f77fe640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f8069850 0x7f18f81ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 -- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f80c6340 msgr2=0x7f18f814bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f80c6340 0x7f18f814bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 -- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f8069850 msgr2=0x7f18f81ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f8069850 0x7f18f81ef4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f6ffd640 1 -- 172.21.15.71:0/1875807377 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00094d0 con 0x7f18f80c5f60 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.182+0000 7f18f7fff640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f80c6340 0x7f18f814bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.183+0000 7f18f6ffd640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 0x7f18f8149aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f18e002b9d0 tx=0x7f18e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.183+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0030840 con 0x7f18f80c5f60 2023-08-25T09:50:43.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.183+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e0039d90 con 0x7f18f80c5f60 2023-08-25T09:50:43.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.183+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0043650 con 0x7f18f80c5f60 2023-08-25T09:50:43.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.183+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f81efa30 con 0x7f18f80c5f60 2023-08-25T09:50:43.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.183+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f81de6b0 con 0x7f18f80c5f60 2023-08-25T09:50:43.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.184+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f8069160 con 0x7f18f80c5f60 2023-08-25T09:50:43.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.185+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f18e002e070 con 0x7f18f80c5f60 2023-08-25T09:50:43.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.186+0000 7f18effff640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f18d806a080 0x7f18d806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.186+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f18e00aaa30 con 0x7f18f80c5f60 2023-08-25T09:50:43.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.186+0000 7f18f77fe640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f18d806a080 0x7f18d806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.187+0000 7f18f77fe640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f18d806a080 0x7f18d806c540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f18e8007130 tx=0x7f18e8008460 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:43.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.190+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f18e0075de0 con 0x7f18f80c5f60 2023-08-25T09:50:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.378+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18f80c58a0 con 0x7f18f80c5f60 2023-08-25T09:50:43.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.379+0000 7f18effff640 1 -- 172.21.15.71:0/1875807377 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f18e0048020 con 0x7f18f80c5f60 2023-08-25T09:50:43.381 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T09:50:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f18d806a080 msgr2=0x7f18d806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f18d806a080 0x7f18d806c540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f18e8007130 tx=0x7f18e8008460 comp rx=0 tx=0).stop 2023-08-25T09:50:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 msgr2=0x7f18f8149aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 0x7f18f8149aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f18e002b9d0 tx=0x7f18e0004730 comp rx=0 tx=0).stop 2023-08-25T09:50:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 shutdown_connections 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f18d806a080 0x7f18d806c540 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18f80c6340 0x7f18f814bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18f80c5f60 0x7f18f8149aa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 --2- 172.21.15.71:0/1875807377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18f8069850 0x7f18f81ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 >> 172.21.15.71:0/1875807377 conn(0x7f18f80654f0 msgr2=0x7f18f814e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 shutdown_connections 2023-08-25T09:50:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.382+0000 7f18fdc14640 1 -- 172.21.15.71:0/1875807377 wait complete. 2023-08-25T09:50:43.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T09:50:43.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.619+0000 7f14ebf96640 1 Processor -- start 2023-08-25T09:50:43.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.620+0000 7f14ebf96640 1 -- start start 2023-08-25T09:50:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.621+0000 7f14ebf96640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f14e4150d60 0x7f14e40c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.621+0000 7f14ebf96640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e40c7690 con 0x7f14e406e400 2023-08-25T09:50:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.621+0000 7f14ebf96640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e40c7f30 con 0x7f14e41572f0 2023-08-25T09:50:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.621+0000 7f14ebf96640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e4151140 con 0x7f14e4150d60 2023-08-25T09:50:43.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.622+0000 7f14e9d32640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f14e4150d60 0x7f14e40c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.622+0000 7f14e9531640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f14e406e400 0x7f14e4156be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43626/0 (socket says 172.21.15.71:43626) 2023-08-25T09:50:43.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.622+0000 7f14e9531640 1 -- 172.21.15.71:0/3178947649 learned_addr learned my addr 172.21.15.71:0/3178947649 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:43.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.622+0000 7f14e9d32640 1 -- 172.21.15.71:0/3178947649 >> v1:172.21.15.164:6789/0 conn(0x7f14e41572f0 legacy=0x7f14e4161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:43.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.622+0000 7f14e9d32640 1 -- 172.21.15.71:0/3178947649 >> v1:172.21.15.71:6789/0 conn(0x7f14e406e400 legacy=0x7f14e4156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.622+0000 7f14e9d32640 1 -- 172.21.15.71:0/3178947649 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e406f2d0 con 0x7f14e4150d60 2023-08-25T09:50:43.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.623+0000 7f14e9d32640 1 --2- 172.21.15.71:0/3178947649 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e4150d60 0x7f14e40c8ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009d90 tx=0x7f14d802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0ddf50c14b7a64 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.624+0000 7f14e8d30640 1 -- 172.21.15.71:0/3178947649 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d802cbd0 con 0x7f14e4150d60 2023-08-25T09:50:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.624+0000 7f14e8d30640 1 -- 172.21.15.71:0/3178947649 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d802cd30 con 0x7f14e4150d60 2023-08-25T09:50:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.624+0000 7f14e8d30640 1 -- 172.21.15.71:0/3178947649 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d802cea0 con 0x7f14e4150d60 2023-08-25T09:50:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.624+0000 7f14ebf96640 1 -- 172.21.15.71:0/3178947649 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e4150d60 msgr2=0x7f14e40c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.624+0000 7f14ebf96640 1 --2- 172.21.15.71:0/3178947649 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e4150d60 0x7f14e40c8ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009d90 tx=0x7f14d802be20 comp rx=0 tx=0).stop 2023-08-25T09:50:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.625+0000 7f14ebf96640 1 -- 172.21.15.71:0/3178947649 shutdown_connections 2023-08-25T09:50:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.625+0000 7f14ebf96640 1 --2- 172.21.15.71:0/3178947649 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e4150d60 0x7f14e40c8ed0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.625+0000 7f14ebf96640 1 -- 172.21.15.71:0/3178947649 >> 172.21.15.71:0/3178947649 conn(0x7f14e40654f0 msgr2=0x7f14e4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.625+0000 7f14ebf96640 1 -- 172.21.15.71:0/3178947649 shutdown_connections 2023-08-25T09:50:43.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.625+0000 7f14ebf96640 1 -- 172.21.15.71:0/3178947649 wait complete. 2023-08-25T09:50:43.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.626+0000 7f14ebf96640 1 Processor -- start 2023-08-25T09:50:43.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.627+0000 7f14ebf96640 1 -- start start 2023-08-25T09:50:43.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.627+0000 7f14ebf96640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 0x7f14e414fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.627+0000 7f14ebf96640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e4150d60 0x7f14e4146e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.627+0000 7f14ebf96640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e41541e0 0x7f14e4147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14ebf96640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e4150340 con 0x7f14e406e400 2023-08-25T09:50:43.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14ebf96640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e41eeaa0 con 0x7f14e41541e0 2023-08-25T09:50:43.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14ebf96640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e4151780 con 0x7f14e4150d60 2023-08-25T09:50:43.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 0x7f14e414fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 0x7f14e414fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52712/0 (socket says 172.21.15.71:52712) 2023-08-25T09:50:43.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 -- 172.21.15.71:0/637743451 learned_addr learned my addr 172.21.15.71:0/637743451 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:43.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14ea533640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e41541e0 0x7f14e4147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9531640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e4150d60 0x7f14e4146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 -- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e4150d60 msgr2=0x7f14e4146e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e4150d60 0x7f14e4146e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 -- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e41541e0 msgr2=0x7f14e4147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e41541e0 0x7f14e4147340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9d32640 1 -- 172.21.15.71:0/637743451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8009a70 con 0x7f14e406e400 2023-08-25T09:50:43.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14ea533640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e41541e0 0x7f14e4147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:43.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.628+0000 7f14e9531640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e4150d60 0x7f14e4146e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:43.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.629+0000 7f14e9d32640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 0x7f14e414fe00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f14d8037040 tx=0x7f14d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:43.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.629+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d802cea0 con 0x7f14e406e400 2023-08-25T09:50:43.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.629+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d803e610 con 0x7f14e406e400 2023-08-25T09:50:43.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.629+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d803e9c0 con 0x7f14e406e400 2023-08-25T09:50:43.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.629+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e4151a60 con 0x7f14e406e400 2023-08-25T09:50:43.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.629+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e406bb20 con 0x7f14e406e400 2023-08-25T09:50:43.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.630+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e4065e30 con 0x7f14e406e400 2023-08-25T09:50:43.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.631+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f14d803b050 con 0x7f14e406e400 2023-08-25T09:50:43.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.633+0000 7f14e27fc640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14c806a050 0x7f14c806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:43.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.633+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f14d80ab190 con 0x7f14e406e400 2023-08-25T09:50:43.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.633+0000 7f14e9531640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14c806a050 0x7f14c806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:43.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.634+0000 7f14e9531640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14c806a050 0x7f14c806c510 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14e41ec6b0 tx=0x7f14d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:43.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.637+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f14d80765e0 con 0x7f14e406e400 2023-08-25T09:50:43.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.828+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f14e40c8530 con 0x7f14e406e400 2023-08-25T09:50:43.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.829+0000 7f14e27fc640 1 -- 172.21.15.71:0/637743451 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f14d8039020 con 0x7f14e406e400 2023-08-25T09:50:43.831 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T09:50:43.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14c806a050 msgr2=0x7f14c806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14c806a050 0x7f14c806c510 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14e41ec6b0 tx=0x7f14d0009040 comp rx=0 tx=0).stop 2023-08-25T09:50:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 msgr2=0x7f14e414fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 0x7f14e414fe00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f14d8037040 tx=0x7f14d8004a40 comp rx=0 tx=0).stop 2023-08-25T09:50:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 shutdown_connections 2023-08-25T09:50:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f14e41541e0 0x7f14e4147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14c806a050 0x7f14c806c510 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f14e4150d60 0x7f14e4146e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 --2- 172.21.15.71:0/637743451 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14e406e400 0x7f14e414fe00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 >> 172.21.15.71:0/637743451 conn(0x7f14e40654f0 msgr2=0x7f14e40caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 shutdown_connections 2023-08-25T09:50:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:43.832+0000 7f14ebf96640 1 -- 172.21.15.71:0/637743451 wait complete. 2023-08-25T09:50:43.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T09:50:44.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.070+0000 7f9c3c6a7640 1 Processor -- start 2023-08-25T09:50:44.072 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.070+0000 7f9c3c6a7640 1 -- start start 2023-08-25T09:50:44.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c3c6a7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9c3406cc70 0x7f9c340c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c3c6a7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3406a960 con 0x7f9c341594d0 2023-08-25T09:50:44.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c3c6a7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3406ac20 con 0x7f9c340c3940 2023-08-25T09:50:44.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c3c6a7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3406af00 con 0x7f9c3406cc70 2023-08-25T09:50:44.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c3a443640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9c341594d0 0x7f9c3414f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43630/0 (socket says 172.21.15.71:43630) 2023-08-25T09:50:44.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c3a443640 1 -- 172.21.15.71:0/582013193 learned_addr learned my addr 172.21.15.71:0/582013193 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:44.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.072+0000 7f9c39c42640 1 --2- 172.21.15.71:0/582013193 >> v2:172.21.15.71:3300/0 conn(0x7f9c3406cc70 0x7f9c340c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.073+0000 7f9c39441640 1 -- 172.21.15.71:0/582013193 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046587617 0 0) 0x7f9c3406a960 con 0x7f9c341594d0 2023-08-25T09:50:44.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.073+0000 7f9c39441640 1 -- 172.21.15.71:0/582013193 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c20005730 con 0x7f9c341594d0 2023-08-25T09:50:44.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.073+0000 7f9c39c42640 1 -- 172.21.15.71:0/582013193 >> v1:172.21.15.164:6789/0 conn(0x7f9c340c3940 legacy=0x7f9c341598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:44.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.073+0000 7f9c39c42640 1 -- 172.21.15.71:0/582013193 >> v1:172.21.15.71:6789/0 conn(0x7f9c341594d0 legacy=0x7f9c3414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.073+0000 7f9c39c42640 1 -- 172.21.15.71:0/582013193 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340c2e60 con 0x7f9c3406cc70 2023-08-25T09:50:44.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.074+0000 7f9c39c42640 1 --2- 172.21.15.71:0/582013193 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c4c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9c240097a0 tx=0x7f9c2402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d739ccba4a7c0aab server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.074+0000 7f9c39441640 1 -- 172.21.15.71:0/582013193 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24031830 con 0x7f9c3406cc70 2023-08-25T09:50:44.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.074+0000 7f9c39441640 1 -- 172.21.15.71:0/582013193 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c24031990 con 0x7f9c3406cc70 2023-08-25T09:50:44.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.074+0000 7f9c39441640 1 -- 172.21.15.71:0/582013193 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24031d20 con 0x7f9c3406cc70 2023-08-25T09:50:44.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/582013193 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 msgr2=0x7f9c340c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/582013193 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c4c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9c240097a0 tx=0x7f9c2402b8d0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/582013193 shutdown_connections 2023-08-25T09:50:44.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/582013193 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c4c90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/582013193 >> 172.21.15.71:0/582013193 conn(0x7f9c340654f0 msgr2=0x7f9c34065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:44.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/582013193 shutdown_connections 2023-08-25T09:50:44.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.075+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/582013193 wait complete. 2023-08-25T09:50:44.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.077+0000 7f9c3c6a7640 1 Processor -- start 2023-08-25T09:50:44.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.077+0000 7f9c3c6a7640 1 -- start start 2023-08-25T09:50:44.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3c6a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3c6a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c340c3940 0x7f9c3414b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3c6a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c341594d0 0x7f9c34069850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3c6a7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340c9be0 con 0x7f9c3406cc70 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3c6a7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340c9ec0 con 0x7f9c340c3940 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3c6a7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341f1790 con 0x7f9c341594d0 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52730/0 (socket says 172.21.15.71:52730) 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 -- 172.21.15.71:0/3570131092 learned_addr learned my addr 172.21.15.71:0/3570131092 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:44.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 -- 172.21.15.71:0/3570131092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c341594d0 msgr2=0x7f9c34069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c341594d0 0x7f9c34069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 -- 172.21.15.71:0/3570131092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c340c3940 msgr2=0x7f9c3414b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c340c3940 0x7f9c3414b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.078+0000 7f9c3a443640 1 -- 172.21.15.71:0/3570131092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24009400 con 0x7f9c3406cc70 2023-08-25T09:50:44.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.079+0000 7f9c3a443640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c80f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c3000c500 tx=0x7f9c3000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.079+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30005ce0 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.079+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c30005e40 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.079+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c30007330 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.079+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c341f1a70 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.079+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c340c8630 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.080+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c341f15f0 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.081+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9c30007550 con 0x7f9c3406cc70 2023-08-25T09:50:44.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.082+0000 7f9c2affd640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c1806a050 0x7f9c1806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.082+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f9c30087fc0 con 0x7f9c3406cc70 2023-08-25T09:50:44.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.083+0000 7f9c39c42640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c1806a050 0x7f9c1806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.083+0000 7f9c39c42640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c1806a050 0x7f9c1806c510 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c24036000 tx=0x7f9c24030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.087+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9c341f15f0 con 0x7f9c3406cc70 2023-08-25T09:50:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.274+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c341f15f0 con 0x7f9c3406cc70 2023-08-25T09:50:44.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.275+0000 7f9c2affd640 1 -- 172.21.15.71:0/3570131092 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v31) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c341f15f0 con 0x7f9c3406cc70 2023-08-25T09:50:44.277 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:44.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.277+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c1806a050 msgr2=0x7f9c1806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.277+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c1806a050 0x7f9c1806c510 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c24036000 tx=0x7f9c24030000 comp rx=0 tx=0).stop 2023-08-25T09:50:44.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.277+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 msgr2=0x7f9c340c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.277+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c80f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c3000c500 tx=0x7f9c3000c9c0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 shutdown_connections 2023-08-25T09:50:44.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c340c3940 0x7f9c3414b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c1806a050 0x7f9c1806c510 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c341594d0 0x7f9c34069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 --2- 172.21.15.71:0/3570131092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9c3406cc70 0x7f9c340c80f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 >> 172.21.15.71:0/3570131092 conn(0x7f9c340654f0 msgr2=0x7f9c340c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:44.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 shutdown_connections 2023-08-25T09:50:44.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.278+0000 7f9c3c6a7640 1 -- 172.21.15.71:0/3570131092 wait complete. 2023-08-25T09:50:44.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T09:50:44.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.506+0000 7fae5adbd640 1 Processor -- start 2023-08-25T09:50:44.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.506+0000 7fae5adbd640 1 -- start start 2023-08-25T09:50:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.507+0000 7fae5adbd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fae541542d0 0x7fae541576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.507+0000 7fae5adbd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae54069850 0x7fae540c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae5adbd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae54157bf0 con 0x7fae540c75c0 2023-08-25T09:50:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae5adbd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae540c4b60 con 0x7fae541542d0 2023-08-25T09:50:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae5adbd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5406a130 con 0x7fae54069850 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fae541542d0 0x7fae541576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fae541542d0 0x7fae541576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52732/0 (socket says 172.21.15.71:52732) 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 -- 172.21.15.71:0/1026967694 learned_addr learned my addr 172.21.15.71:0/1026967694 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 -- 172.21.15.71:0/1026967694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae54069850 msgr2=0x7fae540c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 --2- 172.21.15.71:0/1026967694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae54069850 0x7fae540c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 -- 172.21.15.71:0/1026967694 >> v1:172.21.15.164:6789/0 conn(0x7fae540c75c0 legacy=0x7fae540c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.508+0000 7fae59dbb640 1 -- 172.21.15.71:0/1026967694 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae541546b0 con 0x7fae541542d0 2023-08-25T09:50:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.509+0000 7fae59dbb640 1 --2- 172.21.15.71:0/1026967694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae541576b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae50009870 tx=0x7fae5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67bceb20a8ef4b8c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.510+0000 7fae58db9640 1 -- 172.21.15.71:0/1026967694 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50031640 con 0x7fae541542d0 2023-08-25T09:50:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.510+0000 7fae58db9640 1 -- 172.21.15.71:0/1026967694 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae500317a0 con 0x7fae541542d0 2023-08-25T09:50:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.510+0000 7fae58db9640 1 -- 172.21.15.71:0/1026967694 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50031b30 con 0x7fae541542d0 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.510+0000 7fae5adbd640 1 -- 172.21.15.71:0/1026967694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 msgr2=0x7fae541576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.510+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1026967694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae541576b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae50009870 tx=0x7fae5002b8f0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.511+0000 7fae5adbd640 1 -- 172.21.15.71:0/1026967694 shutdown_connections 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.511+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1026967694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae54069850 0x7fae540c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.511+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1026967694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae541576b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.511+0000 7fae5adbd640 1 -- 172.21.15.71:0/1026967694 >> 172.21.15.71:0/1026967694 conn(0x7fae5406c280 msgr2=0x7fae5406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:44.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.511+0000 7fae5adbd640 1 -- 172.21.15.71:0/1026967694 shutdown_connections 2023-08-25T09:50:44.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.511+0000 7fae5adbd640 1 -- 172.21.15.71:0/1026967694 wait complete. 2023-08-25T09:50:44.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.512+0000 7fae5adbd640 1 Processor -- start 2023-08-25T09:50:44.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 -- start start 2023-08-25T09:50:44.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae54069850 0x7fae5414b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae540c75c0 0x7fae541ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae54148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540c3440 con 0x7fae541542d0 2023-08-25T09:50:44.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae541f1800 con 0x7fae54069850 2023-08-25T09:50:44.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.513+0000 7fae5adbd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae541f2060 con 0x7fae540c75c0 2023-08-25T09:50:44.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae54148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae54148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52744/0 (socket says 172.21.15.71:52744) 2023-08-25T09:50:44.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 -- 172.21.15.71:0/1616426126 learned_addr learned my addr 172.21.15.71:0/1616426126 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:44.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae59dbb640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae54069850 0x7fae5414b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae595ba640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae540c75c0 0x7fae541ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 -- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae540c75c0 msgr2=0x7fae541ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae540c75c0 0x7fae541ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 -- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae54069850 msgr2=0x7fae5414b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae54069850 0x7fae5414b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 -- 172.21.15.71:0/1616426126 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae500094d0 con 0x7fae541542d0 2023-08-25T09:50:44.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae59dbb640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae54069850 0x7fae5414b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:44.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae595ba640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae540c75c0 0x7fae541ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:44.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.514+0000 7fae5a5bc640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae54148eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c780 tx=0x7fae4c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.515+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c004b60 con 0x7fae541542d0 2023-08-25T09:50:44.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.515+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c00ce90 con 0x7fae541542d0 2023-08-25T09:50:44.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.515+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c00a850 con 0x7fae541542d0 2023-08-25T09:50:44.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.515+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae540c0b00 con 0x7fae541542d0 2023-08-25T09:50:44.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.515+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae541f1a10 con 0x7fae541542d0 2023-08-25T09:50:44.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.516+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae540c41c0 con 0x7fae541542d0 2023-08-25T09:50:44.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.517+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fae4c00a9b0 con 0x7fae541542d0 2023-08-25T09:50:44.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.518+0000 7fae4a7fc640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fae3806a050 0x7fae3806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.518+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7fae4c088bd0 con 0x7fae541542d0 2023-08-25T09:50:44.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.518+0000 7fae59dbb640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fae3806a050 0x7fae3806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.519+0000 7fae59dbb640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fae3806a050 0x7fae3806c510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fae50036000 tx=0x7fae5003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.522+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fae4c053ef0 con 0x7fae541542d0 2023-08-25T09:50:44.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.711+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae5404e420 con 0x7fae541542d0 2023-08-25T09:50:44.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.712+0000 7fae4a7fc640 1 -- 172.21.15.71:0/1616426126 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v31) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae4c057510 con 0x7fae541542d0 2023-08-25T09:50:44.714 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T09:50:44.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.714+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fae3806a050 msgr2=0x7fae3806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.714+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fae3806a050 0x7fae3806c510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fae50036000 tx=0x7fae5003c040 comp rx=0 tx=0).stop 2023-08-25T09:50:44.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.714+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 msgr2=0x7fae54148eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.714+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae54148eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c780 tx=0x7fae4c00cc40 comp rx=0 tx=0).stop 2023-08-25T09:50:44.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 shutdown_connections 2023-08-25T09:50:44.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fae3806a050 0x7fae3806c510 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fae541542d0 0x7fae54148eb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae540c75c0 0x7fae541ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 --2- 172.21.15.71:0/1616426126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae54069850 0x7fae5414b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 >> 172.21.15.71:0/1616426126 conn(0x7fae5406c280 msgr2=0x7fae54151170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:44.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 shutdown_connections 2023-08-25T09:50:44.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.715+0000 7fae5adbd640 1 -- 172.21.15.71:0/1616426126 wait complete. 2023-08-25T09:50:44.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T09:50:44.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.945+0000 7f30db312640 1 Processor -- start 2023-08-25T09:50:44.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.946+0000 7f30db312640 1 -- start start 2023-08-25T09:50:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.947+0000 7f30db312640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f30d41594d0 0x7f30d40c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.947+0000 7f30db312640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f30d406cc70 0x7f30d414f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.947+0000 7f30db312640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d40c3940 0x7f30d414fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.947+0000 7f30db312640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d406a960 con 0x7f30d406cc70 2023-08-25T09:50:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.947+0000 7f30db312640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d406ac20 con 0x7f30d41594d0 2023-08-25T09:50:44.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.947+0000 7f30db312640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d406af00 con 0x7f30d40c3940 2023-08-25T09:50:44.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f30d406cc70 0x7f30d414f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f30d406cc70 0x7f30d414f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52760/0 (socket says 172.21.15.71:52760) 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 -- 172.21.15.71:0/3900703143 learned_addr learned my addr 172.21.15.71:0/3900703143 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 -- 172.21.15.71:0/3900703143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d40c3940 msgr2=0x7f30d414fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d90ae640 1 --2- 172.21.15.71:0/3900703143 >> v2:172.21.15.164:3300/0 conn(0x7f30d41594d0 0x7f30d40c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d98af640 1 --2- 172.21.15.71:0/3900703143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d40c3940 0x7f30d414fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 --2- 172.21.15.71:0/3900703143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d40c3940 0x7f30d414fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 -- 172.21.15.71:0/3900703143 >> v2:172.21.15.164:3300/0 conn(0x7f30d41594d0 msgr2=0x7f30d40c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:44.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 --2- 172.21.15.71:0/3900703143 >> v2:172.21.15.164:3300/0 conn(0x7f30d41594d0 0x7f30d40c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.948+0000 7f30d88ad640 1 -- 172.21.15.71:0/3900703143 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d40c2e60 con 0x7f30d406cc70 2023-08-25T09:50:44.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.949+0000 7f30d88ad640 1 --2- 172.21.15.71:0/3900703143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d406cc70 0x7f30d414f850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f30d4071d10 tx=0x7f30c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb134be5937093d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.950+0000 7f30cbfff640 1 -- 172.21.15.71:0/3900703143 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4031870 con 0x7f30d406cc70 2023-08-25T09:50:44.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.950+0000 7f30cbfff640 1 -- 172.21.15.71:0/3900703143 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c40319d0 con 0x7f30d406cc70 2023-08-25T09:50:44.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.950+0000 7f30cbfff640 1 -- 172.21.15.71:0/3900703143 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4031d60 con 0x7f30d406cc70 2023-08-25T09:50:44.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.950+0000 7f30db312640 1 -- 172.21.15.71:0/3900703143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d406cc70 msgr2=0x7f30d414f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.950+0000 7f30db312640 1 --2- 172.21.15.71:0/3900703143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d406cc70 0x7f30d414f850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f30d4071d10 tx=0x7f30c402b910 comp rx=0 tx=0).stop 2023-08-25T09:50:44.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 -- 172.21.15.71:0/3900703143 shutdown_connections 2023-08-25T09:50:44.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 --2- 172.21.15.71:0/3900703143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d40c3940 0x7f30d414fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 --2- 172.21.15.71:0/3900703143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d406cc70 0x7f30d414f850 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 --2- 172.21.15.71:0/3900703143 >> v2:172.21.15.164:3300/0 conn(0x7f30d41594d0 0x7f30d40c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 -- 172.21.15.71:0/3900703143 >> 172.21.15.71:0/3900703143 conn(0x7f30d40654f0 msgr2=0x7f30d4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:44.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 -- 172.21.15.71:0/3900703143 shutdown_connections 2023-08-25T09:50:44.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.951+0000 7f30db312640 1 -- 172.21.15.71:0/3900703143 wait complete. 2023-08-25T09:50:44.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.952+0000 7f30db312640 1 Processor -- start 2023-08-25T09:50:44.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.953+0000 7f30db312640 1 -- start start 2023-08-25T09:50:44.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.953+0000 7f30db312640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d406cc70 0x7f30d41e9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30db312640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30d40c3940 0x7f30d4146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30db312640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 0x7f30d4158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30db312640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40c9b20 con 0x7f30d41594d0 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30db312640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4157bc0 con 0x7f30d40c3940 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30db312640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4147fa0 con 0x7f30d406cc70 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 0x7f30d4158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 0x7f30d4158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52762/0 (socket says 172.21.15.71:52762) 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 -- 172.21.15.71:0/3955740529 learned_addr learned my addr 172.21.15.71:0/3955740529 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d88ad640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30d40c3940 0x7f30d4146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d90ae640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d406cc70 0x7f30d41e9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 -- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d406cc70 msgr2=0x7f30d41e9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d406cc70 0x7f30d41e9a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 -- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30d40c3940 msgr2=0x7f30d4146f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30d40c3940 0x7f30d4146f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.954+0000 7f30d98af640 1 -- 172.21.15.71:0/3955740529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40094d0 con 0x7f30d41594d0 2023-08-25T09:50:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.955+0000 7f30d98af640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 0x7f30d4158010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00cb80 tx=0x7f30cc00a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.955+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc00ce70 con 0x7f30d41594d0 2023-08-25T09:50:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.955+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc00abd0 con 0x7f30d41594d0 2023-08-25T09:50:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.955+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc011410 con 0x7f30d41594d0 2023-08-25T09:50:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.955+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d4148280 con 0x7f30d41594d0 2023-08-25T09:50:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.955+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d4148840 con 0x7f30d41594d0 2023-08-25T09:50:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.956+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d4157d30 con 0x7f30d41594d0 2023-08-25T09:50:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.957+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f30cc011570 con 0x7f30d41594d0 2023-08-25T09:50:44.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.958+0000 7f30c9ffb640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f30b806a050 0x7f30b806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:44.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.958+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f30cc088790 con 0x7f30d41594d0 2023-08-25T09:50:44.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.958+0000 7f30d90ae640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f30b806a050 0x7f30b806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:44.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.959+0000 7f30d90ae640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f30b806a050 0x7f30b806c510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30d0006fd0 tx=0x7f30d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:44.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:44.964+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f30d4157d30 con 0x7f30d41594d0 2023-08-25T09:50:45.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.138+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f30d4157d30 con 0x7f30d41594d0 2023-08-25T09:50:45.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.139+0000 7f30c9ffb640 1 -- 172.21.15.71:0/3955740529 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) v1 ==== 103+0+11 (secure 0 0 0) 0x7f30d4157d30 con 0x7f30d41594d0 2023-08-25T09:50:45.141 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T09:50:45.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f30b806a050 msgr2=0x7f30b806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f30b806a050 0x7f30b806c510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f30d0006fd0 tx=0x7f30d0008040 comp rx=0 tx=0).stop 2023-08-25T09:50:45.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 msgr2=0x7f30d4158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 0x7f30d4158010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00cb80 tx=0x7f30cc00a4c0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 shutdown_connections 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30d40c3940 0x7f30d4146f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f30b806a050 0x7f30b806c510 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30d41594d0 0x7f30d4158010 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 --2- 172.21.15.71:0/3955740529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30d406cc70 0x7f30d41e9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 >> 172.21.15.71:0/3955740529 conn(0x7f30d40654f0 msgr2=0x7f30d40c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 shutdown_connections 2023-08-25T09:50:45.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.142+0000 7f30db312640 1 -- 172.21.15.71:0/3955740529 wait complete. 2023-08-25T09:50:45.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:50:45.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.360+0000 7f25c5883640 1 Processor -- start 2023-08-25T09:50:45.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.360+0000 7f25c5883640 1 -- start start 2023-08-25T09:50:45.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.361+0000 7f25c5883640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 0x7f25c00c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.361+0000 7f25c5883640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c006a960 con 0x7f25c00c3940 2023-08-25T09:50:45.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.361+0000 7f25c5883640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c006ac20 con 0x7f25c01594d0 2023-08-25T09:50:45.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.361+0000 7f25c5883640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c006af00 con 0x7f25c006cc70 2023-08-25T09:50:45.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.362+0000 7f25bf7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f25c00c3940 0x7f25c01598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43640/0 (socket says 172.21.15.71:43640) 2023-08-25T09:50:45.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.362+0000 7f25bf7fe640 1 -- 172.21.15.71:0/3792640523 learned_addr learned my addr 172.21.15.71:0/3792640523 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:50:45.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.362+0000 7f25b7fff640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 0x7f25c00c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.362+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888861771 0 0) 0x7f25c006a960 con 0x7f25c00c3940 2023-08-25T09:50:45.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.362+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac003520 con 0x7f25c00c3940 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169951867 0 0) 0x7f25ac003520 con 0x7f25c00c3940 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 msgr2=0x7f25c00c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 0x7f25c00c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 >> v1:172.21.15.164:6789/0 conn(0x7f25c01594d0 legacy=0x7f25c014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00c2e60 con 0x7f25c00c3940 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f25b80019f0 con 0x7f25c00c3940 2023-08-25T09:50:45.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 >> v1:172.21.15.71:6789/0 conn(0x7f25c00c3940 legacy=0x7f25c01598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 0x7f25ac003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25ac003f90 0x7f25ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25ac004d00 0x7f25ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c006af00 con 0x7f25ac004d00 2023-08-25T09:50:45.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac00dcf0 con 0x7f25ac003f90 2023-08-25T09:50:45.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac00deb0 con 0x7f25c006cc70 2023-08-25T09:50:45.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25ac004d00 0x7f25ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25beffd640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 0x7f25ac003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25b7fff640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25ac003f90 0x7f25ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25bf7fe640 1 -- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 msgr2=0x7f25ac003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:45.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25c006cc70 0x7f25ac003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25bf7fe640 1 -- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25ac003f90 msgr2=0x7f25ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:45.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25ac003f90 0x7f25ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.363+0000 7f25bf7fe640 1 -- 172.21.15.71:0/3792640523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac003520 con 0x7f25ac004d00 2023-08-25T09:50:45.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.364+0000 7f25b7fff640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25ac003f90 0x7f25ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:45.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25ac004d00 0x7f25ac005180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f25b800a830 tx=0x7f25ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3a68f18c974f82 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:45.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80072d0 con 0x7f25ac004d00 2023-08-25T09:50:45.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b802f070 con 0x7f25ac004d00 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25be7fc640 1 -- 172.21.15.71:0/3792640523 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80076f0 con 0x7f25ac004d00 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25c5883640 1 -- 172.21.15.71:0/3792640523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25ac004d00 msgr2=0x7f25ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25c5883640 1 --2- 172.21.15.71:0/3792640523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25ac004d00 0x7f25ac005180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f25b800a830 tx=0x7f25ac0057f0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25bf7fe640 1 -- 172.21.15.71:0/3792640523 reap_dead start 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25c5883640 1 -- 172.21.15.71:0/3792640523 shutdown_connections 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25c5883640 1 -- 172.21.15.71:0/3792640523 >> 172.21.15.71:0/3792640523 conn(0x7f25c00654f0 msgr2=0x7f25c0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:45.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.365+0000 7f25c5883640 1 -- 172.21.15.71:0/3792640523 shutdown_connections 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.366+0000 7f25c5883640 1 -- 172.21.15.71:0/3792640523 wait complete. 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.367+0000 7f25c5883640 1 Processor -- start 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.367+0000 7f25c5883640 1 -- start start 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25c5883640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25ac004d00 0x7f25c006c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25c5883640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25c006fcd0 0x7f25c01f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25c5883640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 0x7f25c01e1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25c5883640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00c8d70 con 0x7f25c01f1ae0 2023-08-25T09:50:45.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25c5883640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00c9200 con 0x7f25c006fcd0 2023-08-25T09:50:45.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25c5883640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00c84b0 con 0x7f25ac004d00 2023-08-25T09:50:45.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.368+0000 7f25bf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 0x7f25c01e1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 0x7f25c01e1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52790/0 (socket says 172.21.15.71:52790) 2023-08-25T09:50:45.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 -- 172.21.15.71:0/2709413925 learned_addr learned my addr 172.21.15.71:0/2709413925 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:45.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25beffd640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25ac004d00 0x7f25c006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25b7fff640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25c006fcd0 0x7f25c01f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 -- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25ac004d00 msgr2=0x7f25c006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25ac004d00 0x7f25c006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 -- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25c006fcd0 msgr2=0x7f25c01f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25c006fcd0 0x7f25c01f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 -- 172.21.15.71:0/2709413925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00c97d0 con 0x7f25c01f1ae0 2023-08-25T09:50:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bf7fe640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 0x7f25c01e1900 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f25b800a830 tx=0x7f25b8001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.369+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80074e0 con 0x7f25c01f1ae0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.370+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b8007640 con 0x7f25c01f1ae0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.370+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80079b0 con 0x7f25c01f1ae0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.370+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00c8720 con 0x7f25c01f1ae0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.370+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c00c9a70 con 0x7f25c01f1ae0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.371+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c00c8f20 con 0x7f25c01f1ae0 2023-08-25T09:50:45.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.372+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f25b8031030 con 0x7f25c01f1ae0 2023-08-25T09:50:45.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.372+0000 7f25bcff9640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f25a006a050 0x7f25a006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.373+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f25b80aa580 con 0x7f25c01f1ae0 2023-08-25T09:50:45.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.373+0000 7f25beffd640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f25a006a050 0x7f25a006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.373+0000 7f25beffd640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f25a006a050 0x7f25a006c510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f25b0007fd0 tx=0x7f25b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:45.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.377+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f25c00c8f20 con 0x7f25c01f1ae0 2023-08-25T09:50:45.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.603+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25c00c8f20 con 0x7f25c01f1ae0 2023-08-25T09:50:45.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.605+0000 7f25bcff9640 1 -- 172.21.15.71:0/2709413925 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+6523 (secure 0 0 0) 0x7f25c00c8f20 con 0x7f25c01f1ae0 2023-08-25T09:50:45.607 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:45.607 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:50:23.760465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:50:45.608 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 17 2023-08-25T09:50:45.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.607+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f25a006a050 msgr2=0x7f25a006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.607+0000 7f25c5883640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f25a006a050 0x7f25a006c510 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f25b0007fd0 tx=0x7f25b0009040 comp rx=0 tx=0).stop 2023-08-25T09:50:45.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.607+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 msgr2=0x7f25c01e1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.607+0000 7f25c5883640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 0x7f25c01e1900 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f25b800a830 tx=0x7f25b8001d80 comp rx=0 tx=0).stop 2023-08-25T09:50:45.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 shutdown_connections 2023-08-25T09:50:45.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25c006fcd0 0x7f25c01f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f25a006a050 0x7f25a006c510 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25c01f1ae0 0x7f25c01e1900 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 --2- 172.21.15.71:0/2709413925 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25ac004d00 0x7f25c006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 >> 172.21.15.71:0/2709413925 conn(0x7f25c00654f0 msgr2=0x7f25c014d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:45.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 shutdown_connections 2023-08-25T09:50:45.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.608+0000 7f25c5883640 1 -- 172.21.15.71:0/2709413925 wait complete. 2023-08-25T09:50:45.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T09:50:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.830+0000 7f6d1cc99640 1 Processor -- start 2023-08-25T09:50:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.830+0000 7f6d1cc99640 1 -- start start 2023-08-25T09:50:45.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d1cc99640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3a40 0x7f6d180c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d1cc99640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6d181509d0 0x7f6d18150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d1cc99640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180c1dc0 con 0x7f6d180c3660 2023-08-25T09:50:45.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d1cc99640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d18157a80 con 0x7f6d181509d0 2023-08-25T09:50:45.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d1cc99640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181513a0 con 0x7f6d180c3a40 2023-08-25T09:50:45.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d16d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6d181509d0 0x7f6d18150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d16d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6d181509d0 0x7f6d18150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52794/0 (socket says 172.21.15.71:52794) 2023-08-25T09:50:45.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.832+0000 7f6d16d9d640 1 -- 172.21.15.71:0/2349411140 learned_addr learned my addr 172.21.15.71:0/2349411140 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:45.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.833+0000 7f6d0ffff640 1 --2- 172.21.15.71:0/2349411140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3a40 0x7f6d180c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.833+0000 7f6d16d9d640 1 -- 172.21.15.71:0/2349411140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3a40 msgr2=0x7f6d180c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:45.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.833+0000 7f6d16d9d640 1 --2- 172.21.15.71:0/2349411140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3a40 0x7f6d180c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.833+0000 7f6d16d9d640 1 -- 172.21.15.71:0/2349411140 >> v1:172.21.15.164:6789/0 conn(0x7f6d180c3660 legacy=0x7f6d1806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:50:45.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.833+0000 7f6d16d9d640 1 -- 172.21.15.71:0/2349411140 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18156e00 con 0x7f6d181509d0 2023-08-25T09:50:45.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.834+0000 7f6d16d9d640 1 --2- 172.21.15.71:0/2349411140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d181509d0 0x7f6d18150e60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009e50 tx=0x7f6d1002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4ce4fed1690e6b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:45.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.834+0000 7f6d15d9b640 1 -- 172.21.15.71:0/2349411140 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1002cb00 con 0x7f6d181509d0 2023-08-25T09:50:45.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.834+0000 7f6d15d9b640 1 -- 172.21.15.71:0/2349411140 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1002cc60 con 0x7f6d181509d0 2023-08-25T09:50:45.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.834+0000 7f6d15d9b640 1 -- 172.21.15.71:0/2349411140 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100343f0 con 0x7f6d181509d0 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 -- 172.21.15.71:0/2349411140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d181509d0 msgr2=0x7f6d18150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/2349411140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d181509d0 0x7f6d18150e60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009e50 tx=0x7f6d1002bf60 comp rx=0 tx=0).stop 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 -- 172.21.15.71:0/2349411140 shutdown_connections 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/2349411140 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d181509d0 0x7f6d18150e60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/2349411140 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3a40 0x7f6d180c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 -- 172.21.15.71:0/2349411140 >> 172.21.15.71:0/2349411140 conn(0x7f6d180654f0 msgr2=0x7f6d18065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:45.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 -- 172.21.15.71:0/2349411140 shutdown_connections 2023-08-25T09:50:45.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.835+0000 7f6d1cc99640 1 -- 172.21.15.71:0/2349411140 wait complete. 2023-08-25T09:50:45.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.837+0000 7f6d1cc99640 1 Processor -- start 2023-08-25T09:50:45.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.838+0000 7f6d1cc99640 1 -- start start 2023-08-25T09:50:45.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1cc99640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3660 0x7f6d18145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1cc99640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 0x7f6d18145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1cc99640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d181509d0 0x7f6d181ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1cc99640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18149f30 con 0x7f6d180c3a40 2023-08-25T09:50:45.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1cc99640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181ed7d0 con 0x7f6d181509d0 2023-08-25T09:50:45.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1cc99640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18147d00 con 0x7f6d180c3660 2023-08-25T09:50:45.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 0x7f6d18145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d0ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 0x7f6d18145f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52796/0 (socket says 172.21.15.71:52796) 2023-08-25T09:50:45.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d0ffff640 1 -- 172.21.15.71:0/159387288 learned_addr learned my addr 172.21.15.71:0/159387288 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:50:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d1659c640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3660 0x7f6d18145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.839+0000 7f6d16d9d640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d181509d0 0x7f6d181ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0ffff640 1 -- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3660 msgr2=0x7f6d18145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0ffff640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3660 0x7f6d18145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0ffff640 1 -- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d181509d0 msgr2=0x7f6d181ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0ffff640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d181509d0 0x7f6d181ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0ffff640 1 -- 172.21.15.71:0/159387288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10009b30 con 0x7f6d180c3a40 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d16d9d640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d181509d0 0x7f6d181ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0ffff640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 0x7f6d18145f80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000d620 tx=0x7f6d0000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d000125f0 con 0x7f6d180c3a40 2023-08-25T09:50:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d00012750 con 0x7f6d180c3a40 2023-08-25T09:50:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d18149a00 con 0x7f6d180c3a40 2023-08-25T09:50:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00012b00 con 0x7f6d180c3a40 2023-08-25T09:50:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.840+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d18149c50 con 0x7f6d180c3a40 2023-08-25T09:50:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.842+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d180c29c0 con 0x7f6d180c3a40 2023-08-25T09:50:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.843+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6d00012c60 con 0x7f6d180c3a40 2023-08-25T09:50:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.843+0000 7f6d0effd640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6cf806a080 0x7f6cf806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:50:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.844+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6871+0+0 (secure 0 0 0) 0x7f6d00088c60 con 0x7f6d180c3a40 2023-08-25T09:50:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.844+0000 7f6d1659c640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6cf806a080 0x7f6cf806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:50:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.845+0000 7f6d1659c640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6cf806a080 0x7f6cf806c540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d08006fd0 tx=0x7f6d08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:50:45.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:45.848+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6d00054010 con 0x7f6d180c3a40 2023-08-25T09:50:46.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.037+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d180c97a0 con 0x7f6d180c3a40 2023-08-25T09:50:46.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.039+0000 7f6d0effd640 1 -- 172.21.15.71:0/159387288 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+20215 (secure 0 0 0) 0x7f6d00057630 con 0x7f6d180c3a40 2023-08-25T09:50:46.041 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:50:46.042 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":31,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:50:12.530283+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T09:50:46.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.041+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6cf806a080 msgr2=0x7f6cf806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:46.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.041+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6cf806a080 0x7f6cf806c540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d08006fd0 tx=0x7f6d08008040 comp rx=0 tx=0).stop 2023-08-25T09:50:46.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.041+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 msgr2=0x7f6d18145f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:50:46.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 0x7f6d18145f80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6d0000d620 tx=0x7f6d0000dae0 comp rx=0 tx=0).stop 2023-08-25T09:50:46.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 shutdown_connections 2023-08-25T09:50:46.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d181509d0 0x7f6d181ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:46.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6cf806a080 0x7f6cf806c540 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:46.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6d180c3a40 0x7f6d18145f80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:46.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 --2- 172.21.15.71:0/159387288 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d180c3660 0x7f6d18145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:50:46.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 >> 172.21.15.71:0/159387288 conn(0x7f6d180654f0 msgr2=0x7f6d180c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:50:46.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 shutdown_connections 2023-08-25T09:50:46.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:50:46.042+0000 7f6d1cc99640 1 -- 172.21.15.71:0/159387288 wait complete. 2023-08-25T09:50:46.062 DEBUG:tasks.mds_thrash:Yielding 2023-08-25T09:50:46.063 INFO:teuthology.run_tasks:Running task workunit... 2023-08-25T09:50:46.075 INFO:tasks.workunit:Pulling workunits from ref 4b369cf18ed1391a426ab4ae86da834e9c074f81 2023-08-25T09:50:46.076 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-08-25T09:50:46.076 INFO:tasks.workunit:timeout=6h 2023-08-25T09:50:46.077 INFO:tasks.workunit:cleanup=True 2023-08-25T09:50:46.077 DEBUG:teuthology.orchestra.run.smithi071:> stat -- /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:46.078 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-08-25T09:50:46.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 86 secs before thrashing 2023-08-25T09:50:46.134 INFO:teuthology.orchestra.run.smithi071.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:46.134 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-08-25T09:50:46.134 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 27h/39d Inode: 1 Links: 2 2023-08-25T09:50:46.135 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-25T09:50:46.135 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:fusefs_t:s0 2023-08-25T09:50:46.135 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-08-25T09:50:46.135 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2023-08-25 09:49:39.614531423 +0000 2023-08-25T09:50:46.136 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2023-08-25 09:49:51.200276759 +0000 2023-08-25T09:50:46.136 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2023-08-25T09:50:46.136 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-08-25T09:50:46.136 DEBUG:teuthology.orchestra.run.smithi071:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-08-25T09:50:46.255 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/rishabh-d-dave/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4b369cf18ed1391a426ab4ae86da834e9c074f81 2023-08-25T09:50:46.317 INFO:tasks.workunit.client.0.smithi071.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-08-25T09:50:49.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:50:49.489 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:51:19.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:51:19.622 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:51:28.432 INFO:tasks.workunit.client.0.smithi071.stderr:Note: switching to '4b369cf18ed1391a426ab4ae86da834e9c074f81'. 2023-08-25T09:51:28.432 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr:changes and commit them, and you can discard any commits you make in this 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr:state without impacting any branches by switching back to a branch. 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr:If you want to create a new branch to retain commits you create, you may 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr:do so (now or later) by using -c with the switch command. Example: 2023-08-25T09:51:28.433 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr: git switch -c 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr:Or undo this operation with: 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr: git switch - 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-08-25T09:51:28.434 INFO:tasks.workunit.client.0.smithi071.stderr: 2023-08-25T09:51:28.435 INFO:tasks.workunit.client.0.smithi071.stderr:HEAD is now at 4b369cf18ed qa: inherit RunCephCmd in CephTestCase instead of CephFSTestCase 2023-08-25T09:51:28.446 DEBUG:teuthology.orchestra.run.smithi071:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-08-25T09:51:28.502 INFO:tasks.workunit.client.0.smithi071.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-08-25T09:51:28.509 INFO:tasks.workunit.client.0.smithi071.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-08-25T09:51:28.509 INFO:tasks.workunit.client.0.smithi071.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-08-25T09:51:29.176 INFO:tasks.workunit.client.0.smithi071.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-08-25T09:51:29.291 INFO:tasks.workunit.client.0.smithi071.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-08-25T09:51:29.378 INFO:tasks.workunit.client.0.smithi071.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-08-25T09:51:29.385 INFO:tasks.workunit.client.0.smithi071.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-08-25T09:51:29.385 INFO:tasks.workunit.client.0.smithi071.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-08-25T09:51:29.467 INFO:tasks.workunit.client.0.smithi071.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-08-25T09:51:29.476 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T09:51:29.477 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-08-25T09:51:29.529 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-08-25T09:51:29.530 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-08-25T09:51:29.531 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-08-25T09:51:29.626 INFO:tasks.workunit.client.0.smithi071.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-08-25T09:51:39.097 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir foo 2023-08-25T09:51:39.121 INFO:tasks.workunit.client.0.smithi071.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-08-25T09:51:41.008 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir foo/.snap/barsnap 2023-08-25T09:51:41.058 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-08-25T09:51:41.117 INFO:tasks.workunit.client.0.smithi071.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-08-25T09:51:42.158 INFO:tasks.workunit.client.0.smithi071.stdout:passed: files are identical 2023-08-25T09:51:42.159 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 'passed: files are identical' 2023-08-25T09:51:42.159 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir foo/.snap/barsnap 2023-08-25T09:51:42.208 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:51:42.209 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:51:42.210 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:51:42.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:51:42.379 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-08-25T09:51:42.379 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-08-25T09:51:42.469 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 1 2023-08-25T09:51:42.473 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 2 2023-08-25T09:51:42.477 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 3 2023-08-25T09:51:42.481 INFO:tasks.workunit.client.0.smithi071.stderr:+ '[' -e file4 ']' 2023-08-25T09:51:42.482 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir .snap/snap1 2023-08-25T09:51:42.518 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 4 2023-08-25T09:51:42.525 INFO:tasks.workunit.client.0.smithi071.stderr:++ ls 2023-08-25T09:51:42.543 INFO:tasks.workunit.client.0.smithi071.stderr:+ now='file1 2023-08-25T09:51:42.544 INFO:tasks.workunit.client.0.smithi071.stderr:file2 2023-08-25T09:51:42.544 INFO:tasks.workunit.client.0.smithi071.stderr:file3 2023-08-25T09:51:42.544 INFO:tasks.workunit.client.0.smithi071.stderr:file4' 2023-08-25T09:51:42.545 INFO:tasks.workunit.client.0.smithi071.stderr:++ ls .snap/snap1 2023-08-25T09:51:42.565 INFO:tasks.workunit.client.0.smithi071.stderr:+ then='file1 2023-08-25T09:51:42.565 INFO:tasks.workunit.client.0.smithi071.stderr:file2 2023-08-25T09:51:42.565 INFO:tasks.workunit.client.0.smithi071.stderr:file3' 2023-08-25T09:51:42.566 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir .snap/snap1 2023-08-25T09:51:42.590 INFO:tasks.workunit.client.0.smithi071.stderr:+ '[' 'file1 2023-08-25T09:51:42.591 INFO:tasks.workunit.client.0.smithi071.stderr:file2 2023-08-25T09:51:42.591 INFO:tasks.workunit.client.0.smithi071.stderr:file3 2023-08-25T09:51:42.591 INFO:tasks.workunit.client.0.smithi071.stderr:file4' = 'file1 2023-08-25T09:51:42.591 INFO:tasks.workunit.client.0.smithi071.stderr:file2 2023-08-25T09:51:42.591 INFO:tasks.workunit.client.0.smithi071.stderr:file3' ']' 2023-08-25T09:51:42.592 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 1 2023-08-25T09:51:42.606 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 2 2023-08-25T09:51:42.635 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 3 2023-08-25T09:51:42.655 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir .snap/snap1 2023-08-25T09:51:42.683 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 4 2023-08-25T09:51:42.710 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir .snap/snap1 2023-08-25T09:51:42.753 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm file1 file2 file3 file4 2023-08-25T09:51:42.768 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:51:42.769 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:51:42.769 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:51:42.769 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:51:42.876 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-08-25T09:51:42.876 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-08-25T09:51:42.965 INFO:tasks.workunit.client.0.smithi071.stdout:Create dir 100 to 199 ... 2023-08-25T09:51:42.979 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 100 2023-08-25T09:51:43.050 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 101 2023-08-25T09:51:43.125 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 102 2023-08-25T09:51:43.224 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 103 2023-08-25T09:51:43.304 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 104 2023-08-25T09:51:43.381 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 105 2023-08-25T09:51:43.460 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 106 2023-08-25T09:51:43.522 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 107 2023-08-25T09:51:43.580 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 108 2023-08-25T09:51:43.644 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 109 2023-08-25T09:51:43.704 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 110 2023-08-25T09:51:43.759 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 111 2023-08-25T09:51:43.810 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 112 2023-08-25T09:51:43.860 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 113 2023-08-25T09:51:44.191 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 114 2023-08-25T09:51:44.329 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 115 2023-08-25T09:51:44.410 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 116 2023-08-25T09:51:44.488 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 117 2023-08-25T09:51:44.558 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 118 2023-08-25T09:51:44.623 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 119 2023-08-25T09:51:44.685 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 120 2023-08-25T09:51:44.764 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 121 2023-08-25T09:51:44.803 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 122 2023-08-25T09:51:44.842 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 123 2023-08-25T09:51:44.880 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 124 2023-08-25T09:51:44.918 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 125 2023-08-25T09:51:44.960 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 126 2023-08-25T09:51:45.004 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 127 2023-08-25T09:51:45.049 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 128 2023-08-25T09:51:45.096 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 129 2023-08-25T09:51:45.141 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 130 2023-08-25T09:51:45.193 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 131 2023-08-25T09:51:45.250 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 132 2023-08-25T09:51:45.324 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 133 2023-08-25T09:51:45.369 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 134 2023-08-25T09:51:45.428 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 135 2023-08-25T09:51:45.489 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 136 2023-08-25T09:51:45.638 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 137 2023-08-25T09:51:45.695 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 138 2023-08-25T09:51:45.743 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 139 2023-08-25T09:51:45.830 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 140 2023-08-25T09:51:45.872 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 141 2023-08-25T09:51:45.909 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 142 2023-08-25T09:51:45.949 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 143 2023-08-25T09:51:45.986 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 144 2023-08-25T09:51:46.027 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 145 2023-08-25T09:51:46.070 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 146 2023-08-25T09:51:46.116 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 147 2023-08-25T09:51:46.160 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 148 2023-08-25T09:51:46.207 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 149 2023-08-25T09:51:46.260 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 150 2023-08-25T09:51:46.309 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 151 2023-08-25T09:51:46.361 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 152 2023-08-25T09:51:46.414 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 153 2023-08-25T09:51:46.464 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 154 2023-08-25T09:51:46.520 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 155 2023-08-25T09:51:46.587 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 156 2023-08-25T09:51:46.652 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 157 2023-08-25T09:51:46.715 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 158 2023-08-25T09:51:46.766 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 159 2023-08-25T09:51:46.812 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 160 2023-08-25T09:51:46.859 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 161 2023-08-25T09:51:46.911 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 162 2023-08-25T09:51:46.961 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 163 2023-08-25T09:51:47.011 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 164 2023-08-25T09:51:47.065 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 165 2023-08-25T09:51:47.765 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 166 2023-08-25T09:51:47.853 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 167 2023-08-25T09:51:47.930 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 168 2023-08-25T09:51:48.008 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 169 2023-08-25T09:51:48.082 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 170 2023-08-25T09:51:48.166 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 171 2023-08-25T09:51:48.293 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 172 2023-08-25T09:51:48.371 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 173 2023-08-25T09:51:48.446 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 174 2023-08-25T09:51:48.529 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 175 2023-08-25T09:51:48.605 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 176 2023-08-25T09:51:48.674 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 177 2023-08-25T09:51:48.739 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 178 2023-08-25T09:51:48.798 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 179 2023-08-25T09:51:48.856 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 180 2023-08-25T09:51:48.913 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 181 2023-08-25T09:51:48.978 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 182 2023-08-25T09:51:49.410 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 183 2023-08-25T09:51:49.497 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 184 2023-08-25T09:51:49.580 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 185 2023-08-25T09:51:49.656 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 186 2023-08-25T09:51:49.713 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 187 2023-08-25T09:51:49.771 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 188 2023-08-25T09:51:49.825 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 189 2023-08-25T09:51:49.874 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 190 2023-08-25T09:51:50.503 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 191 2023-08-25T09:51:50.600 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 192 2023-08-25T09:51:50.719 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 193 2023-08-25T09:51:50.780 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 194 2023-08-25T09:51:50.839 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 195 2023-08-25T09:51:50.901 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 196 2023-08-25T09:51:50.961 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 197 2023-08-25T09:51:51.028 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 198 2023-08-25T09:51:51.090 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 199 2023-08-25T09:51:51.149 INFO:tasks.workunit.client.0.smithi071.stdout:Take first snapshot .snap/test1 2023-08-25T09:51:51.364 INFO:tasks.workunit.client.0.smithi071.stdout:Create dir 200 to 299 ... 2023-08-25T09:51:51.369 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 200 2023-08-25T09:51:52.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:51:52.233 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:51:52.386 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 201 2023-08-25T09:51:52.448 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 202 2023-08-25T09:51:52.504 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 203 2023-08-25T09:51:52.562 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 204 2023-08-25T09:51:52.700 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 205 2023-08-25T09:51:52.777 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 206 2023-08-25T09:51:52.838 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 207 2023-08-25T09:51:52.900 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 208 2023-08-25T09:51:52.963 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 209 2023-08-25T09:51:53.024 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 210 2023-08-25T09:51:53.090 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 211 2023-08-25T09:51:53.150 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 212 2023-08-25T09:51:53.209 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 213 2023-08-25T09:51:53.270 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 214 2023-08-25T09:51:53.401 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 215 2023-08-25T09:51:53.476 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 216 2023-08-25T09:51:53.541 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 217 2023-08-25T09:51:53.608 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 218 2023-08-25T09:51:53.679 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 219 2023-08-25T09:51:53.744 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 220 2023-08-25T09:51:53.804 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 221 2023-08-25T09:51:53.865 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 222 2023-08-25T09:51:53.923 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 223 2023-08-25T09:51:53.987 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 224 2023-08-25T09:51:54.049 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 225 2023-08-25T09:51:54.116 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 226 2023-08-25T09:51:54.172 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 227 2023-08-25T09:51:54.238 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 228 2023-08-25T09:51:54.385 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 229 2023-08-25T09:51:54.467 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 230 2023-08-25T09:51:54.943 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 231 2023-08-25T09:51:55.020 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 232 2023-08-25T09:51:55.087 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 233 2023-08-25T09:51:55.160 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 234 2023-08-25T09:51:55.225 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 235 2023-08-25T09:51:55.290 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 236 2023-08-25T09:51:55.705 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 237 2023-08-25T09:51:55.750 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 238 2023-08-25T09:51:55.794 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 239 2023-08-25T09:51:55.834 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 240 2023-08-25T09:51:55.874 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 241 2023-08-25T09:51:55.913 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 242 2023-08-25T09:51:55.953 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 243 2023-08-25T09:51:55.992 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 244 2023-08-25T09:51:56.031 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 245 2023-08-25T09:51:56.072 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 246 2023-08-25T09:51:56.112 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 247 2023-08-25T09:51:56.333 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 248 2023-08-25T09:51:57.115 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 249 2023-08-25T09:51:57.183 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 250 2023-08-25T09:51:57.248 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 251 2023-08-25T09:51:57.378 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 252 2023-08-25T09:51:57.457 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 253 2023-08-25T09:51:57.522 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 254 2023-08-25T09:51:57.589 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 255 2023-08-25T09:51:57.653 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 256 2023-08-25T09:51:57.709 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 257 2023-08-25T09:51:57.769 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 258 2023-08-25T09:51:57.828 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 259 2023-08-25T09:51:57.889 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 260 2023-08-25T09:51:57.948 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 261 2023-08-25T09:51:58.006 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 262 2023-08-25T09:51:58.069 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 263 2023-08-25T09:51:58.141 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 264 2023-08-25T09:51:58.214 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 265 2023-08-25T09:51:58.286 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 266 2023-08-25T09:51:58.438 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 267 2023-08-25T09:51:58.513 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 268 2023-08-25T09:51:58.588 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 269 2023-08-25T09:51:58.656 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 270 2023-08-25T09:51:58.709 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 271 2023-08-25T09:51:58.763 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 272 2023-08-25T09:51:58.815 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 273 2023-08-25T09:51:58.860 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 274 2023-08-25T09:51:58.903 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 275 2023-08-25T09:51:58.944 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 276 2023-08-25T09:51:58.988 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 277 2023-08-25T09:51:59.034 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 278 2023-08-25T09:51:59.074 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 279 2023-08-25T09:51:59.113 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 280 2023-08-25T09:51:59.153 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 281 2023-08-25T09:51:59.197 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 282 2023-08-25T09:51:59.242 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 283 2023-08-25T09:51:59.281 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 284 2023-08-25T09:51:59.323 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 285 2023-08-25T09:51:59.432 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 286 2023-08-25T09:51:59.485 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 287 2023-08-25T09:51:59.541 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 288 2023-08-25T09:51:59.596 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 289 2023-08-25T09:51:59.651 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 290 2023-08-25T09:51:59.713 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 291 2023-08-25T09:51:59.772 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 292 2023-08-25T09:51:59.832 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 293 2023-08-25T09:51:59.890 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 294 2023-08-25T09:51:59.950 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 295 2023-08-25T09:52:00.017 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 296 2023-08-25T09:52:00.083 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 297 2023-08-25T09:52:00.144 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 298 2023-08-25T09:52:00.212 INFO:tasks.workunit.client.0.smithi071.stdout: create dir 299 2023-08-25T09:52:00.278 INFO:tasks.workunit.client.0.smithi071.stdout:Create a snapshot in every first level dir ... 2023-08-25T09:52:00.766 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/.snap/snap-subdir-test 2023-08-25T09:52:00.826 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/30 file after the snapshot 2023-08-25T09:52:00.852 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/31 file after the snapshot 2023-08-25T09:52:00.881 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/32 file after the snapshot 2023-08-25T09:52:00.912 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/33 file after the snapshot 2023-08-25T09:52:00.945 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/34 file after the snapshot 2023-08-25T09:52:00.978 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/35 file after the snapshot 2023-08-25T09:52:01.010 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/36 file after the snapshot 2023-08-25T09:52:01.041 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/37 file after the snapshot 2023-08-25T09:52:01.756 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/38 file after the snapshot 2023-08-25T09:52:01.789 INFO:tasks.workunit.client.0.smithi071.stdout: create 100/39 file after the snapshot 2023-08-25T09:52:01.815 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/.snap/snap-subdir-test 2023-08-25T09:52:02.782 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/30 file after the snapshot 2023-08-25T09:52:02.803 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/31 file after the snapshot 2023-08-25T09:52:02.831 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/32 file after the snapshot 2023-08-25T09:52:02.860 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/33 file after the snapshot 2023-08-25T09:52:02.888 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/34 file after the snapshot 2023-08-25T09:52:02.915 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/35 file after the snapshot 2023-08-25T09:52:02.946 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/36 file after the snapshot 2023-08-25T09:52:03.127 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/37 file after the snapshot 2023-08-25T09:52:03.157 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/38 file after the snapshot 2023-08-25T09:52:03.185 INFO:tasks.workunit.client.0.smithi071.stdout: create 101/39 file after the snapshot 2023-08-25T09:52:03.212 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/.snap/snap-subdir-test 2023-08-25T09:52:03.271 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/30 file after the snapshot 2023-08-25T09:52:03.290 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/31 file after the snapshot 2023-08-25T09:52:03.319 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/32 file after the snapshot 2023-08-25T09:52:04.226 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/33 file after the snapshot 2023-08-25T09:52:04.255 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/34 file after the snapshot 2023-08-25T09:52:04.283 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/35 file after the snapshot 2023-08-25T09:52:04.312 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/36 file after the snapshot 2023-08-25T09:52:04.341 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/37 file after the snapshot 2023-08-25T09:52:04.371 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/38 file after the snapshot 2023-08-25T09:52:04.400 INFO:tasks.workunit.client.0.smithi071.stdout: create 102/39 file after the snapshot 2023-08-25T09:52:04.429 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/.snap/snap-subdir-test 2023-08-25T09:52:04.479 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/30 file after the snapshot 2023-08-25T09:52:04.591 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/31 file after the snapshot 2023-08-25T09:52:04.597 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/32 file after the snapshot 2023-08-25T09:52:04.603 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/33 file after the snapshot 2023-08-25T09:52:04.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/34 file after the snapshot 2023-08-25T09:52:04.616 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/35 file after the snapshot 2023-08-25T09:52:04.622 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/36 file after the snapshot 2023-08-25T09:52:04.628 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/37 file after the snapshot 2023-08-25T09:52:04.635 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/38 file after the snapshot 2023-08-25T09:52:04.641 INFO:tasks.workunit.client.0.smithi071.stdout: create 103/39 file after the snapshot 2023-08-25T09:52:04.649 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/.snap/snap-subdir-test 2023-08-25T09:52:04.771 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/30 file after the snapshot 2023-08-25T09:52:04.777 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/31 file after the snapshot 2023-08-25T09:52:04.785 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/32 file after the snapshot 2023-08-25T09:52:04.792 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/33 file after the snapshot 2023-08-25T09:52:04.798 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/34 file after the snapshot 2023-08-25T09:52:04.805 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/35 file after the snapshot 2023-08-25T09:52:04.812 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/36 file after the snapshot 2023-08-25T09:52:04.819 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/37 file after the snapshot 2023-08-25T09:52:04.826 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/38 file after the snapshot 2023-08-25T09:52:04.833 INFO:tasks.workunit.client.0.smithi071.stdout: create 104/39 file after the snapshot 2023-08-25T09:52:04.840 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/.snap/snap-subdir-test 2023-08-25T09:52:04.893 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/30 file after the snapshot 2023-08-25T09:52:04.899 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/31 file after the snapshot 2023-08-25T09:52:04.903 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/32 file after the snapshot 2023-08-25T09:52:04.910 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/33 file after the snapshot 2023-08-25T09:52:04.916 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/34 file after the snapshot 2023-08-25T09:52:04.923 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/35 file after the snapshot 2023-08-25T09:52:04.929 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/36 file after the snapshot 2023-08-25T09:52:04.935 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/37 file after the snapshot 2023-08-25T09:52:04.940 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/38 file after the snapshot 2023-08-25T09:52:04.944 INFO:tasks.workunit.client.0.smithi071.stdout: create 105/39 file after the snapshot 2023-08-25T09:52:04.950 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/.snap/snap-subdir-test 2023-08-25T09:52:05.001 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/30 file after the snapshot 2023-08-25T09:52:05.006 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/31 file after the snapshot 2023-08-25T09:52:05.013 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/32 file after the snapshot 2023-08-25T09:52:05.018 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/33 file after the snapshot 2023-08-25T09:52:05.025 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/34 file after the snapshot 2023-08-25T09:52:05.032 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/35 file after the snapshot 2023-08-25T09:52:05.039 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/36 file after the snapshot 2023-08-25T09:52:05.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/37 file after the snapshot 2023-08-25T09:52:05.054 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/38 file after the snapshot 2023-08-25T09:52:05.061 INFO:tasks.workunit.client.0.smithi071.stdout: create 106/39 file after the snapshot 2023-08-25T09:52:05.068 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/.snap/snap-subdir-test 2023-08-25T09:52:05.125 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/30 file after the snapshot 2023-08-25T09:52:05.131 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/31 file after the snapshot 2023-08-25T09:52:05.138 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/32 file after the snapshot 2023-08-25T09:52:05.144 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/33 file after the snapshot 2023-08-25T09:52:05.152 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/34 file after the snapshot 2023-08-25T09:52:05.158 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/35 file after the snapshot 2023-08-25T09:52:05.166 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/36 file after the snapshot 2023-08-25T09:52:05.173 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/37 file after the snapshot 2023-08-25T09:52:05.180 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/38 file after the snapshot 2023-08-25T09:52:05.187 INFO:tasks.workunit.client.0.smithi071.stdout: create 107/39 file after the snapshot 2023-08-25T09:52:05.194 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/.snap/snap-subdir-test 2023-08-25T09:52:05.247 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/30 file after the snapshot 2023-08-25T09:52:05.252 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/31 file after the snapshot 2023-08-25T09:52:05.259 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/32 file after the snapshot 2023-08-25T09:52:05.267 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/33 file after the snapshot 2023-08-25T09:52:05.274 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/34 file after the snapshot 2023-08-25T09:52:05.281 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/35 file after the snapshot 2023-08-25T09:52:05.287 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/36 file after the snapshot 2023-08-25T09:52:05.295 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/37 file after the snapshot 2023-08-25T09:52:05.301 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/38 file after the snapshot 2023-08-25T09:52:05.309 INFO:tasks.workunit.client.0.smithi071.stdout: create 108/39 file after the snapshot 2023-08-25T09:52:05.316 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/.snap/snap-subdir-test 2023-08-25T09:52:05.359 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/30 file after the snapshot 2023-08-25T09:52:05.365 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/31 file after the snapshot 2023-08-25T09:52:05.370 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/32 file after the snapshot 2023-08-25T09:52:05.377 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/33 file after the snapshot 2023-08-25T09:52:05.384 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/34 file after the snapshot 2023-08-25T09:52:05.391 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/35 file after the snapshot 2023-08-25T09:52:05.398 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/36 file after the snapshot 2023-08-25T09:52:05.406 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/37 file after the snapshot 2023-08-25T09:52:05.413 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/38 file after the snapshot 2023-08-25T09:52:05.419 INFO:tasks.workunit.client.0.smithi071.stdout: create 109/39 file after the snapshot 2023-08-25T09:52:05.426 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/.snap/snap-subdir-test 2023-08-25T09:52:05.490 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/30 file after the snapshot 2023-08-25T09:52:05.812 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/31 file after the snapshot 2023-08-25T09:52:05.824 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/32 file after the snapshot 2023-08-25T09:52:05.907 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/33 file after the snapshot 2023-08-25T09:52:05.916 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/34 file after the snapshot 2023-08-25T09:52:05.955 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/35 file after the snapshot 2023-08-25T09:52:05.969 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/36 file after the snapshot 2023-08-25T09:52:05.989 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/37 file after the snapshot 2023-08-25T09:52:06.004 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/38 file after the snapshot 2023-08-25T09:52:06.014 INFO:tasks.workunit.client.0.smithi071.stdout: create 110/39 file after the snapshot 2023-08-25T09:52:06.023 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/.snap/snap-subdir-test 2023-08-25T09:52:06.410 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/30 file after the snapshot 2023-08-25T09:52:06.415 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/31 file after the snapshot 2023-08-25T09:52:06.420 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/32 file after the snapshot 2023-08-25T09:52:06.424 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/33 file after the snapshot 2023-08-25T09:52:06.428 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/34 file after the snapshot 2023-08-25T09:52:06.433 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/35 file after the snapshot 2023-08-25T09:52:06.436 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/36 file after the snapshot 2023-08-25T09:52:06.439 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/37 file after the snapshot 2023-08-25T09:52:06.442 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/38 file after the snapshot 2023-08-25T09:52:06.446 INFO:tasks.workunit.client.0.smithi071.stdout: create 111/39 file after the snapshot 2023-08-25T09:52:06.448 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/.snap/snap-subdir-test 2023-08-25T09:52:07.074 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/30 file after the snapshot 2023-08-25T09:52:07.080 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/31 file after the snapshot 2023-08-25T09:52:07.087 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/32 file after the snapshot 2023-08-25T09:52:07.093 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/33 file after the snapshot 2023-08-25T09:52:07.101 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/34 file after the snapshot 2023-08-25T09:52:07.109 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/35 file after the snapshot 2023-08-25T09:52:07.115 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/36 file after the snapshot 2023-08-25T09:52:07.122 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/37 file after the snapshot 2023-08-25T09:52:07.129 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/38 file after the snapshot 2023-08-25T09:52:07.136 INFO:tasks.workunit.client.0.smithi071.stdout: create 112/39 file after the snapshot 2023-08-25T09:52:07.144 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/.snap/snap-subdir-test 2023-08-25T09:52:07.189 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/30 file after the snapshot 2023-08-25T09:52:07.198 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/31 file after the snapshot 2023-08-25T09:52:07.203 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/32 file after the snapshot 2023-08-25T09:52:07.207 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/33 file after the snapshot 2023-08-25T09:52:07.213 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/34 file after the snapshot 2023-08-25T09:52:07.222 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/35 file after the snapshot 2023-08-25T09:52:07.229 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/36 file after the snapshot 2023-08-25T09:52:07.236 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/37 file after the snapshot 2023-08-25T09:52:07.241 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/38 file after the snapshot 2023-08-25T09:52:07.246 INFO:tasks.workunit.client.0.smithi071.stdout: create 113/39 file after the snapshot 2023-08-25T09:52:07.253 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/.snap/snap-subdir-test 2023-08-25T09:52:07.294 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/30 file after the snapshot 2023-08-25T09:52:07.304 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/31 file after the snapshot 2023-08-25T09:52:07.309 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/32 file after the snapshot 2023-08-25T09:52:07.314 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/33 file after the snapshot 2023-08-25T09:52:07.320 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/34 file after the snapshot 2023-08-25T09:52:07.325 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/35 file after the snapshot 2023-08-25T09:52:07.332 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/36 file after the snapshot 2023-08-25T09:52:07.336 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/37 file after the snapshot 2023-08-25T09:52:07.342 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/38 file after the snapshot 2023-08-25T09:52:07.347 INFO:tasks.workunit.client.0.smithi071.stdout: create 114/39 file after the snapshot 2023-08-25T09:52:07.352 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/.snap/snap-subdir-test 2023-08-25T09:52:07.394 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/30 file after the snapshot 2023-08-25T09:52:07.401 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/31 file after the snapshot 2023-08-25T09:52:07.409 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/32 file after the snapshot 2023-08-25T09:52:07.415 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/33 file after the snapshot 2023-08-25T09:52:07.422 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/34 file after the snapshot 2023-08-25T09:52:07.427 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/35 file after the snapshot 2023-08-25T09:52:07.433 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/36 file after the snapshot 2023-08-25T09:52:07.438 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/37 file after the snapshot 2023-08-25T09:52:07.445 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/38 file after the snapshot 2023-08-25T09:52:07.451 INFO:tasks.workunit.client.0.smithi071.stdout: create 115/39 file after the snapshot 2023-08-25T09:52:07.457 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/.snap/snap-subdir-test 2023-08-25T09:52:07.504 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/30 file after the snapshot 2023-08-25T09:52:07.512 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/31 file after the snapshot 2023-08-25T09:52:07.518 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/32 file after the snapshot 2023-08-25T09:52:07.524 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/33 file after the snapshot 2023-08-25T09:52:07.530 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/34 file after the snapshot 2023-08-25T09:52:07.537 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/35 file after the snapshot 2023-08-25T09:52:07.544 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/36 file after the snapshot 2023-08-25T09:52:07.549 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/37 file after the snapshot 2023-08-25T09:52:07.556 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/38 file after the snapshot 2023-08-25T09:52:07.563 INFO:tasks.workunit.client.0.smithi071.stdout: create 116/39 file after the snapshot 2023-08-25T09:52:07.567 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/.snap/snap-subdir-test 2023-08-25T09:52:07.612 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/30 file after the snapshot 2023-08-25T09:52:07.617 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/31 file after the snapshot 2023-08-25T09:52:07.622 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/32 file after the snapshot 2023-08-25T09:52:07.629 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/33 file after the snapshot 2023-08-25T09:52:07.634 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/34 file after the snapshot 2023-08-25T09:52:07.641 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/35 file after the snapshot 2023-08-25T09:52:07.648 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/36 file after the snapshot 2023-08-25T09:52:07.654 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/37 file after the snapshot 2023-08-25T09:52:07.660 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/38 file after the snapshot 2023-08-25T09:52:07.666 INFO:tasks.workunit.client.0.smithi071.stdout: create 117/39 file after the snapshot 2023-08-25T09:52:07.673 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/.snap/snap-subdir-test 2023-08-25T09:52:08.572 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/30 file after the snapshot 2023-08-25T09:52:08.576 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/31 file after the snapshot 2023-08-25T09:52:08.580 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/32 file after the snapshot 2023-08-25T09:52:08.584 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/33 file after the snapshot 2023-08-25T09:52:08.592 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/34 file after the snapshot 2023-08-25T09:52:08.598 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/35 file after the snapshot 2023-08-25T09:52:08.604 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/36 file after the snapshot 2023-08-25T09:52:08.609 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/37 file after the snapshot 2023-08-25T09:52:08.616 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/38 file after the snapshot 2023-08-25T09:52:08.624 INFO:tasks.workunit.client.0.smithi071.stdout: create 118/39 file after the snapshot 2023-08-25T09:52:08.632 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/.snap/snap-subdir-test 2023-08-25T09:52:08.666 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/30 file after the snapshot 2023-08-25T09:52:08.672 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/31 file after the snapshot 2023-08-25T09:52:08.678 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/32 file after the snapshot 2023-08-25T09:52:08.684 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/33 file after the snapshot 2023-08-25T09:52:08.690 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/34 file after the snapshot 2023-08-25T09:52:08.698 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/35 file after the snapshot 2023-08-25T09:52:08.703 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/36 file after the snapshot 2023-08-25T09:52:08.708 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/37 file after the snapshot 2023-08-25T09:52:08.713 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/38 file after the snapshot 2023-08-25T09:52:08.720 INFO:tasks.workunit.client.0.smithi071.stdout: create 119/39 file after the snapshot 2023-08-25T09:52:08.724 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/.snap/snap-subdir-test 2023-08-25T09:52:08.766 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/30 file after the snapshot 2023-08-25T09:52:08.771 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/31 file after the snapshot 2023-08-25T09:52:08.777 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/32 file after the snapshot 2023-08-25T09:52:08.782 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/33 file after the snapshot 2023-08-25T09:52:08.787 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/34 file after the snapshot 2023-08-25T09:52:08.793 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/35 file after the snapshot 2023-08-25T09:52:08.797 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/36 file after the snapshot 2023-08-25T09:52:08.801 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/37 file after the snapshot 2023-08-25T09:52:08.805 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/38 file after the snapshot 2023-08-25T09:52:08.809 INFO:tasks.workunit.client.0.smithi071.stdout: create 120/39 file after the snapshot 2023-08-25T09:52:08.813 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/.snap/snap-subdir-test 2023-08-25T09:52:08.845 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/30 file after the snapshot 2023-08-25T09:52:08.849 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/31 file after the snapshot 2023-08-25T09:52:08.855 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/32 file after the snapshot 2023-08-25T09:52:08.860 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/33 file after the snapshot 2023-08-25T09:52:08.865 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/34 file after the snapshot 2023-08-25T09:52:08.870 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/35 file after the snapshot 2023-08-25T09:52:08.875 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/36 file after the snapshot 2023-08-25T09:52:08.879 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/37 file after the snapshot 2023-08-25T09:52:08.883 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/38 file after the snapshot 2023-08-25T09:52:08.887 INFO:tasks.workunit.client.0.smithi071.stdout: create 121/39 file after the snapshot 2023-08-25T09:52:08.891 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/.snap/snap-subdir-test 2023-08-25T09:52:08.930 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/30 file after the snapshot 2023-08-25T09:52:08.931 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/31 file after the snapshot 2023-08-25T09:52:08.936 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/32 file after the snapshot 2023-08-25T09:52:09.122 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/33 file after the snapshot 2023-08-25T09:52:09.122 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/34 file after the snapshot 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/35 file after the snapshot 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/36 file after the snapshot 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/37 file after the snapshot 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/38 file after the snapshot 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 122/39 file after the snapshot 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/.snap/snap-subdir-test 2023-08-25T09:52:09.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/30 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/31 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/32 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/33 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/34 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/35 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/36 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/37 file after the snapshot 2023-08-25T09:52:09.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/38 file after the snapshot 2023-08-25T09:52:09.125 INFO:tasks.workunit.client.0.smithi071.stdout: create 123/39 file after the snapshot 2023-08-25T09:52:09.125 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/.snap/snap-subdir-test 2023-08-25T09:52:09.129 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/30 file after the snapshot 2023-08-25T09:52:09.135 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/31 file after the snapshot 2023-08-25T09:52:09.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/32 file after the snapshot 2023-08-25T09:52:09.147 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/33 file after the snapshot 2023-08-25T09:52:09.152 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/34 file after the snapshot 2023-08-25T09:52:09.158 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/35 file after the snapshot 2023-08-25T09:52:09.164 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/36 file after the snapshot 2023-08-25T09:52:09.169 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/37 file after the snapshot 2023-08-25T09:52:09.175 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/38 file after the snapshot 2023-08-25T09:52:09.179 INFO:tasks.workunit.client.0.smithi071.stdout: create 124/39 file after the snapshot 2023-08-25T09:52:09.184 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/.snap/snap-subdir-test 2023-08-25T09:52:09.222 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/30 file after the snapshot 2023-08-25T09:52:09.227 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/31 file after the snapshot 2023-08-25T09:52:09.231 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/32 file after the snapshot 2023-08-25T09:52:09.237 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/33 file after the snapshot 2023-08-25T09:52:09.243 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/34 file after the snapshot 2023-08-25T09:52:09.248 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/35 file after the snapshot 2023-08-25T09:52:09.253 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/36 file after the snapshot 2023-08-25T09:52:09.259 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/37 file after the snapshot 2023-08-25T09:52:09.265 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/38 file after the snapshot 2023-08-25T09:52:09.272 INFO:tasks.workunit.client.0.smithi071.stdout: create 125/39 file after the snapshot 2023-08-25T09:52:09.275 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/.snap/snap-subdir-test 2023-08-25T09:52:09.313 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/30 file after the snapshot 2023-08-25T09:52:09.318 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/31 file after the snapshot 2023-08-25T09:52:09.324 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/32 file after the snapshot 2023-08-25T09:52:09.329 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/33 file after the snapshot 2023-08-25T09:52:09.336 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/34 file after the snapshot 2023-08-25T09:52:09.342 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/35 file after the snapshot 2023-08-25T09:52:09.347 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/36 file after the snapshot 2023-08-25T09:52:09.353 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/37 file after the snapshot 2023-08-25T09:52:09.361 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/38 file after the snapshot 2023-08-25T09:52:09.367 INFO:tasks.workunit.client.0.smithi071.stdout: create 126/39 file after the snapshot 2023-08-25T09:52:09.373 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/.snap/snap-subdir-test 2023-08-25T09:52:09.424 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/30 file after the snapshot 2023-08-25T09:52:09.428 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/31 file after the snapshot 2023-08-25T09:52:09.432 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/32 file after the snapshot 2023-08-25T09:52:09.438 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/33 file after the snapshot 2023-08-25T09:52:09.688 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/34 file after the snapshot 2023-08-25T09:52:09.697 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/35 file after the snapshot 2023-08-25T09:52:09.704 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/36 file after the snapshot 2023-08-25T09:52:09.712 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/37 file after the snapshot 2023-08-25T09:52:09.719 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/38 file after the snapshot 2023-08-25T09:52:09.725 INFO:tasks.workunit.client.0.smithi071.stdout: create 127/39 file after the snapshot 2023-08-25T09:52:09.733 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/.snap/snap-subdir-test 2023-08-25T09:52:09.772 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/30 file after the snapshot 2023-08-25T09:52:09.778 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/31 file after the snapshot 2023-08-25T09:52:09.785 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/32 file after the snapshot 2023-08-25T09:52:09.792 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/33 file after the snapshot 2023-08-25T09:52:09.799 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/34 file after the snapshot 2023-08-25T09:52:09.808 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/35 file after the snapshot 2023-08-25T09:52:09.814 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/36 file after the snapshot 2023-08-25T09:52:09.822 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/37 file after the snapshot 2023-08-25T09:52:09.836 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/38 file after the snapshot 2023-08-25T09:52:09.840 INFO:tasks.workunit.client.0.smithi071.stdout: create 128/39 file after the snapshot 2023-08-25T09:52:09.846 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/.snap/snap-subdir-test 2023-08-25T09:52:09.902 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/30 file after the snapshot 2023-08-25T09:52:09.908 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/31 file after the snapshot 2023-08-25T09:52:09.914 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/32 file after the snapshot 2023-08-25T09:52:09.919 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/33 file after the snapshot 2023-08-25T09:52:09.925 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/34 file after the snapshot 2023-08-25T09:52:09.930 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/35 file after the snapshot 2023-08-25T09:52:09.937 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/36 file after the snapshot 2023-08-25T09:52:09.942 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/37 file after the snapshot 2023-08-25T09:52:09.948 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/38 file after the snapshot 2023-08-25T09:52:09.953 INFO:tasks.workunit.client.0.smithi071.stdout: create 129/39 file after the snapshot 2023-08-25T09:52:09.959 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/.snap/snap-subdir-test 2023-08-25T09:52:09.993 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/30 file after the snapshot 2023-08-25T09:52:09.997 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/31 file after the snapshot 2023-08-25T09:52:10.001 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/32 file after the snapshot 2023-08-25T09:52:10.006 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/33 file after the snapshot 2023-08-25T09:52:10.011 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/34 file after the snapshot 2023-08-25T09:52:10.015 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/35 file after the snapshot 2023-08-25T09:52:10.022 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/36 file after the snapshot 2023-08-25T09:52:10.507 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/37 file after the snapshot 2023-08-25T09:52:10.526 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/38 file after the snapshot 2023-08-25T09:52:10.535 INFO:tasks.workunit.client.0.smithi071.stdout: create 130/39 file after the snapshot 2023-08-25T09:52:10.543 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/.snap/snap-subdir-test 2023-08-25T09:52:10.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/30 file after the snapshot 2023-08-25T09:52:10.615 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/31 file after the snapshot 2023-08-25T09:52:10.621 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/32 file after the snapshot 2023-08-25T09:52:10.627 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/33 file after the snapshot 2023-08-25T09:52:10.634 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/34 file after the snapshot 2023-08-25T09:52:10.642 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/35 file after the snapshot 2023-08-25T09:52:10.649 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/36 file after the snapshot 2023-08-25T09:52:10.656 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/37 file after the snapshot 2023-08-25T09:52:10.661 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/38 file after the snapshot 2023-08-25T09:52:10.666 INFO:tasks.workunit.client.0.smithi071.stdout: create 131/39 file after the snapshot 2023-08-25T09:52:10.672 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/.snap/snap-subdir-test 2023-08-25T09:52:10.721 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/30 file after the snapshot 2023-08-25T09:52:10.726 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/31 file after the snapshot 2023-08-25T09:52:10.732 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/32 file after the snapshot 2023-08-25T09:52:10.738 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/33 file after the snapshot 2023-08-25T09:52:10.744 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/34 file after the snapshot 2023-08-25T09:52:10.749 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/35 file after the snapshot 2023-08-25T09:52:10.755 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/36 file after the snapshot 2023-08-25T09:52:10.761 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/37 file after the snapshot 2023-08-25T09:52:10.766 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/38 file after the snapshot 2023-08-25T09:52:10.772 INFO:tasks.workunit.client.0.smithi071.stdout: create 132/39 file after the snapshot 2023-08-25T09:52:10.778 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/.snap/snap-subdir-test 2023-08-25T09:52:10.832 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/30 file after the snapshot 2023-08-25T09:52:10.840 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/31 file after the snapshot 2023-08-25T09:52:10.844 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/32 file after the snapshot 2023-08-25T09:52:10.852 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/33 file after the snapshot 2023-08-25T09:52:10.857 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/34 file after the snapshot 2023-08-25T09:52:10.862 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/35 file after the snapshot 2023-08-25T09:52:10.867 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/36 file after the snapshot 2023-08-25T09:52:10.873 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/37 file after the snapshot 2023-08-25T09:52:10.878 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/38 file after the snapshot 2023-08-25T09:52:10.884 INFO:tasks.workunit.client.0.smithi071.stdout: create 133/39 file after the snapshot 2023-08-25T09:52:10.889 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/.snap/snap-subdir-test 2023-08-25T09:52:10.929 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/30 file after the snapshot 2023-08-25T09:52:10.935 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/31 file after the snapshot 2023-08-25T09:52:10.940 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/32 file after the snapshot 2023-08-25T09:52:10.947 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/33 file after the snapshot 2023-08-25T09:52:10.953 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/34 file after the snapshot 2023-08-25T09:52:10.961 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/35 file after the snapshot 2023-08-25T09:52:10.969 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/36 file after the snapshot 2023-08-25T09:52:10.976 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/37 file after the snapshot 2023-08-25T09:52:10.983 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/38 file after the snapshot 2023-08-25T09:52:10.990 INFO:tasks.workunit.client.0.smithi071.stdout: create 134/39 file after the snapshot 2023-08-25T09:52:10.996 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/.snap/snap-subdir-test 2023-08-25T09:52:11.045 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/30 file after the snapshot 2023-08-25T09:52:11.050 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/31 file after the snapshot 2023-08-25T09:52:11.057 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/32 file after the snapshot 2023-08-25T09:52:11.065 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/33 file after the snapshot 2023-08-25T09:52:11.073 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/34 file after the snapshot 2023-08-25T09:52:11.080 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/35 file after the snapshot 2023-08-25T09:52:11.087 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/36 file after the snapshot 2023-08-25T09:52:11.150 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/37 file after the snapshot 2023-08-25T09:52:11.156 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/38 file after the snapshot 2023-08-25T09:52:11.162 INFO:tasks.workunit.client.0.smithi071.stdout: create 135/39 file after the snapshot 2023-08-25T09:52:11.170 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/.snap/snap-subdir-test 2023-08-25T09:52:11.970 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/30 file after the snapshot 2023-08-25T09:52:11.976 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/31 file after the snapshot 2023-08-25T09:52:11.982 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/32 file after the snapshot 2023-08-25T09:52:11.988 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/33 file after the snapshot 2023-08-25T09:52:11.993 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/34 file after the snapshot 2023-08-25T09:52:11.997 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/35 file after the snapshot 2023-08-25T09:52:12.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:12.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.299+0000 7f8b159c9640 1 Processor -- start 2023-08-25T09:52:12.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.299+0000 7f8b159c9640 1 -- start start 2023-08-25T09:52:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.300+0000 7f8b159c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b100c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.301+0000 7f8b159c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b101509d0 0x7f8b10150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.301+0000 7f8b159c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100c1dc0 con 0x7f8b100c3a40 2023-08-25T09:52:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.301+0000 7f8b159c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b10157a80 con 0x7f8b101509d0 2023-08-25T09:52:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.301+0000 7f8b159c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b101513a0 con 0x7f8b100c3660 2023-08-25T09:52:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.301+0000 7f8b0effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b100c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.301+0000 7f8b0f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b101509d0 0x7f8b10150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.302+0000 7f8b0effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b100c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:39132/0 (socket says 172.21.15.71:39132) 2023-08-25T09:52:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.302+0000 7f8b0f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b101509d0 0x7f8b10150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:45460/0 (socket says 172.21.15.71:45460) 2023-08-25T09:52:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.302+0000 7f8b0effd640 1 -- 172.21.15.71:0/3000392608 learned_addr learned my addr 172.21.15.71:0/3000392608 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:12.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.302+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 msgr2=0x7f8b100c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:12.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.302+0000 7f8b0f7fe640 1 --2- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b100c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.303+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/3000392608 >> v1:172.21.15.164:6789/0 conn(0x7f8b100c3a40 legacy=0x7f8b1006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.303+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/3000392608 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10156e00 con 0x7f8b101509d0 2023-08-25T09:52:12.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.303+0000 7f8b0effd640 1 --2- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:52:12.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.304+0000 7f8b0f7fe640 1 --2- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b101509d0 0x7f8b10150e60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009d90 tx=0x7f8b0802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58fa127c2fc06f5b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:12.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.304+0000 7f8b0dffb640 1 -- 172.21.15.71:0/3000392608 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0802cce0 con 0x7f8b101509d0 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.304+0000 7f8b0dffb640 1 -- 172.21.15.71:0/3000392608 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0802ce40 con 0x7f8b101509d0 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.304+0000 7f8b0dffb640 1 -- 172.21.15.71:0/3000392608 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08034660 con 0x7f8b101509d0 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 -- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b101509d0 msgr2=0x7f8b10150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 --2- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b101509d0 0x7f8b10150e60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009d90 tx=0x7f8b0802bf30 comp rx=0 tx=0).stop 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 -- 172.21.15.71:0/3000392608 shutdown_connections 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 --2- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b101509d0 0x7f8b10150e60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 --2- 172.21.15.71:0/3000392608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 -- 172.21.15.71:0/3000392608 >> 172.21.15.71:0/3000392608 conn(0x7f8b100654f0 msgr2=0x7f8b10065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 -- 172.21.15.71:0/3000392608 shutdown_connections 2023-08-25T09:52:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.305+0000 7f8b159c9640 1 -- 172.21.15.71:0/3000392608 wait complete. 2023-08-25T09:52:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.307+0000 7f8b159c9640 1 Processor -- start 2023-08-25T09:52:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.307+0000 7f8b159c9640 1 -- start start 2023-08-25T09:52:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b159c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b1006a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b159c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b100c3a40 0x7f8b10159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b159c9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 0x7f8b101595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b159c9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1014a9f0 con 0x7f8b101509d0 2023-08-25T09:52:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b159c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1015a760 con 0x7f8b100c3a40 2023-08-25T09:52:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b159c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10158c00 con 0x7f8b100c3660 2023-08-25T09:52:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b0f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 0x7f8b101595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b0f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 0x7f8b101595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54876/0 (socket says 172.21.15.71:54876) 2023-08-25T09:52:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/1601602102 learned_addr learned my addr 172.21.15.71:0/1601602102 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b0effd640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b1006a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.308+0000 7f8b0e7fc640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b100c3a40 0x7f8b10159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 msgr2=0x7f8b1006a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b0f7fe640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b1006a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b100c3a40 msgr2=0x7f8b10159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b0f7fe640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b100c3a40 0x7f8b10159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b0f7fe640 1 -- 172.21.15.71:0/1601602102 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b000094d0 con 0x7f8b101509d0 2023-08-25T09:52:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b0f7fe640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 0x7f8b101595d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009eb0 tx=0x7f8b08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0803d600 con 0x7f8b101509d0 2023-08-25T09:52:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0803d780 con 0x7f8b101509d0 2023-08-25T09:52:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0803db30 con 0x7f8b101509d0 2023-08-25T09:52:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.309+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b08009a70 con 0x7f8b101509d0 2023-08-25T09:52:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.310+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b101588d0 con 0x7f8b101509d0 2023-08-25T09:52:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.311+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b100c29c0 con 0x7f8b101509d0 2023-08-25T09:52:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.311+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8b0803b040 con 0x7f8b101509d0 2023-08-25T09:52:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.312+0000 7f8af7fff640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8afc06a050 0x7f8afc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.313+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7f8b080aae40 con 0x7f8b101509d0 2023-08-25T09:52:12.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.313+0000 7f8b0effd640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8afc06a050 0x7f8afc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.313+0000 7f8b0effd640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8afc06a050 0x7f8afc06c510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009c50 tx=0x7f8b00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:12.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.317+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8b080761c0 con 0x7f8b101509d0 2023-08-25T09:52:12.474 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/36 file after the snapshot 2023-08-25T09:52:12.480 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/37 file after the snapshot 2023-08-25T09:52:12.485 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/38 file after the snapshot 2023-08-25T09:52:12.491 INFO:tasks.workunit.client.0.smithi071.stdout: create 136/39 file after the snapshot 2023-08-25T09:52:12.498 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/.snap/snap-subdir-test 2023-08-25T09:52:12.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.559+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b10069160 con 0x7f8b101509d0 2023-08-25T09:52:12.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.561+0000 7f8af7fff640 1 -- 172.21.15.71:0/1601602102 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8b08039090 con 0x7f8b101509d0 2023-08-25T09:52:12.564 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:12.564 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:12.564 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:12.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8afc06a050 msgr2=0x7f8afc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8afc06a050 0x7f8afc06c510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009c50 tx=0x7f8b00008040 comp rx=0 tx=0).stop 2023-08-25T09:52:12.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 msgr2=0x7f8b101595d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 0x7f8b101595d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009eb0 tx=0x7f8b08004a40 comp rx=0 tx=0).stop 2023-08-25T09:52:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 shutdown_connections 2023-08-25T09:52:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b100c3a40 0x7f8b10159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8afc06a050 0x7f8afc06c510 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b101509d0 0x7f8b101595d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 --2- 172.21.15.71:0/1601602102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b100c3660 0x7f8b1006a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.563+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 >> 172.21.15.71:0/1601602102 conn(0x7f8b100654f0 msgr2=0x7f8b10069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:12.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.564+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 shutdown_connections 2023-08-25T09:52:12.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.564+0000 7f8b159c9640 1 -- 172.21.15.71:0/1601602102 wait complete. 2023-08-25T09:52:12.587 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=4) 2023-08-25T09:52:12.588 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-08-25T09:52:12.588 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T09:52:12.590 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/30 file after the snapshot 2023-08-25T09:52:12.595 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/31 file after the snapshot 2023-08-25T09:52:12.598 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/32 file after the snapshot 2023-08-25T09:52:12.602 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/33 file after the snapshot 2023-08-25T09:52:12.605 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/34 file after the snapshot 2023-08-25T09:52:12.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/35 file after the snapshot 2023-08-25T09:52:12.616 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/36 file after the snapshot 2023-08-25T09:52:12.622 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/37 file after the snapshot 2023-08-25T09:52:12.626 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/38 file after the snapshot 2023-08-25T09:52:12.627 INFO:tasks.ceph.mds.d:Stopped 2023-08-25T09:52:12.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:12.636 INFO:tasks.workunit.client.0.smithi071.stdout: create 137/39 file after the snapshot 2023-08-25T09:52:12.637 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/.snap/snap-subdir-test 2023-08-25T09:52:12.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.825+0000 7f8d3307c640 1 Processor -- start 2023-08-25T09:52:12.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.825+0000 7f8d3307c640 1 -- start start 2023-08-25T09:52:12.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.826+0000 7f8d3307c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8d2c1594d0 0x7f8d2c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.826+0000 7f8d3307c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.826+0000 7f8d3307c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c06a960 con 0x7f8d2c0c3940 2023-08-25T09:52:12.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.826+0000 7f8d3307c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c06ac20 con 0x7f8d2c1594d0 2023-08-25T09:52:12.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.826+0000 7f8d3307c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c06af00 con 0x7f8d2c06cc70 2023-08-25T09:52:12.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8d2c1594d0 0x7f8d2c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8d2c1594d0 0x7f8d2c0c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54882/0 (socket says 172.21.15.71:54882) 2023-08-25T09:52:12.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 -- 172.21.15.71:0/2700104632 learned_addr learned my addr 172.21.15.71:0/2700104632 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:12.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d2bfff640 1 --2- 172.21.15.71:0/2700104632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 -- 172.21.15.71:0/2700104632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 msgr2=0x7f8d2c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:12.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 --2- 172.21.15.71:0/2700104632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 -- 172.21.15.71:0/2700104632 >> v1:172.21.15.71:6789/0 conn(0x7f8d2c0c3940 legacy=0x7f8d2c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.827+0000 7f8d30e18640 1 -- 172.21.15.71:0/2700104632 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0c2e60 con 0x7f8d2c1594d0 2023-08-25T09:52:12.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.828+0000 7f8d30e18640 1 --2- 172.21.15.71:0/2700104632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c0c4c90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009870 tx=0x7f8d2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2f8a8a45a609c5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:12.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d2b7fe640 1 -- 172.21.15.71:0/2700104632 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20030840 con 0x7f8d2c1594d0 2023-08-25T09:52:12.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d2b7fe640 1 -- 172.21.15.71:0/2700104632 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d200309a0 con 0x7f8d2c1594d0 2023-08-25T09:52:12.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d2b7fe640 1 -- 172.21.15.71:0/2700104632 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20030d30 con 0x7f8d2c1594d0 2023-08-25T09:52:12.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d3307c640 1 -- 172.21.15.71:0/2700104632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c1594d0 msgr2=0x7f8d2c0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d3307c640 1 --2- 172.21.15.71:0/2700104632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c0c4c90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009870 tx=0x7f8d2002b980 comp rx=0 tx=0).stop 2023-08-25T09:52:12.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d3307c640 1 -- 172.21.15.71:0/2700104632 shutdown_connections 2023-08-25T09:52:12.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d3307c640 1 --2- 172.21.15.71:0/2700104632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d3307c640 1 --2- 172.21.15.71:0/2700104632 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c0c4c90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.829+0000 7f8d3307c640 1 -- 172.21.15.71:0/2700104632 >> 172.21.15.71:0/2700104632 conn(0x7f8d2c0654f0 msgr2=0x7f8d2c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:12.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.830+0000 7f8d3307c640 1 -- 172.21.15.71:0/2700104632 shutdown_connections 2023-08-25T09:52:12.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.830+0000 7f8d3307c640 1 -- 172.21.15.71:0/2700104632 wait complete. 2023-08-25T09:52:12.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.831+0000 7f8d3307c640 1 Processor -- start 2023-08-25T09:52:12.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.832+0000 7f8d3307c640 1 -- start start 2023-08-25T09:52:12.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.832+0000 7f8d3307c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d3307c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 0x7f8d2c1f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d3307c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c1f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d3307c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c144b80 con 0x7f8d2c0c3940 2023-08-25T09:52:12.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d3307c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c143c70 con 0x7f8d2c1594d0 2023-08-25T09:52:12.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d3307c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c143f30 con 0x7f8d2c06cc70 2023-08-25T09:52:12.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 0x7f8d2c1f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 0x7f8d2c1f23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54896/0 (socket says 172.21.15.71:54896) 2023-08-25T09:52:12.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 -- 172.21.15.71:0/238377301 learned_addr learned my addr 172.21.15.71:0/238377301 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:12.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d30e18640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 -- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 msgr2=0x7f8d2c14e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d31619640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c1f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 -- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d2c1594d0 msgr2=0x7f8d2c1f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:12.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c1f2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:12.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.833+0000 7f8d2bfff640 1 -- 172.21.15.71:0/238377301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200094d0 con 0x7f8d2c0c3940 2023-08-25T09:52:12.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.834+0000 7f8d2bfff640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 0x7f8d2c1f23d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800c7f0 tx=0x7f8d1800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:12.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.834+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18004220 con 0x7f8d2c0c3940 2023-08-25T09:52:12.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.834+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d18004380 con 0x7f8d2c0c3940 2023-08-25T09:52:12.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.834+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18007840 con 0x7f8d2c0c3940 2023-08-25T09:52:12.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.834+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c144dd0 con 0x7f8d2c0c3940 2023-08-25T09:52:12.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.834+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c1f1eb0 con 0x7f8d2c0c3940 2023-08-25T09:52:12.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.835+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c1f20d0 con 0x7f8d2c0c3940 2023-08-25T09:52:12.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.836+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8d18007bb0 con 0x7f8d2c0c3940 2023-08-25T09:52:12.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.837+0000 7f8d297fa640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8d1006a050 0x7f8d1006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:12.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.837+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7f8d18088880 con 0x7f8d2c0c3940 2023-08-25T09:52:12.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.837+0000 7f8d30e18640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8d1006a050 0x7f8d1006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:12.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.839+0000 7f8d30e18640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8d1006a050 0x7f8d1006c510 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8d20036000 tx=0x7f8d2003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:12.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:12.843+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8d18053c00 con 0x7f8d2c0c3940 2023-08-25T09:52:13.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.072+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d2c069160 con 0x7f8d2c0c3940 2023-08-25T09:52:13.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.073+0000 7f8d297fa640 1 -- 172.21.15.71:0/238377301 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8d18007e60 con 0x7f8d2c0c3940 2023-08-25T09:52:13.077 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:13.077 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:13.078 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:13.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.075+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8d1006a050 msgr2=0x7f8d1006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:13.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8d1006a050 0x7f8d1006c510 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8d20036000 tx=0x7f8d2003b040 comp rx=0 tx=0).stop 2023-08-25T09:52:13.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 msgr2=0x7f8d2c1f23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:13.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 0x7f8d2c1f23d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800c7f0 tx=0x7f8d1800ccb0 comp rx=0 tx=0).stop 2023-08-25T09:52:13.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 shutdown_connections 2023-08-25T09:52:13.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8d2c1594d0 0x7f8d2c1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:13.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8d1006a050 0x7f8d1006c510 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d2c0c3940 0x7f8d2c1f23d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 --2- 172.21.15.71:0/238377301 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8d2c06cc70 0x7f8d2c14e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 >> 172.21.15.71:0/238377301 conn(0x7f8d2c0654f0 msgr2=0x7f8d2c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 shutdown_connections 2023-08-25T09:52:13.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:13.076+0000 7f8d3307c640 1 -- 172.21.15.71:0/238377301 wait complete. 2023-08-25T09:52:13.097 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:15.098 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:15.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.318+0000 7fe0d16de640 1 Processor -- start 2023-08-25T09:52:15.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.318+0000 7fe0d16de640 1 -- start start 2023-08-25T09:52:15.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0d16de640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0cc070cf0 0x7fe0cc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:15.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0d16de640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc06eb60 con 0x7fe0cc152d80 2023-08-25T09:52:15.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0d16de640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc06ca30 con 0x7fe0cc0c4560 2023-08-25T09:52:15.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0d16de640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0cc06cf90 con 0x7fe0cc070cf0 2023-08-25T09:52:15.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0cb7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0cc070cf0 0x7fe0cc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:15.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0cb7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0cc070cf0 0x7fe0cc06e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54912/0 (socket says 172.21.15.71:54912) 2023-08-25T09:52:15.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.320+0000 7fe0cb7fe640 1 -- 172.21.15.71:0/550290098 learned_addr learned my addr 172.21.15.71:0/550290098 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:15.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.321+0000 7fe0cb7fe640 1 -- 172.21.15.71:0/550290098 >> v1:172.21.15.164:6789/0 conn(0x7fe0cc0c4560 legacy=0x7fe0cc1617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:15.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.321+0000 7fe0cb7fe640 1 -- 172.21.15.71:0/550290098 >> v1:172.21.15.71:6789/0 conn(0x7fe0cc152d80 legacy=0x7fe0cc0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.321+0000 7fe0cb7fe640 1 -- 172.21.15.71:0/550290098 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc158fd0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.322+0000 7fe0cb7fe640 1 --2- 172.21.15.71:0/550290098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc06e620 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4009e50 tx=0x7fe0c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18f6742d54e37c1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.322+0000 7fe0c9ffb640 1 -- 172.21.15.71:0/550290098 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c402ca80 con 0x7fe0cc070cf0 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.322+0000 7fe0c9ffb640 1 -- 172.21.15.71:0/550290098 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c402cbe0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.322+0000 7fe0c9ffb640 1 -- 172.21.15.71:0/550290098 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40343f0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.322+0000 7fe0d16de640 1 -- 172.21.15.71:0/550290098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 msgr2=0x7fe0cc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:15.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.322+0000 7fe0d16de640 1 --2- 172.21.15.71:0/550290098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc06e620 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4009e50 tx=0x7fe0c402bee0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.323+0000 7fe0d16de640 1 -- 172.21.15.71:0/550290098 shutdown_connections 2023-08-25T09:52:15.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.323+0000 7fe0d16de640 1 --2- 172.21.15.71:0/550290098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc06e620 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.323+0000 7fe0d16de640 1 -- 172.21.15.71:0/550290098 >> 172.21.15.71:0/550290098 conn(0x7fe0cc0654f0 msgr2=0x7fe0cc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:15.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.323+0000 7fe0d16de640 1 -- 172.21.15.71:0/550290098 shutdown_connections 2023-08-25T09:52:15.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.323+0000 7fe0d16de640 1 -- 172.21.15.71:0/550290098 wait complete. 2023-08-25T09:52:15.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.325+0000 7fe0d16de640 1 Processor -- start 2023-08-25T09:52:15.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.325+0000 7fe0d16de640 1 -- start start 2023-08-25T09:52:15.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0d16de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc1f1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0d16de640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0cc0c4560 0x7fe0cc0c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0d16de640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0cc152d80 0x7fe0cc0c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0d16de640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc14c2b0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0d16de640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc149f30 con 0x7fe0cc0c4560 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0d16de640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc1f2120 con 0x7fe0cc152d80 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc1f1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:15.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc1f1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54926/0 (socket says 172.21.15.71:54926) 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 -- 172.21.15.71:0/2194727866 learned_addr learned my addr 172.21.15.71:0/2194727866 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 -- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0cc152d80 msgr2=0x7fe0cc0c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0cc152d80 0x7fe0cc0c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 -- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0cc0c4560 msgr2=0x7fe0cc0c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0ca7fc640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0cc0c4560 0x7fe0cc0c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0cc0c4560 0x7fe0cc0c3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.326+0000 7fe0caffd640 1 -- 172.21.15.71:0/2194727866 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4009b30 con 0x7fe0cc070cf0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.327+0000 7fe0caffd640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc1f1be0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00c450 tx=0x7fe0bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.327+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc0052d0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.327+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc005430 con 0x7fe0cc070cf0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.327+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc149970 con 0x7fe0cc070cf0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.327+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc012870 con 0x7fe0cc070cf0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.327+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc0c2560 con 0x7fe0cc070cf0 2023-08-25T09:52:15.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.328+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc069160 con 0x7fe0cc070cf0 2023-08-25T09:52:15.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.329+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe0bc0129d0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.330+0000 7fe0b37fe640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0b806a080 0x7fe0b806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:15.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.330+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7fe0bc088530 con 0x7fe0cc070cf0 2023-08-25T09:52:15.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.330+0000 7fe0ca7fc640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0b806a080 0x7fe0b806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:15.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.331+0000 7fe0ca7fc640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0b806a080 0x7fe0b806c540 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc156960 tx=0x7fe0c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:15.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.334+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe0bc053790 con 0x7fe0cc070cf0 2023-08-25T09:52:15.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.569+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0cc14e2d0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.570+0000 7fe0b37fe640 1 -- 172.21.15.71:0/2194727866 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fe0bc056db0 con 0x7fe0cc070cf0 2023-08-25T09:52:15.573 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:15.573 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:15.574 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:15.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0b806a080 msgr2=0x7fe0b806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:15.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0b806a080 0x7fe0b806c540 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc156960 tx=0x7fe0c0009040 comp rx=0 tx=0).stop 2023-08-25T09:52:15.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 msgr2=0x7fe0cc1f1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc1f1be0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00c450 tx=0x7fe0bc00c910 comp rx=0 tx=0).stop 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 shutdown_connections 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0cc0c4560 0x7fe0cc0c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.573+0000 7fe0d16de640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0b806a080 0x7fe0b806c540 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.574+0000 7fe0d16de640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0cc152d80 0x7fe0cc0c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.574+0000 7fe0d16de640 1 --2- 172.21.15.71:0/2194727866 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0cc070cf0 0x7fe0cc1f1be0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:15.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.574+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 >> 172.21.15.71:0/2194727866 conn(0x7fe0cc0654f0 msgr2=0x7fe0cc150960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:15.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.574+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 shutdown_connections 2023-08-25T09:52:15.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:15.574+0000 7fe0d16de640 1 -- 172.21.15.71:0/2194727866 wait complete. 2023-08-25T09:52:15.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:17.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:17.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.822+0000 7f8eed7b5640 1 Processor -- start 2023-08-25T09:52:17.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.822+0000 7f8eed7b5640 1 -- start start 2023-08-25T09:52:17.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.823+0000 7f8eed7b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:17.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.823+0000 7f8eed7b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8ee80c3a40 0x7f8ee806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:17.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8eed7b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80c1dc0 con 0x7f8ee81509d0 2023-08-25T09:52:17.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8eed7b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee8157a80 con 0x7f8ee80c3a40 2023-08-25T09:52:17.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8eed7b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee806cf70 con 0x7f8ee80c3660 2023-08-25T09:52:17.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8ee6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:17.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8edffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8ee80c3a40 0x7f8ee806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:17.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8edffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8ee80c3a40 0x7f8ee806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:45496/0 (socket says 172.21.15.71:45496) 2023-08-25T09:52:17.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8ee6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee80c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:39176/0 (socket says 172.21.15.71:39176) 2023-08-25T09:52:17.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.824+0000 7f8edffff640 1 -- 172.21.15.71:0/1835889012 learned_addr learned my addr 172.21.15.71:0/1835889012 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:17.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.825+0000 7f8edffff640 1 -- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 msgr2=0x7f8ee80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:17.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.825+0000 7f8edffff640 1 --2- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:17.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.825+0000 7f8edffff640 1 -- 172.21.15.71:0/1835889012 >> v1:172.21.15.164:6789/0 conn(0x7f8ee81509d0 legacy=0x7f8ee8150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:17.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.825+0000 7f8edffff640 1 -- 172.21.15.71:0/1835889012 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee8156e00 con 0x7f8ee80c3a40 2023-08-25T09:52:17.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.825+0000 7f8ee6ffd640 1 --2- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:52:17.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.826+0000 7f8edffff640 1 --2- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806ca30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009870 tx=0x7f8ed002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba33bc704c7811e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.827+0000 7f8ee67fc640 1 -- 172.21.15.71:0/1835889012 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed002f830 con 0x7f8ee80c3a40 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.827+0000 7f8ee67fc640 1 -- 172.21.15.71:0/1835889012 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed002f990 con 0x7f8ee80c3a40 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.827+0000 7f8ee67fc640 1 -- 172.21.15.71:0/1835889012 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed002fd20 con 0x7f8ee80c3a40 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.827+0000 7f8eed7b5640 1 -- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee80c3a40 msgr2=0x7f8ee806ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.827+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806ca30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009870 tx=0x7f8ed002ba10 comp rx=0 tx=0).stop 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.828+0000 7f8eed7b5640 1 -- 172.21.15.71:0/1835889012 shutdown_connections 2023-08-25T09:52:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.828+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806ca30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:17.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.828+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/1835889012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:17.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.828+0000 7f8eed7b5640 1 -- 172.21.15.71:0/1835889012 >> 172.21.15.71:0/1835889012 conn(0x7f8ee80654f0 msgr2=0x7f8ee8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:17.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.828+0000 7f8eed7b5640 1 -- 172.21.15.71:0/1835889012 shutdown_connections 2023-08-25T09:52:17.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.828+0000 7f8eed7b5640 1 -- 172.21.15.71:0/1835889012 wait complete. 2023-08-25T09:52:17.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.830+0000 7f8eed7b5640 1 Processor -- start 2023-08-25T09:52:17.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.830+0000 7f8eed7b5640 1 -- start start 2023-08-25T09:52:17.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.831+0000 7f8eed7b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee81522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:17.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.831+0000 7f8eed7b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:17.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.831+0000 7f8eed7b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee81509d0 0x7f8ee81ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:17.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.831+0000 7f8eed7b5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80c05e0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.831+0000 7f8eed7b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee806e620 con 0x7f8ee81509d0 2023-08-25T09:52:17.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.831+0000 7f8eed7b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee81ee9c0 con 0x7f8ee80c3660 2023-08-25T09:52:17.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:17.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54936/0 (socket says 172.21.15.71:54936) 2023-08-25T09:52:17.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 -- 172.21.15.71:0/2490576933 learned_addr learned my addr 172.21.15.71:0/2490576933 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:17.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8ee6ffd640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee81522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:17.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8ee77fe640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee81509d0 0x7f8ee81ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:17.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 -- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 msgr2=0x7f8ee81522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:17.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee81522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:17.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 -- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee81509d0 msgr2=0x7f8ee81ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:17.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee81509d0 0x7f8ee81ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:17.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 -- 172.21.15.71:0/2490576933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8009400 con 0x7f8ee80c3a40 2023-08-25T09:52:17.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.832+0000 7f8edffff640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806eab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8ed002ba50 tx=0x7f8ed0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:17.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.833+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0037c40 con 0x7f8ee80c3a40 2023-08-25T09:52:17.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.833+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed0037da0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.833+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed00426c0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.833+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed00094d0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.833+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee80c8fe0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.834+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee80c29c0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.835+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8ed002d3c0 con 0x7f8ee80c3a40 2023-08-25T09:52:17.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.836+0000 7f8edf7fe640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8ec806a050 0x7f8ec806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:17.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.836+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7f8ed003c070 con 0x7f8ee80c3a40 2023-08-25T09:52:17.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.837+0000 7f8ee6ffd640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8ec806a050 0x7f8ec806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:17.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.837+0000 7f8ee6ffd640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8ec806a050 0x7f8ec806c510 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8009bc0 tx=0x7f8ed8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:17.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:17.840+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8ed0076270 con 0x7f8ee80c3a40 2023-08-25T09:52:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.064+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ee806e7d0 con 0x7f8ee80c3a40 2023-08-25T09:52:18.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.065+0000 7f8edf7fe640 1 -- 172.21.15.71:0/2490576933 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8ed0047090 con 0x7f8ee80c3a40 2023-08-25T09:52:18.068 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:18.068 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:18.068 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:18.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8ec806a050 msgr2=0x7f8ec806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:18.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8ec806a050 0x7f8ec806c510 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8009bc0 tx=0x7f8ed8008040 comp rx=0 tx=0).stop 2023-08-25T09:52:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 msgr2=0x7f8ee806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806eab0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8ed002ba50 tx=0x7f8ed0004730 comp rx=0 tx=0).stop 2023-08-25T09:52:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 shutdown_connections 2023-08-25T09:52:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ee81509d0 0x7f8ee81ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8ec806a050 0x7f8ec806c510 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:18.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee80c3a40 0x7f8ee806eab0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:18.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 --2- 172.21.15.71:0/2490576933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ee80c3660 0x7f8ee81522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:18.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 >> 172.21.15.71:0/2490576933 conn(0x7f8ee80654f0 msgr2=0x7f8ee80658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:18.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 shutdown_connections 2023-08-25T09:52:18.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:18.068+0000 7f8eed7b5640 1 -- 172.21.15.71:0/2490576933 wait complete. 2023-08-25T09:52:18.090 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:20.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:20.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.314+0000 7f8f4901f640 1 Processor -- start 2023-08-25T09:52:20.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.315+0000 7f8f4901f640 1 -- start start 2023-08-25T09:52:20.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.316+0000 7f8f4901f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8f44150f90 0x7f8f4415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:20.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.316+0000 7f8f4901f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44070f70 0x7f8f440c3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:20.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.316+0000 7f8f4901f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f44156d20 con 0x7f8f44071390 2023-08-25T09:52:20.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.316+0000 7f8f4901f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f44156b30 con 0x7f8f44150f90 2023-08-25T09:52:20.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.316+0000 7f8f4901f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44157000 con 0x7f8f44070f70 2023-08-25T09:52:20.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8f44150f90 0x7f8f4415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:20.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8f44150f90 0x7f8f4415ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41820/0 (socket says 172.21.15.71:41820) 2023-08-25T09:52:20.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 -- 172.21.15.71:0/581430053 learned_addr learned my addr 172.21.15.71:0/581430053 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:20.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f4259c640 1 --2- 172.21.15.71:0/581430053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44070f70 0x7f8f440c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:20.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 -- 172.21.15.71:0/581430053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44070f70 msgr2=0x7f8f440c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:20.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 --2- 172.21.15.71:0/581430053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44070f70 0x7f8f440c3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 -- 172.21.15.71:0/581430053 >> v1:172.21.15.71:6789/0 conn(0x7f8f44071390 legacy=0x7f8f440c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:20.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.317+0000 7f8f42d9d640 1 -- 172.21.15.71:0/581430053 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f441572e0 con 0x7f8f44150f90 2023-08-25T09:52:20.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.318+0000 7f8f42d9d640 1 --2- 172.21.15.71:0/581430053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f4415ac30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009e50 tx=0x7f8f3402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8797e3556b3ecdd8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:20.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.318+0000 7f8f41d9b640 1 -- 172.21.15.71:0/581430053 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3402cb90 con 0x7f8f44150f90 2023-08-25T09:52:20.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f41d9b640 1 -- 172.21.15.71:0/581430053 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3402ccf0 con 0x7f8f44150f90 2023-08-25T09:52:20.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f41d9b640 1 -- 172.21.15.71:0/581430053 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34034610 con 0x7f8f44150f90 2023-08-25T09:52:20.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 -- 172.21.15.71:0/581430053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 msgr2=0x7f8f4415ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 --2- 172.21.15.71:0/581430053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f4415ac30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009e50 tx=0x7f8f3402bff0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 -- 172.21.15.71:0/581430053 shutdown_connections 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 --2- 172.21.15.71:0/581430053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44070f70 0x7f8f440c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 --2- 172.21.15.71:0/581430053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f4415ac30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 -- 172.21.15.71:0/581430053 >> 172.21.15.71:0/581430053 conn(0x7f8f44069e80 msgr2=0x7f8f440654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 -- 172.21.15.71:0/581430053 shutdown_connections 2023-08-25T09:52:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.319+0000 7f8f4901f640 1 -- 172.21.15.71:0/581430053 wait complete. 2023-08-25T09:52:20.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.321+0000 7f8f4901f640 1 Processor -- start 2023-08-25T09:52:20.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.321+0000 7f8f4901f640 1 -- start start 2023-08-25T09:52:20.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4901f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f44070f70 0x7f8f4414fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:20.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4901f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44071390 0x7f8f441efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:20.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4901f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f441ec430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:20.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4901f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f441ef950 con 0x7f8f44150f90 2023-08-25T09:52:20.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4901f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44153ea0 con 0x7f8f44070f70 2023-08-25T09:52:20.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4901f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44151ad0 con 0x7f8f44071390 2023-08-25T09:52:20.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.322+0000 7f8f4359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f441ec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:20.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f441ec430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41830/0 (socket says 172.21.15.71:41830) 2023-08-25T09:52:20.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 -- 172.21.15.71:0/42369009 learned_addr learned my addr 172.21.15.71:0/42369009 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:20.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4259c640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44071390 0x7f8f441efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:20.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f42d9d640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f44070f70 0x7f8f4414fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:20.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 -- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44071390 msgr2=0x7f8f441efde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44071390 0x7f8f441efde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 -- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f44070f70 msgr2=0x7f8f4414fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f44070f70 0x7f8f4414fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 -- 172.21.15.71:0/42369009 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34009b30 con 0x7f8f44150f90 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f4359e640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f441ec430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c00c400 tx=0x7f8f3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.323+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c0052d0 con 0x7f8f44150f90 2023-08-25T09:52:20.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.324+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3c005430 con 0x7f8f44150f90 2023-08-25T09:52:20.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.324+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c003810 con 0x7f8f44150f90 2023-08-25T09:52:20.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.324+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f44154110 con 0x7f8f44150f90 2023-08-25T09:52:20.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.324+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f441ef5e0 con 0x7f8f44150f90 2023-08-25T09:52:20.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.325+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f441efb00 con 0x7f8f44150f90 2023-08-25T09:52:20.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.326+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8f3c003970 con 0x7f8f44150f90 2023-08-25T09:52:20.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.327+0000 7f8f2b7fe640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8f3006a050 0x7f8f3006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:20.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.327+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7f8f3c087f30 con 0x7f8f44150f90 2023-08-25T09:52:20.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.327+0000 7f8f42d9d640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8f3006a050 0x7f8f3006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:20.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.328+0000 7f8f42d9d640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8f3006a050 0x7f8f3006c510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8f34035000 tx=0x7f8f34030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:20.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.332+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8f441efb00 con 0x7f8f44150f90 2023-08-25T09:52:20.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.560+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f441efb00 con 0x7f8f44150f90 2023-08-25T09:52:20.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.561+0000 7f8f2b7fe640 1 -- 172.21.15.71:0/42369009 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f8f441efb00 con 0x7f8f44150f90 2023-08-25T09:52:20.564 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:20.564 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:20.564 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.563+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8f3006a050 msgr2=0x7f8f3006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.563+0000 7f8f4901f640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8f3006a050 0x7f8f3006c510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8f34035000 tx=0x7f8f34030000 comp rx=0 tx=0).stop 2023-08-25T09:52:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.563+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 msgr2=0x7f8f441ec430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.563+0000 7f8f4901f640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f441ec430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c00c400 tx=0x7f8f3c00c8c0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 shutdown_connections 2023-08-25T09:52:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8f3006a050 0x7f8f3006c510 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f44150f90 0x7f8f441ec430 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f44071390 0x7f8f441efde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 --2- 172.21.15.71:0/42369009 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f44070f70 0x7f8f4414fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 >> 172.21.15.71:0/42369009 conn(0x7f8f44069e80 msgr2=0x7f8f440c46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:20.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 shutdown_connections 2023-08-25T09:52:20.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:20.564+0000 7f8f4901f640 1 -- 172.21.15.71:0/42369009 wait complete. 2023-08-25T09:52:20.586 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:22.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:52:22.382 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:52:22.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:22.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.803+0000 7f170008e640 1 Processor -- start 2023-08-25T09:52:22.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.804+0000 7f170008e640 1 -- start start 2023-08-25T09:52:22.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.805+0000 7f170008e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f16f80c3660 0x7f16f80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:22.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.805+0000 7f170008e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 0x7f16f806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:22.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.805+0000 7f170008e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80c1dc0 con 0x7f16f81509d0 2023-08-25T09:52:22.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.805+0000 7f170008e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f8157a80 con 0x7f16f80c3660 2023-08-25T09:52:22.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.805+0000 7f170008e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f806cf70 con 0x7f16f80c3a40 2023-08-25T09:52:22.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.806+0000 7f16fd629640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 0x7f16f806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:22.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.806+0000 7f16fde2a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f16f80c3660 0x7f16f80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:22.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.806+0000 7f16fd629640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 0x7f16f806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:36634/0 (socket says 172.21.15.71:36634) 2023-08-25T09:52:22.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.806+0000 7f16fd629640 1 -- 172.21.15.71:0/1113366005 learned_addr learned my addr 172.21.15.71:0/1113366005 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:22.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.807+0000 7f16fde2a640 1 -- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 msgr2=0x7f16f806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:22.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.807+0000 7f16fde2a640 1 --2- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 0x7f16f806ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:22.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.807+0000 7f16fde2a640 1 -- 172.21.15.71:0/1113366005 >> v1:172.21.15.164:6789/0 conn(0x7f16f81509d0 legacy=0x7f16f8150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:22.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.807+0000 7f16fde2a640 1 -- 172.21.15.71:0/1113366005 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8156e00 con 0x7f16f80c3660 2023-08-25T09:52:22.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.807+0000 7f16fd629640 1 --2- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 0x7f16f806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:52:22.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.808+0000 7f16fde2a640 1 --2- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f80c1710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f16e80097a0 tx=0x7f16e802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27f0d7f7ca22c04f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:22.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.808+0000 7f16fce28640 1 -- 172.21.15.71:0/1113366005 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8030800 con 0x7f16f80c3660 2023-08-25T09:52:22.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.808+0000 7f16fce28640 1 -- 172.21.15.71:0/1113366005 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e8030960 con 0x7f16f80c3660 2023-08-25T09:52:22.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.808+0000 7f16fce28640 1 -- 172.21.15.71:0/1113366005 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8030cf0 con 0x7f16f80c3660 2023-08-25T09:52:22.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 -- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 msgr2=0x7f16f80c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:22.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 --2- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f80c1710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f16e80097a0 tx=0x7f16e802b940 comp rx=0 tx=0).stop 2023-08-25T09:52:22.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 -- 172.21.15.71:0/1113366005 shutdown_connections 2023-08-25T09:52:22.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 --2- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f80c3a40 0x7f16f806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:22.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 --2- 172.21.15.71:0/1113366005 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f80c1710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:22.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 -- 172.21.15.71:0/1113366005 >> 172.21.15.71:0/1113366005 conn(0x7f16f80654f0 msgr2=0x7f16f8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:22.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 -- 172.21.15.71:0/1113366005 shutdown_connections 2023-08-25T09:52:22.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.809+0000 7f170008e640 1 -- 172.21.15.71:0/1113366005 wait complete. 2023-08-25T09:52:22.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.811+0000 7f170008e640 1 Processor -- start 2023-08-25T09:52:22.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.811+0000 7f170008e640 1 -- start start 2023-08-25T09:52:22.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f170008e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f170008e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 0x7f16f814b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f170008e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f81509d0 0x7f16f814b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f170008e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8154510 con 0x7f16f80c3a40 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f170008e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8155490 con 0x7f16f80c3660 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f170008e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8155d30 con 0x7f16f81509d0 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f16fd629640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 0x7f16f814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:22.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f16fd629640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 0x7f16f814b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41840/0 (socket says 172.21.15.71:41840) 2023-08-25T09:52:22.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f16fd629640 1 -- 172.21.15.71:0/2900278605 learned_addr learned my addr 172.21.15.71:0/2900278605 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.812+0000 7f16fde2a640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16fd629640 1 -- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f81509d0 msgr2=0x7f16f814b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16fd629640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f81509d0 0x7f16f814b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16fd629640 1 -- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 msgr2=0x7f16f814e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16fd629640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f814e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16fd629640 1 -- 172.21.15.71:0/2900278605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40094d0 con 0x7f16f80c3a40 2023-08-25T09:52:22.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16fd629640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 0x7f16f814b070 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9e0 tx=0x7f16f400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:22.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400c970 con 0x7f16f80c3a40 2023-08-25T09:52:22.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f400cad0 con 0x7f16f80c3a40 2023-08-25T09:52:22.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400ce40 con 0x7f16f80c3a40 2023-08-25T09:52:22.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e8009400 con 0x7f16f80c3a40 2023-08-25T09:52:22.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.813+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81567d0 con 0x7f16f80c3a40 2023-08-25T09:52:22.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.815+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f80c29a0 con 0x7f16f80c3a40 2023-08-25T09:52:22.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.815+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f16f4013470 con 0x7f16f80c3a40 2023-08-25T09:52:22.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.816+0000 7f16ee7fc640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f16e406a050 0x7f16e406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:22.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.817+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7f16f4088b10 con 0x7f16f80c3a40 2023-08-25T09:52:22.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.817+0000 7f16fde2a640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f16e406a050 0x7f16e406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:22.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.818+0000 7f16fde2a640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f16e406a050 0x7f16e406c510 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f16e8036000 tx=0x7f16e803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:22.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:22.821+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f16f4053e90 con 0x7f16f80c3a40 2023-08-25T09:52:23.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.054+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16f8069160 con 0x7f16f80c3a40 2023-08-25T09:52:23.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.055+0000 7f16ee7fc640 1 -- 172.21.15.71:0/2900278605 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f16f40574b0 con 0x7f16f80c3a40 2023-08-25T09:52:23.058 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:23.058 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:23.059 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:23.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.057+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f16e406a050 msgr2=0x7f16e406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:23.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.057+0000 7f170008e640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f16e406a050 0x7f16e406c510 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f16e8036000 tx=0x7f16e803b040 comp rx=0 tx=0).stop 2023-08-25T09:52:23.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.057+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 msgr2=0x7f16f814b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:23.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.057+0000 7f170008e640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 0x7f16f814b070 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9e0 tx=0x7f16f400eea0 comp rx=0 tx=0).stop 2023-08-25T09:52:23.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 shutdown_connections 2023-08-25T09:52:23.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f16e406a050 0x7f16e406c510 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:23.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16f81509d0 0x7f16f814b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:23.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16f80c3a40 0x7f16f814b070 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:23.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 --2- 172.21.15.71:0/2900278605 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16f80c3660 0x7f16f814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:23.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 >> 172.21.15.71:0/2900278605 conn(0x7f16f80654f0 msgr2=0x7f16f80c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:23.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 shutdown_connections 2023-08-25T09:52:23.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:23.058+0000 7f170008e640 1 -- 172.21.15.71:0/2900278605 wait complete. 2023-08-25T09:52:23.080 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:25.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:25.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.302+0000 7f5a67c1f640 1 Processor -- start 2023-08-25T09:52:25.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.302+0000 7f5a67c1f640 1 -- start start 2023-08-25T09:52:25.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.303+0000 7f5a67c1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 0x7f5a600c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a67c1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a600c1dc0 con 0x7f5a600c3660 2023-08-25T09:52:25.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a67c1f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a60156e00 con 0x7f5a600c3a40 2023-08-25T09:52:25.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a67c1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a60157a80 con 0x7f5a601510e0 2023-08-25T09:52:25.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a659bb640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5a600c3660 0x7f5a6006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60108/0 (socket says 172.21.15.71:60108) 2023-08-25T09:52:25.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a659bb640 1 -- 172.21.15.71:0/1899096369 learned_addr learned my addr 172.21.15.71:0/1899096369 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:25.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351803580 0 0) 0x7f5a600c1dc0 con 0x7f5a600c3660 2023-08-25T09:52:25.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c003520 con 0x7f5a600c3660 2023-08-25T09:52:25.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.304+0000 7f5a661bc640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 0x7f5a600c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 295510261 0 0) 0x7f5a4c003520 con 0x7f5a600c3660 2023-08-25T09:52:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 msgr2=0x7f5a600c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 0x7f5a600c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 >> v1:172.21.15.164:6789/0 conn(0x7f5a600c3a40 legacy=0x7f5a601509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a601570e0 con 0x7f5a600c3660 2023-08-25T09:52:25.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f5a5c001670 con 0x7f5a600c3660 2023-08-25T09:52:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 >> v1:172.21.15.71:6789/0 conn(0x7f5a600c3660 legacy=0x7f5a6006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a4c003a10 0x7f5a4c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 0x7f5a4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a4c004d50 0x7f5a4c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a60157a80 con 0x7f5a4c003a10 2023-08-25T09:52:25.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c00bbd0 con 0x7f5a4c004d50 2023-08-25T09:52:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c00bd90 con 0x7f5a601510e0 2023-08-25T09:52:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a4c003a10 0x7f5a4c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a651ba640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 0x7f5a4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.305+0000 7f5a659bb640 1 -- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 msgr2=0x7f5a4c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.306+0000 7f5a661bc640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a4c004d50 0x7f5a4c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.306+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a601510e0 0x7f5a4c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.306+0000 7f5a659bb640 1 -- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a4c004d50 msgr2=0x7f5a4c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.306+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a4c004d50 0x7f5a4c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.306+0000 7f5a659bb640 1 -- 172.21.15.71:0/1899096369 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c003520 con 0x7f5a4c003a10 2023-08-25T09:52:25.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.307+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a4c003a10 0x7f5a4c003e30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00c240 tx=0x7f5a4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed8b92bd1676c03 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:25.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.307+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c005ad0 con 0x7f5a4c003a10 2023-08-25T09:52:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.307+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a5c005c30 con 0x7f5a4c003a10 2023-08-25T09:52:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.307+0000 7f5a649b9640 1 -- 172.21.15.71:0/1899096369 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c02c8b0 con 0x7f5a4c003a10 2023-08-25T09:52:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.307+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1899096369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a4c003a10 msgr2=0x7f5a4c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.307+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1899096369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a4c003a10 0x7f5a4c003e30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00c240 tx=0x7f5a4c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.308+0000 7f5a661bc640 1 -- 172.21.15.71:0/1899096369 reap_dead start 2023-08-25T09:52:25.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.308+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1899096369 shutdown_connections 2023-08-25T09:52:25.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.308+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1899096369 >> 172.21.15.71:0/1899096369 conn(0x7f5a600654f0 msgr2=0x7f5a60065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:25.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.308+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1899096369 shutdown_connections 2023-08-25T09:52:25.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.308+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1899096369 wait complete. 2023-08-25T09:52:25.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.309+0000 7f5a67c1f640 1 Processor -- start 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.310+0000 7f5a67c1f640 1 -- start start 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.310+0000 7f5a67c1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4c003a10 0x7f5a601e9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.310+0000 7f5a67c1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 0x7f5a6014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a67c1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a60151aa0 0x7f5a6014a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a67c1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601492d0 con 0x7f5a601ee0a0 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a67c1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6006a0d0 con 0x7f5a60151aa0 2023-08-25T09:52:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a67c1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6006a8c0 con 0x7f5a4c003a10 2023-08-25T09:52:25.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 0x7f5a6014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 0x7f5a6014e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41856/0 (socket says 172.21.15.71:41856) 2023-08-25T09:52:25.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 -- 172.21.15.71:0/1097035034 learned_addr learned my addr 172.21.15.71:0/1097035034 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:25.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4c003a10 0x7f5a601e9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a661bc640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a60151aa0 0x7f5a6014a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 -- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4c003a10 msgr2=0x7f5a601e9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4c003a10 0x7f5a601e9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 -- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a60151aa0 msgr2=0x7f5a6014a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a60151aa0 0x7f5a6014a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.311+0000 7f5a651ba640 1 -- 172.21.15.71:0/1097035034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a601edc10 con 0x7f5a601ee0a0 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.312+0000 7f5a651ba640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 0x7f5a6014e050 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000e580 tx=0x7f5a5000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:25.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.312+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a500052e0 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.312+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a50005440 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.312+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a50003810 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.312+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6006a300 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.312+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a601ed5c0 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.313+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a60069160 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.314+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5a50003970 con 0x7f5a601ee0a0 2023-08-25T09:52:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.315+0000 7f5a567fc640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a4406a050 0x7f5a4406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:25.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.315+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7f5a50088850 con 0x7f5a601ee0a0 2023-08-25T09:52:25.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.315+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a4406a050 0x7f5a4406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:25.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.316+0000 7f5a659bb640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a4406a050 0x7f5a4406c510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c034000 tx=0x7f5a5c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:25.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.319+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5a50053b50 con 0x7f5a601ee0a0 2023-08-25T09:52:25.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.539+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a600c4350 con 0x7f5a601ee0a0 2023-08-25T09:52:25.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.540+0000 7f5a567fc640 1 -- 172.21.15.71:0/1097035034 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f5a50057170 con 0x7f5a601ee0a0 2023-08-25T09:52:25.543 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:25.543 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:25.544 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:25.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a4406a050 msgr2=0x7f5a4406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a4406a050 0x7f5a4406c510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c034000 tx=0x7f5a5c009030 comp rx=0 tx=0).stop 2023-08-25T09:52:25.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 msgr2=0x7f5a6014e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:25.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 0x7f5a6014e050 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000e580 tx=0x7f5a5000ea40 comp rx=0 tx=0).stop 2023-08-25T09:52:25.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 shutdown_connections 2023-08-25T09:52:25.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a60151aa0 0x7f5a6014a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a4406a050 0x7f5a4406c510 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5a601ee0a0 0x7f5a6014e050 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 --2- 172.21.15.71:0/1097035034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a4c003a10 0x7f5a601e9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 >> 172.21.15.71:0/1097035034 conn(0x7f5a600654f0 msgr2=0x7f5a60065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 shutdown_connections 2023-08-25T09:52:25.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:25.543+0000 7f5a67c1f640 1 -- 172.21.15.71:0/1097035034 wait complete. 2023-08-25T09:52:25.561 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:27.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:27.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.784+0000 7fc60eb24640 1 Processor -- start 2023-08-25T09:52:27.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.784+0000 7fc60eb24640 1 -- start start 2023-08-25T09:52:27.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.785+0000 7fc60eb24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc6080c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc60eb24640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60806a960 con 0x7fc6080c3940 2023-08-25T09:52:27.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc60eb24640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc60806ac20 con 0x7fc6081594d0 2023-08-25T09:52:27.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc60eb24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc60806af00 con 0x7fc60806cc70 2023-08-25T09:52:27.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc60d0c1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc6080c3940 0x7fc6081598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60112/0 (socket says 172.21.15.71:60112) 2023-08-25T09:52:27.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc60d0c1640 1 -- 172.21.15.71:0/347591347 learned_addr learned my addr 172.21.15.71:0/347591347 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:27.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc607fff640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc6080c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163340710 0 0) 0x7fc60806a960 con 0x7fc6080c3940 2023-08-25T09:52:27.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.786+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f8003520 con 0x7fc6080c3940 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232174412 0 0) 0x7fc5f8003520 con 0x7fc6080c3940 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 msgr2=0x7fc6080c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc6080c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 >> v1:172.21.15.164:6789/0 conn(0x7fc6081594d0 legacy=0x7fc60814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080c2e60 con 0x7fc6080c3940 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fc6000019f0 con 0x7fc6080c3940 2023-08-25T09:52:27.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 >> v1:172.21.15.71:6789/0 conn(0x7fc6080c3940 legacy=0x7fc6081598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f8003a10 0x7fc5f8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5f80047c0 0x7fc5f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc5f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60806af00 con 0x7fc5f8003a10 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f800dc90 con 0x7fc5f80047c0 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f800de50 con 0x7fc60806cc70 2023-08-25T09:52:27.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.787+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f8003a10 0x7fc5f8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc607fff640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5f80047c0 0x7fc5f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60c8c0640 1 -- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 msgr2=0x7fc5f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:27.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60d0c1640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc5f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc5f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60c8c0640 1 -- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5f80047c0 msgr2=0x7fc5f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5f80047c0 0x7fc5f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60c8c0640 1 -- 172.21.15.71:0/347591347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8003520 con 0x7fc5f8003a10 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.788+0000 7fc60d0c1640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc60806cc70 0x7fc5f8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.789+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f8003a10 0x7fc5f8003e30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc001750 tx=0x7fc5f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b964cb302b1f58e1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.789+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc0052d0 con 0x7fc5f8003a10 2023-08-25T09:52:27.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.789+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc02e050 con 0x7fc5f8003a10 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.789+0000 7fc6077fe640 1 -- 172.21.15.71:0/347591347 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc0035f0 con 0x7fc5f8003a10 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60eb24640 1 -- 172.21.15.71:0/347591347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f8003a10 msgr2=0x7fc5f8003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60eb24640 1 --2- 172.21.15.71:0/347591347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5f8003a10 0x7fc5f8003e30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc001750 tx=0x7fc5f80057f0 comp rx=0 tx=0).stop 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60d0c1640 1 -- 172.21.15.71:0/347591347 reap_dead start 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60eb24640 1 -- 172.21.15.71:0/347591347 shutdown_connections 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60eb24640 1 -- 172.21.15.71:0/347591347 >> 172.21.15.71:0/347591347 conn(0x7fc6080654f0 msgr2=0x7fc608065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:27.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60eb24640 1 -- 172.21.15.71:0/347591347 shutdown_connections 2023-08-25T09:52:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.790+0000 7fc60eb24640 1 -- 172.21.15.71:0/347591347 wait complete. 2023-08-25T09:52:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.792+0000 7fc60eb24640 1 Processor -- start 2023-08-25T09:52:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.792+0000 7fc60eb24640 1 -- start start 2023-08-25T09:52:27.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60eb24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5f8003a10 0x7fc6080c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60eb24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6081eab00 0x7fc608156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60eb24640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 0x7fc6080c1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60eb24640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6081e9e30 con 0x7fc60813f460 2023-08-25T09:52:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60eb24640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc608157370 con 0x7fc6081eab00 2023-08-25T09:52:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60eb24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc608158e70 con 0x7fc5f8003a10 2023-08-25T09:52:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60d0c1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 0x7fc6080c1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60d0c1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 0x7fc6080c1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41866/0 (socket says 172.21.15.71:41866) 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.793+0000 7fc60d0c1640 1 -- 172.21.15.71:0/4266052022 learned_addr learned my addr 172.21.15.71:0/4266052022 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc607fff640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6081eab00 0x7fc608156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5f8003a10 0x7fc6080c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60d0c1640 1 -- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5f8003a10 msgr2=0x7fc6080c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60d0c1640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5f8003a10 0x7fc6080c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60d0c1640 1 -- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6081eab00 msgr2=0x7fc608156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60d0c1640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6081eab00 0x7fc608156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60d0c1640 1 -- 172.21.15.71:0/4266052022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6081eddb0 con 0x7fc60813f460 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc607fff640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6081eab00 0x7fc608156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60d0c1640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 0x7fc6080c1c60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc608071e00 tx=0x7fc60000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600004d60 con 0x7fc60813f460 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60000bb80 con 0x7fc60813f460 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600007070 con 0x7fc60813f460 2023-08-25T09:52:27.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.794+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6081ee090 con 0x7fc60813f460 2023-08-25T09:52:27.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.795+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6081ea300 con 0x7fc60813f460 2023-08-25T09:52:27.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.796+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc608069160 con 0x7fc60813f460 2023-08-25T09:52:27.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.797+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc6000071d0 con 0x7fc60813f460 2023-08-25T09:52:27.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.797+0000 7fc6057fa640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ec06a050 0x7fc5ec06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:27.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.798+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7fc600088770 con 0x7fc60813f460 2023-08-25T09:52:27.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.798+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ec06a050 0x7fc5ec06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:27.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.798+0000 7fc60c8c0640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ec06a050 0x7fc5ec06c510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc033000 tx=0x7fc5fc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:27.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:27.802+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc600053a60 con 0x7fc60813f460 2023-08-25T09:52:28.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.029+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc60806e100 con 0x7fc60813f460 2023-08-25T09:52:28.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.030+0000 7fc6057fa640 1 -- 172.21.15.71:0/4266052022 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fc600057080 con 0x7fc60813f460 2023-08-25T09:52:28.033 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:28.033 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:28.033 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:28.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ec06a050 msgr2=0x7fc5ec06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:28.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ec06a050 0x7fc5ec06c510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc033000 tx=0x7fc5fc009030 comp rx=0 tx=0).stop 2023-08-25T09:52:28.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 msgr2=0x7fc6080c1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:28.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 0x7fc6080c1c60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc608071e00 tx=0x7fc60000b4a0 comp rx=0 tx=0).stop 2023-08-25T09:52:28.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 shutdown_connections 2023-08-25T09:52:28.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6081eab00 0x7fc608156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ec06a050 0x7fc5ec06c510 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc60813f460 0x7fc6080c1c60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.032+0000 7fc60eb24640 1 --2- 172.21.15.71:0/4266052022 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5f8003a10 0x7fc6080c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.033+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 >> 172.21.15.71:0/4266052022 conn(0x7fc6080654f0 msgr2=0x7fc608149300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.033+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 shutdown_connections 2023-08-25T09:52:28.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:28.033+0000 7fc60eb24640 1 -- 172.21.15.71:0/4266052022 wait complete. 2023-08-25T09:52:28.054 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:30.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:30.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.279+0000 7fcda3342640 1 Processor -- start 2023-08-25T09:52:30.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.279+0000 7fcda3342640 1 -- start start 2023-08-25T09:52:30.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda3342640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd9c06e400 0x7fcd9c156be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:30.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda3342640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcd9c157120 0x7fcd9c161560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:30.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda3342640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c0c7690 con 0x7fcd9c150d60 2023-08-25T09:52:30.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda3342640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c157590 con 0x7fcd9c06e400 2023-08-25T09:52:30.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda3342640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c151140 con 0x7fcd9c157120 2023-08-25T09:52:30.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda08dd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd9c06e400 0x7fcd9c156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:30.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda08dd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd9c06e400 0x7fcd9c156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45188/0 (socket says 172.21.15.71:45188) 2023-08-25T09:52:30.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.281+0000 7fcda08dd640 1 -- 172.21.15.71:0/252302752 learned_addr learned my addr 172.21.15.71:0/252302752 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:30.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.282+0000 7fcda18df640 1 --2- 172.21.15.71:0/252302752 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c157120 0x7fcd9c161560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:30.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.282+0000 7fcda08dd640 1 -- 172.21.15.71:0/252302752 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c157120 msgr2=0x7fcd9c161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:30.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.282+0000 7fcda08dd640 1 --2- 172.21.15.71:0/252302752 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c157120 0x7fcd9c161560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.282+0000 7fcda08dd640 1 -- 172.21.15.71:0/252302752 >> v1:172.21.15.164:6789/0 conn(0x7fcd9c150d60 legacy=0x7fcd9c0c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:30.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.282+0000 7fcda08dd640 1 -- 172.21.15.71:0/252302752 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0c7f30 con 0x7fcd9c06e400 2023-08-25T09:52:30.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.282+0000 7fcda18df640 1 --2- 172.21.15.71:0/252302752 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c157120 0x7fcd9c161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:30.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.283+0000 7fcda08dd640 1 --2- 172.21.15.71:0/252302752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c06e400 0x7fcd9c156be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0ca330 tx=0x7fcd9002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71dcc530e2f4832 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:30.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.283+0000 7fcd8ffff640 1 -- 172.21.15.71:0/252302752 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9002c9f0 con 0x7fcd9c06e400 2023-08-25T09:52:30.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.283+0000 7fcd8ffff640 1 -- 172.21.15.71:0/252302752 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9002cb50 con 0x7fcd9c06e400 2023-08-25T09:52:30.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.283+0000 7fcd8ffff640 1 -- 172.21.15.71:0/252302752 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd900343f0 con 0x7fcd9c06e400 2023-08-25T09:52:30.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 -- 172.21.15.71:0/252302752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c06e400 msgr2=0x7fcd9c156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:30.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 --2- 172.21.15.71:0/252302752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c06e400 0x7fcd9c156be0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0ca330 tx=0x7fcd9002be50 comp rx=0 tx=0).stop 2023-08-25T09:52:30.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 -- 172.21.15.71:0/252302752 shutdown_connections 2023-08-25T09:52:30.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 --2- 172.21.15.71:0/252302752 >> v2:172.21.15.164:3300/0 conn(0x7fcd9c157120 0x7fcd9c161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 --2- 172.21.15.71:0/252302752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c06e400 0x7fcd9c156be0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 -- 172.21.15.71:0/252302752 >> 172.21.15.71:0/252302752 conn(0x7fcd9c0654f0 msgr2=0x7fcd9c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:30.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 -- 172.21.15.71:0/252302752 shutdown_connections 2023-08-25T09:52:30.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.284+0000 7fcda3342640 1 -- 172.21.15.71:0/252302752 wait complete. 2023-08-25T09:52:30.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.286+0000 7fcda3342640 1 Processor -- start 2023-08-25T09:52:30.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.286+0000 7fcda3342640 1 -- start start 2023-08-25T09:52:30.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.287+0000 7fcda3342640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c06e400 0x7fcd9c1563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:30.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.287+0000 7fcda3342640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c150d60 0x7fcd9c1f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:30.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.287+0000 7fcda3342640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 0x7fcd9c14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:30.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.287+0000 7fcda3342640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c14b1f0 con 0x7fcd9c157120 2023-08-25T09:52:30.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.287+0000 7fcda3342640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0c2950 con 0x7fcd9c06e400 2023-08-25T09:52:30.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.287+0000 7fcda3342640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c14b400 con 0x7fcd9c150d60 2023-08-25T09:52:30.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 0x7fcd9c14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:30.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 0x7fcd9c14c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45200/0 (socket says 172.21.15.71:45200) 2023-08-25T09:52:30.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 -- 172.21.15.71:0/843498030 learned_addr learned my addr 172.21.15.71:0/843498030 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:30.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda10de640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c06e400 0x7fcd9c1563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:30.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda08dd640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c150d60 0x7fcd9c1f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:30.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 -- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c150d60 msgr2=0x7fcd9c1f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:30.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c150d60 0x7fcd9c1f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 -- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c06e400 msgr2=0x7fcd9c1563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:30.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c06e400 0x7fcd9c1563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.288+0000 7fcda18df640 1 -- 172.21.15.71:0/843498030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90009b30 con 0x7fcd9c157120 2023-08-25T09:52:30.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.289+0000 7fcda18df640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 0x7fcd9c14c2f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400c820 tx=0x7fcd9400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:30.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.289+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94007a30 con 0x7fcd9c157120 2023-08-25T09:52:30.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.289+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94007b90 con 0x7fcd9c157120 2023-08-25T09:52:30.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.289+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c1f2080 con 0x7fcd9c157120 2023-08-25T09:52:30.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.289+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9400ce10 con 0x7fcd9c157120 2023-08-25T09:52:30.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.289+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c06c860 con 0x7fcd9c157120 2023-08-25T09:52:30.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.290+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c0c2b00 con 0x7fcd9c157120 2023-08-25T09:52:30.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.291+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcd940113f0 con 0x7fcd9c157120 2023-08-25T09:52:30.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.292+0000 7fcd8dffb640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcd8006a050 0x7fcd8006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:30.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.292+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6927+0+0 (secure 0 0 0) 0x7fcd94088ba0 con 0x7fcd9c157120 2023-08-25T09:52:30.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.292+0000 7fcda10de640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcd8006a050 0x7fcd8006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:30.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.293+0000 7fcda10de640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcd8006a050 0x7fcd8006c510 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd98007fd0 tx=0x7fcd98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:30.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.296+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fcd9c0c2b00 con 0x7fcd9c157120 2023-08-25T09:52:30.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.521+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9c0c2b00 con 0x7fcd9c157120 2023-08-25T09:52:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.522+0000 7fcd8dffb640 1 -- 172.21.15.71:0/843498030 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fcd9c0c2b00 con 0x7fcd9c157120 2023-08-25T09:52:30.524 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:30.524 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:10.823440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":15,"addr":"172.21.15.164:6835/2707317155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2707317155},{"type":"v1","addr":"172.21.15.164:6835","nonce":2707317155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 21 2023-08-25T09:52:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcd8006a050 msgr2=0x7fcd8006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcd8006a050 0x7fcd8006c510 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd98007fd0 tx=0x7fcd98009040 comp rx=0 tx=0).stop 2023-08-25T09:52:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 msgr2=0x7fcd9c14c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 0x7fcd9c14c2f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400c820 tx=0x7fcd9400cce0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 shutdown_connections 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcd8006a050 0x7fcd8006c510 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd9c157120 0x7fcd9c14c2f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd9c150d60 0x7fcd9c1f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 --2- 172.21.15.71:0/843498030 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd9c06e400 0x7fcd9c1563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 >> 172.21.15.71:0/843498030 conn(0x7fcd9c0654f0 msgr2=0x7fcd9c0707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 shutdown_connections 2023-08-25T09:52:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:30.525+0000 7fcda3342640 1 -- 172.21.15.71:0/843498030 wait complete. 2023-08-25T09:52:30.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-08-25T09:52:32.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:32.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.768+0000 7fc2d3dc2640 1 Processor -- start 2023-08-25T09:52:32.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.768+0000 7fc2d3dc2640 1 -- start start 2023-08-25T09:52:32.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.769+0000 7fc2d3dc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc1594d0 0x7fc2cc0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.769+0000 7fc2d3dc2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc2cc06cc70 0x7fc2cc14f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.769+0000 7fc2d3dc2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc2cc0c3940 0x7fc2cc14fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.769+0000 7fc2d3dc2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2cc06a960 con 0x7fc2cc06cc70 2023-08-25T09:52:32.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.769+0000 7fc2d3dc2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2cc06ac20 con 0x7fc2cc0c3940 2023-08-25T09:52:32.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.769+0000 7fc2d3dc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc06af00 con 0x7fc2cc1594d0 2023-08-25T09:52:32.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc2cc06cc70 0x7fc2cc14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc2cc06cc70 0x7fc2cc14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45214/0 (socket says 172.21.15.71:45214) 2023-08-25T09:52:32.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 -- 172.21.15.71:0/859533605 learned_addr learned my addr 172.21.15.71:0/859533605 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:32.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d1b5e640 1 --2- 172.21.15.71:0/859533605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc1594d0 0x7fc2cc0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d235f640 1 --2- 172.21.15.71:0/859533605 >> v2:172.21.15.164:3300/0 conn(0x7fc2cc0c3940 0x7fc2cc14fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 -- 172.21.15.71:0/859533605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc1594d0 msgr2=0x7fc2cc0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:32.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 --2- 172.21.15.71:0/859533605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc1594d0 0x7fc2cc0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 -- 172.21.15.71:0/859533605 >> v2:172.21.15.164:3300/0 conn(0x7fc2cc0c3940 msgr2=0x7fc2cc14fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:32.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 --2- 172.21.15.71:0/859533605 >> v2:172.21.15.164:3300/0 conn(0x7fc2cc0c3940 0x7fc2cc14fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.770+0000 7fc2d135d640 1 -- 172.21.15.71:0/859533605 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0c2e60 con 0x7fc2cc06cc70 2023-08-25T09:52:32.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.772+0000 7fc2d135d640 1 --2- 172.21.15.71:0/859533605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14f850 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc071cc0 tx=0x7fc2bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265346b8314343cc server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:32.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.772+0000 7fc2d0b5c640 1 -- 172.21.15.71:0/859533605 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc031870 con 0x7fc2cc06cc70 2023-08-25T09:52:32.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.772+0000 7fc2d0b5c640 1 -- 172.21.15.71:0/859533605 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc0319d0 con 0x7fc2cc06cc70 2023-08-25T09:52:32.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.772+0000 7fc2d0b5c640 1 -- 172.21.15.71:0/859533605 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc031d60 con 0x7fc2cc06cc70 2023-08-25T09:52:32.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/859533605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc06cc70 msgr2=0x7fc2cc14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/859533605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14f850 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc071cc0 tx=0x7fc2bc02b910 comp rx=0 tx=0).stop 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/859533605 shutdown_connections 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/859533605 >> v2:172.21.15.164:3300/0 conn(0x7fc2cc0c3940 0x7fc2cc14fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/859533605 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14f850 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/859533605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc1594d0 0x7fc2cc0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/859533605 >> 172.21.15.71:0/859533605 conn(0x7fc2cc0654f0 msgr2=0x7fc2cc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:32.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/859533605 shutdown_connections 2023-08-25T09:52:32.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.773+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/859533605 wait complete. 2023-08-25T09:52:32.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.775+0000 7fc2d3dc2640 1 Processor -- start 2023-08-25T09:52:32.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.775+0000 7fc2d3dc2640 1 -- start start 2023-08-25T09:52:32.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d3dc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d3dc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc0c3940 0x7fc2cc13fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d3dc2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 0x7fc2cc140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d3dc2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc1f2170 con 0x7fc2cc1594d0 2023-08-25T09:52:32.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d3dc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0c6480 con 0x7fc2cc06cc70 2023-08-25T09:52:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d3dc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc1f14f0 con 0x7fc2cc0c3940 2023-08-25T09:52:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d235f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 0x7fc2cc140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d235f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 0x7fc2cc140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45228/0 (socket says 172.21.15.71:45228) 2023-08-25T09:52:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d235f640 1 -- 172.21.15.71:0/4267973859 learned_addr learned my addr 172.21.15.71:0/4267973859 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d1b5e640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.776+0000 7fc2d135d640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc0c3940 0x7fc2cc13fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d235f640 1 -- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc0c3940 msgr2=0x7fc2cc13fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:32.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d235f640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc0c3940 0x7fc2cc13fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d235f640 1 -- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2cc06cc70 msgr2=0x7fc2cc14e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:32.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d235f640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:32.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d235f640 1 -- 172.21.15.71:0/4267973859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0094d0 con 0x7fc2cc1594d0 2023-08-25T09:52:32.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d235f640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 0x7fc2cc140300 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400eed0 tx=0x7fc2c400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:32.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4010040 con 0x7fc2cc1594d0 2023-08-25T09:52:32.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c400bd20 con 0x7fc2cc1594d0 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4014640 con 0x7fc2cc1594d0 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc1f17d0 con 0x7fc2cc1594d0 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.777+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc1f1d90 con 0x7fc2cc1594d0 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.778+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc1ede00 con 0x7fc2cc1594d0 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.779+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc2c40147a0 con 0x7fc2cc1594d0 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.781+0000 7fc2c27fc640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc2b006a080 0x7fc2b006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:32.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.781+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc2c4088fe0 con 0x7fc2cc1594d0 2023-08-25T09:52:32.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.781+0000 7fc2d1b5e640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc2b006a080 0x7fc2b006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:32.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.782+0000 7fc2d1b5e640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc2b006a080 0x7fc2b006c540 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009ad0 tx=0x7fc2c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:32.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:32.785+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc2c40543a0 con 0x7fc2cc1594d0 2023-08-25T09:52:33.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.013+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2cc1f22b0 con 0x7fc2cc1594d0 2023-08-25T09:52:33.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.014+0000 7fc2c27fc640 1 -- 172.21.15.71:0/4267973859 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fc2c4018090 con 0x7fc2cc1594d0 2023-08-25T09:52:33.016 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:33.016 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:30.557371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:33.016 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 22 2023-08-25T09:52:33.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc2b006a080 msgr2=0x7fc2b006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc2b006a080 0x7fc2b006c540 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009ad0 tx=0x7fc2c8008040 comp rx=0 tx=0).stop 2023-08-25T09:52:33.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 msgr2=0x7fc2cc140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 0x7fc2cc140300 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400eed0 tx=0x7fc2c400b480 comp rx=0 tx=0).stop 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 shutdown_connections 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc2b006a080 0x7fc2b006c540 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2cc1594d0 0x7fc2cc140300 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2cc0c3940 0x7fc2cc13fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 --2- 172.21.15.71:0/4267973859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2cc06cc70 0x7fc2cc14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.016+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 >> 172.21.15.71:0/4267973859 conn(0x7fc2cc0654f0 msgr2=0x7fc2cc0c5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:33.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.017+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 shutdown_connections 2023-08-25T09:52:33.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.017+0000 7fc2d3dc2640 1 -- 172.21.15.71:0/4267973859 wait complete. 2023-08-25T09:52:33.036 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-08-25T09:52:33.036 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-08-25T09:52:33.036 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:33.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.246+0000 7f3aa8046640 1 Processor -- start 2023-08-25T09:52:33.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.246+0000 7f3aa8046640 1 -- start start 2023-08-25T09:52:33.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.247+0000 7f3aa8046640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3aa00c5f60 0x7f3aa0155c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:33.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.247+0000 7f3aa8046640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa0069850 0x7f3aa014f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:33.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa8046640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa01561a0 con 0x7f3aa00c6340 2023-08-25T09:52:33.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa8046640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa00c5760 con 0x7f3aa00c5f60 2023-08-25T09:52:33.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa8046640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00c5570 con 0x7f3aa0069850 2023-08-25T09:52:33.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3aa00c5f60 0x7f3aa0155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:33.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3aa00c5f60 0x7f3aa0155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45236/0 (socket says 172.21.15.71:45236) 2023-08-25T09:52:33.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 -- 172.21.15.71:0/528572751 learned_addr learned my addr 172.21.15.71:0/528572751 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:33.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa65e3640 1 --2- 172.21.15.71:0/528572751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa0069850 0x7f3aa014f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 -- 172.21.15.71:0/528572751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa0069850 msgr2=0x7f3aa014f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 --2- 172.21.15.71:0/528572751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa0069850 0x7f3aa014f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 -- 172.21.15.71:0/528572751 >> v1:172.21.15.164:6789/0 conn(0x7f3aa00c6340 legacy=0x7f3aa00c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.248+0000 7f3aa5de2640 1 -- 172.21.15.71:0/528572751 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa006fa50 con 0x7f3aa00c5f60 2023-08-25T09:52:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.249+0000 7f3aa5de2640 1 --2- 172.21.15.71:0/528572751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa00c5f60 0x7f3aa0155c60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009e50 tx=0x7f3a9402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eecbda015785345 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa4de0640 1 -- 172.21.15.71:0/528572751 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9402ed10 con 0x7f3aa00c5f60 2023-08-25T09:52:33.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa4de0640 1 -- 172.21.15.71:0/528572751 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9402ee70 con 0x7f3aa00c5f60 2023-08-25T09:52:33.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa4de0640 1 -- 172.21.15.71:0/528572751 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a940365f0 con 0x7f3aa00c5f60 2023-08-25T09:52:33.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa8046640 1 -- 172.21.15.71:0/528572751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa00c5f60 msgr2=0x7f3aa0155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa8046640 1 --2- 172.21.15.71:0/528572751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa00c5f60 0x7f3aa0155c60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009e50 tx=0x7f3a9402e170 comp rx=0 tx=0).stop 2023-08-25T09:52:33.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa8046640 1 -- 172.21.15.71:0/528572751 shutdown_connections 2023-08-25T09:52:33.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa8046640 1 --2- 172.21.15.71:0/528572751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa0069850 0x7f3aa014f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa8046640 1 --2- 172.21.15.71:0/528572751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa00c5f60 0x7f3aa0155c60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.250+0000 7f3aa8046640 1 -- 172.21.15.71:0/528572751 >> 172.21.15.71:0/528572751 conn(0x7f3aa00654f0 msgr2=0x7f3aa0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:33.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.251+0000 7f3aa8046640 1 -- 172.21.15.71:0/528572751 shutdown_connections 2023-08-25T09:52:33.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.251+0000 7f3aa8046640 1 -- 172.21.15.71:0/528572751 wait complete. 2023-08-25T09:52:33.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.253+0000 7f3aa8046640 1 Processor -- start 2023-08-25T09:52:33.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.253+0000 7f3aa8046640 1 -- start start 2023-08-25T09:52:33.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa8046640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 0x7f3aa014de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:33.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa8046640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aa00c5f60 0x7f3aa01df900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:33.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa8046640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa00c6340 0x7f3aa01f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:33.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa8046640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa014cb50 con 0x7f3aa0069850 2023-08-25T09:52:33.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa8046640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa014e340 con 0x7f3aa00c5f60 2023-08-25T09:52:33.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa8046640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa014cd80 con 0x7f3aa00c6340 2023-08-25T09:52:33.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa5de2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 0x7f3aa014de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:33.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa5de2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 0x7f3aa014de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45250/0 (socket says 172.21.15.71:45250) 2023-08-25T09:52:33.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.254+0000 7f3aa5de2640 1 -- 172.21.15.71:0/96300604 learned_addr learned my addr 172.21.15.71:0/96300604 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:33.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa55e1640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aa00c5f60 0x7f3aa01df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:33.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa65e3640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa00c6340 0x7f3aa01f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:33.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa5de2640 1 -- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa00c6340 msgr2=0x7f3aa01f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa5de2640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa00c6340 0x7f3aa01f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa5de2640 1 -- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aa00c5f60 msgr2=0x7f3aa01df900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa5de2640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aa00c5f60 0x7f3aa01df900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa5de2640 1 -- 172.21.15.71:0/96300604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a94009b30 con 0x7f3aa0069850 2023-08-25T09:52:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa65e3640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa00c6340 0x7f3aa01f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa55e1640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aa00c5f60 0x7f3aa01df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa5de2640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 0x7f3aa014de00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3a94038040 tx=0x7f3a94004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a94044070 con 0x7f3aa0069850 2023-08-25T09:52:33.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9400e120 con 0x7f3aa0069850 2023-08-25T09:52:33.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.255+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa014d040 con 0x7f3aa0069850 2023-08-25T09:52:33.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.256+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a94040670 con 0x7f3aa0069850 2023-08-25T09:52:33.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.256+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa014d600 con 0x7f3aa0069850 2023-08-25T09:52:33.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.257+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa0069160 con 0x7f3aa0069850 2023-08-25T09:52:33.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.258+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3a940407d0 con 0x7f3aa0069850 2023-08-25T09:52:33.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.258+0000 7f3a9e7fc640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3a8406a080 0x7f3a8406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:33.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.259+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3a940aa830 con 0x7f3aa0069850 2023-08-25T09:52:33.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.259+0000 7f3aa55e1640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3a8406a080 0x7f3a8406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:33.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.260+0000 7f3aa55e1640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3a8406a080 0x7f3a8406c540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa01de8c0 tx=0x7f3a8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:33.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.263+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3a94075b10 con 0x7f3aa0069850 2023-08-25T09:52:33.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.488+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3aa0156180 con 0x7f3aa0069850 2023-08-25T09:52:33.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.490+0000 7f3a9e7fc640 1 -- 172.21.15.71:0/96300604 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f3a94079130 con 0x7f3aa0069850 2023-08-25T09:52:33.492 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:33.492 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:30.557371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:33.492 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 22 2023-08-25T09:52:33.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.492+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3a8406a080 msgr2=0x7f3a8406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.492+0000 7f3aa8046640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3a8406a080 0x7f3a8406c540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa01de8c0 tx=0x7f3a8c008040 comp rx=0 tx=0).stop 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.492+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 msgr2=0x7f3aa014de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.492+0000 7f3aa8046640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 0x7f3aa014de00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3a94038040 tx=0x7f3a94004ab0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 shutdown_connections 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3aa00c5f60 0x7f3aa01df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3a8406a080 0x7f3a8406c540 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3aa00c6340 0x7f3aa01f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 --2- 172.21.15.71:0/96300604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3aa0069850 0x7f3aa014de00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:33.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 >> 172.21.15.71:0/96300604 conn(0x7f3aa00654f0 msgr2=0x7f3aa014ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:33.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 shutdown_connections 2023-08-25T09:52:33.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:33.493+0000 7f3aa8046640 1 -- 172.21.15.71:0/96300604 wait complete. 2023-08-25T09:52:33.513 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4235": { "addr": "172.21.15.164:6833/2402500458", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 2402500458, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 2402500458, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4235, "incarnation": 22, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4287": { "addr": "172.21.15.71:6836/1556118065", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 1556118065, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 1556118065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4287, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_4290": { "addr": "172.21.15.71:6837/3404832976", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3404832976, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3404832976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:52:30.557371+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 4290, "mds_3": 4287, "mds_4": 4235 } } } ], "standbys": [] } 2023-08-25T09:52:35.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:35.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.735+0000 7f3d96784640 1 Processor -- start 2023-08-25T09:52:35.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.736+0000 7f3d96784640 1 -- start start 2023-08-25T09:52:35.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.737+0000 7f3d96784640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d90152d80 0x7f3d9006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:35.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.737+0000 7f3d96784640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d90070cf0 0x7f3d901637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:35.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d96784640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9006eb60 con 0x7f3d900c4560 2023-08-25T09:52:35.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d96784640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9006ca30 con 0x7f3d90070cf0 2023-08-25T09:52:35.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d96784640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9006cf90 con 0x7f3d90152d80 2023-08-25T09:52:35.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d8f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3d900c4560 0x7f3d900c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38280/0 (socket says 172.21.15.71:38280) 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d8f7fe640 1 -- 172.21.15.71:0/1315415063 learned_addr learned my addr 172.21.15.71:0/1315415063 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d8ffff640 1 --2- 172.21.15.71:0/1315415063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d90152d80 0x7f3d9006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d94d21640 1 --2- 172.21.15.71:0/1315415063 >> v2:172.21.15.164:3300/0 conn(0x7f3d90070cf0 0x7f3d901637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.738+0000 7f3d8effd640 1 -- 172.21.15.71:0/1315415063 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474402913 0 0) 0x7f3d9006eb60 con 0x7f3d900c4560 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.739+0000 7f3d8effd640 1 -- 172.21.15.71:0/1315415063 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d78003520 con 0x7f3d900c4560 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.739+0000 7f3d8ffff640 1 -- 172.21.15.71:0/1315415063 >> v2:172.21.15.164:3300/0 conn(0x7f3d90070cf0 msgr2=0x7f3d901637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:35.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.739+0000 7f3d8ffff640 1 --2- 172.21.15.71:0/1315415063 >> v2:172.21.15.164:3300/0 conn(0x7f3d90070cf0 0x7f3d901637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.739+0000 7f3d8ffff640 1 -- 172.21.15.71:0/1315415063 >> v1:172.21.15.71:6789/0 conn(0x7f3d900c4560 legacy=0x7f3d900c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:35.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.739+0000 7f3d8ffff640 1 -- 172.21.15.71:0/1315415063 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d90158fd0 con 0x7f3d90152d80 2023-08-25T09:52:35.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.740+0000 7f3d8ffff640 1 --2- 172.21.15.71:0/1315415063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d90152d80 0x7f3d9006e620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d800097a0 tx=0x7f3d8002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed5d8dcbfc9e2695 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:35.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d8effd640 1 -- 172.21.15.71:0/1315415063 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80031780 con 0x7f3d90152d80 2023-08-25T09:52:35.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d8effd640 1 -- 172.21.15.71:0/1315415063 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d800318e0 con 0x7f3d90152d80 2023-08-25T09:52:35.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d8effd640 1 -- 172.21.15.71:0/1315415063 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80031c70 con 0x7f3d90152d80 2023-08-25T09:52:35.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d96784640 1 -- 172.21.15.71:0/1315415063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d90152d80 msgr2=0x7f3d9006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d96784640 1 --2- 172.21.15.71:0/1315415063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d90152d80 0x7f3d9006e620 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d800097a0 tx=0x7f3d8002b820 comp rx=0 tx=0).stop 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d96784640 1 -- 172.21.15.71:0/1315415063 shutdown_connections 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d96784640 1 --2- 172.21.15.71:0/1315415063 >> v2:172.21.15.164:3300/0 conn(0x7f3d90070cf0 0x7f3d901637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d96784640 1 --2- 172.21.15.71:0/1315415063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d90152d80 0x7f3d9006e620 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.741+0000 7f3d96784640 1 -- 172.21.15.71:0/1315415063 >> 172.21.15.71:0/1315415063 conn(0x7f3d900654f0 msgr2=0x7f3d90065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.742+0000 7f3d96784640 1 -- 172.21.15.71:0/1315415063 shutdown_connections 2023-08-25T09:52:35.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.742+0000 7f3d96784640 1 -- 172.21.15.71:0/1315415063 wait complete. 2023-08-25T09:52:35.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.743+0000 7f3d96784640 1 Processor -- start 2023-08-25T09:52:35.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 -- start start 2023-08-25T09:52:35.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 0x7f3d901f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:35.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d900c4560 0x7f3d901ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:35.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d90152d80 0x7f3d9006a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:35.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90153720 con 0x7f3d90070cf0 2023-08-25T09:52:35.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901f1ca0 con 0x7f3d90152d80 2023-08-25T09:52:35.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.744+0000 7f3d96784640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901f1690 con 0x7f3d900c4560 2023-08-25T09:52:35.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 0x7f3d901f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:35.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 0x7f3d901f2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45252/0 (socket says 172.21.15.71:45252) 2023-08-25T09:52:35.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 -- 172.21.15.71:0/2071640964 learned_addr learned my addr 172.21.15.71:0/2071640964 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:35.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d94d21640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d90152d80 0x7f3d9006a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:35.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8f7fe640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d900c4560 0x7f3d901ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:35.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 -- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d900c4560 msgr2=0x7f3d901ee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d900c4560 0x7f3d901ee080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 -- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d90152d80 msgr2=0x7f3d9006a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d90152d80 0x7f3d9006a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d8ffff640 1 -- 172.21.15.71:0/2071640964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d80009400 con 0x7f3d90070cf0 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.745+0000 7f3d94d21640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d90152d80 0x7f3d9006a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.746+0000 7f3d8ffff640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 0x7f3d901f2130 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3d80034040 tx=0x7f3d800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.746+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80032820 con 0x7f3d90070cf0 2023-08-25T09:52:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.746+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80032980 con 0x7f3d90070cf0 2023-08-25T09:52:35.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.746+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80032d30 con 0x7f3d90070cf0 2023-08-25T09:52:35.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.746+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d90153970 con 0x7f3d90070cf0 2023-08-25T09:52:35.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.746+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9015b7e0 con 0x7f3d90070cf0 2023-08-25T09:52:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.747+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d90069160 con 0x7f3d90070cf0 2023-08-25T09:52:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.748+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3d8004e440 con 0x7f3d90070cf0 2023-08-25T09:52:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.749+0000 7f3d8cff9640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d7c06a050 0x7f3d7c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.749+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3d800aa930 con 0x7f3d90070cf0 2023-08-25T09:52:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.749+0000 7f3d8f7fe640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d7c06a050 0x7f3d7c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:35.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.750+0000 7f3d8f7fe640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d7c06a050 0x7f3d7c06c510 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d900c3ad0 tx=0x7f3d84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:35.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.754+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3d80075c90 con 0x7f3d90070cf0 2023-08-25T09:52:35.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.980+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d9015d2c0 con 0x7f3d90070cf0 2023-08-25T09:52:35.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.982+0000 7f3d8cff9640 1 -- 172.21.15.71:0/2071640964 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f3d800792b0 con 0x7f3d90070cf0 2023-08-25T09:52:35.984 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:35.984 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:30.557371+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:35.984 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 22 2023-08-25T09:52:35.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d7c06a050 msgr2=0x7f3d7c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:35.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d7c06a050 0x7f3d7c06c510 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d900c3ad0 tx=0x7f3d84009040 comp rx=0 tx=0).stop 2023-08-25T09:52:35.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 msgr2=0x7f3d901f2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 0x7f3d901f2130 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3d80034040 tx=0x7f3d800046c0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 shutdown_connections 2023-08-25T09:52:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d90152d80 0x7f3d9006a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3d7c06a050 0x7f3d7c06c510 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d900c4560 0x7f3d901ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 --2- 172.21.15.71:0/2071640964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3d90070cf0 0x7f3d901f2130 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:35.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 >> 172.21.15.71:0/2071640964 conn(0x7f3d900654f0 msgr2=0x7f3d900c8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:35.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 shutdown_connections 2023-08-25T09:52:35.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:35.984+0000 7f3d96784640 1 -- 172.21.15.71:0/2071640964 wait complete. 2023-08-25T09:52:36.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:38.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:38.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.230+0000 7fdf0a606640 1 Processor -- start 2023-08-25T09:52:38.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.230+0000 7fdf0a606640 1 -- start start 2023-08-25T09:52:38.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf0a606640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdf04159fc0 0x7fdf040c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:38.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf0a606640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406a960 con 0x7fdf0406cc70 2023-08-25T09:52:38.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf0a606640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406ac20 con 0x7fdf041594d0 2023-08-25T09:52:38.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf0a606640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0406af00 con 0x7fdf04159fc0 2023-08-25T09:52:38.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf08ba3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdf04159fc0 0x7fdf040c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:38.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf08ba3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdf04159fc0 0x7fdf040c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45260/0 (socket says 172.21.15.71:45260) 2023-08-25T09:52:38.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf08ba3640 1 -- 172.21.15.71:0/2959177411 learned_addr learned my addr 172.21.15.71:0/2959177411 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:38.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.232+0000 7fdf08ba3640 1 -- 172.21.15.71:0/2959177411 >> v1:172.21.15.164:6789/0 conn(0x7fdf041594d0 legacy=0x7fdf0414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.233+0000 7fdf08ba3640 1 -- 172.21.15.71:0/2959177411 >> v1:172.21.15.71:6789/0 conn(0x7fdf0406cc70 legacy=0x7fdf041598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.233+0000 7fdf037fe640 1 -- 172.21.15.71:0/2959177411 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197974651 0 0) 0x7fdf0406a960 con 0x7fdf0406cc70 2023-08-25T09:52:38.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.233+0000 7fdf08ba3640 1 -- 172.21.15.71:0/2959177411 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040c2e60 con 0x7fdf04159fc0 2023-08-25T09:52:38.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.234+0000 7fdf08ba3640 1 --2- 172.21.15.71:0/2959177411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf04159fc0 0x7fdf040c4c90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdefc00d150 tx=0x7fdefc02e290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b521256851c403d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:38.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.234+0000 7fdf037fe640 1 -- 172.21.15.71:0/2959177411 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00b2d0 con 0x7fdf04159fc0 2023-08-25T09:52:38.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.234+0000 7fdf037fe640 1 -- 172.21.15.71:0/2959177411 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc00b430 con 0x7fdf04159fc0 2023-08-25T09:52:38.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.234+0000 7fdf037fe640 1 -- 172.21.15.71:0/2959177411 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc009820 con 0x7fdf04159fc0 2023-08-25T09:52:38.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 -- 172.21.15.71:0/2959177411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf04159fc0 msgr2=0x7fdf040c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 --2- 172.21.15.71:0/2959177411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf04159fc0 0x7fdf040c4c90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdefc00d150 tx=0x7fdefc02e290 comp rx=0 tx=0).stop 2023-08-25T09:52:38.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 -- 172.21.15.71:0/2959177411 shutdown_connections 2023-08-25T09:52:38.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 --2- 172.21.15.71:0/2959177411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf04159fc0 0x7fdf040c4c90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 -- 172.21.15.71:0/2959177411 >> 172.21.15.71:0/2959177411 conn(0x7fdf040654f0 msgr2=0x7fdf04065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:38.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 -- 172.21.15.71:0/2959177411 shutdown_connections 2023-08-25T09:52:38.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.235+0000 7fdf0a606640 1 -- 172.21.15.71:0/2959177411 wait complete. 2023-08-25T09:52:38.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.237+0000 7fdf0a606640 1 Processor -- start 2023-08-25T09:52:38.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.237+0000 7fdf0a606640 1 -- start start 2023-08-25T09:52:38.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf0a606640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf0406cc70 0x7fdf0414e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:38.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf0a606640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 0x7fdf041f1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:38.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf0a606640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf04069d60 0x7fdf041f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:38.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf0a606640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041edcb0 con 0x7fdf041594d0 2023-08-25T09:52:38.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf0a606640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0406f420 con 0x7fdf04069d60 2023-08-25T09:52:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf0a606640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040ca0e0 con 0x7fdf0406cc70 2023-08-25T09:52:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 0x7fdf041f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 0x7fdf041f1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45262/0 (socket says 172.21.15.71:45262) 2023-08-25T09:52:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 -- 172.21.15.71:0/1678649332 learned_addr learned my addr 172.21.15.71:0/1678649332 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf03fff640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf0406cc70 0x7fdf0414e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:38.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdf08ba3640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf04069d60 0x7fdf041f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:38.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 -- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf0406cc70 msgr2=0x7fdf0414e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf0406cc70 0x7fdf0414e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 -- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf04069d60 msgr2=0x7fdf041f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf04069d60 0x7fdf041f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.238+0000 7fdefbfff640 1 -- 172.21.15.71:0/1678649332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc00bd40 con 0x7fdf041594d0 2023-08-25T09:52:38.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdf08ba3640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf04069d60 0x7fdf041f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdefbfff640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 0x7fdf041f1870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c460 tx=0x7fdeec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec005480 con 0x7fdf041594d0 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec0035f0 con 0x7fdf041594d0 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0039a0 con 0x7fdf041594d0 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0406ef50 con 0x7fdf041594d0 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.239+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf040c9840 con 0x7fdf041594d0 2023-08-25T09:52:38.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.240+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf04069160 con 0x7fdf041594d0 2023-08-25T09:52:38.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.241+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fdeec003b00 con 0x7fdf041594d0 2023-08-25T09:52:38.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.243+0000 7fdf017fa640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdee406a080 0x7fdee406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:38.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.243+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fdeec0884f0 con 0x7fdf041594d0 2023-08-25T09:52:38.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.243+0000 7fdf03fff640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdee406a080 0x7fdee406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:38.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.244+0000 7fdf03fff640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdee406a080 0x7fdee406c540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdef4000f60 tx=0x7fdef4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:38.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.247+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fdeec0537c0 con 0x7fdf041594d0 2023-08-25T09:52:38.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.474+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf04065c50 con 0x7fdf041594d0 2023-08-25T09:52:38.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.475+0000 7fdf017fa640 1 -- 172.21.15.71:0/1678649332 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fdeec003db0 con 0x7fdf041594d0 2023-08-25T09:52:38.477 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:38.478 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:36.562122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:resolve","state_seq":48,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:38.478 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 23 2023-08-25T09:52:38.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdee406a080 msgr2=0x7fdee406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdee406a080 0x7fdee406c540 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdef4000f60 tx=0x7fdef4009040 comp rx=0 tx=0).stop 2023-08-25T09:52:38.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 msgr2=0x7fdf041f1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:38.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 0x7fdf041f1870 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c460 tx=0x7fdeec00c920 comp rx=0 tx=0).stop 2023-08-25T09:52:38.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 shutdown_connections 2023-08-25T09:52:38.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf04069d60 0x7fdf041f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdee406a080 0x7fdee406c540 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf041594d0 0x7fdf041f1870 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 --2- 172.21.15.71:0/1678649332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf0406cc70 0x7fdf0414e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:38.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 >> 172.21.15.71:0/1678649332 conn(0x7fdf040654f0 msgr2=0x7fdf0406c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:38.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 shutdown_connections 2023-08-25T09:52:38.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:38.478+0000 7fdf0a606640 1 -- 172.21.15.71:0/1678649332 wait complete. 2023-08-25T09:52:38.498 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:40.499 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:40.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.717+0000 7f046ffff640 1 Processor -- start 2023-08-25T09:52:40.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.718+0000 7f046ffff640 1 -- start start 2023-08-25T09:52:40.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.719+0000 7f046ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 0x7f04701576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:40.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.719+0000 7f046ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f04700c4fb0 0x7f04700c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:40.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.719+0000 7f046ffff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0470157bf0 con 0x7f04701542d0 2023-08-25T09:52:40.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.719+0000 7f046ffff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04700c4b60 con 0x7f04700c4fb0 2023-08-25T09:52:40.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.719+0000 7f046ffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047006a130 con 0x7f0470069850 2023-08-25T09:52:40.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.720+0000 7f0467fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 0x7f04701576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:40.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.720+0000 7f046f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f04700c4fb0 0x7f04700c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:40.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.720+0000 7f0467fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 0x7f04701576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:49716/0 (socket says 172.21.15.71:49716) 2023-08-25T09:52:40.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.720+0000 7f0467fff640 1 -- 172.21.15.71:0/2360115705 learned_addr learned my addr 172.21.15.71:0/2360115705 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:40.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.720+0000 7f046effd640 1 --1- 172.21.15.71:0/2360115705 >> v1:172.21.15.164:6789/0 conn(0x7f04701542d0 0x7f04700c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:50572/0 (socket says 172.21.15.71:50572) 2023-08-25T09:52:40.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.721+0000 7f046f7fe640 1 -- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 msgr2=0x7f04701576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:40.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.721+0000 7f046f7fe640 1 --2- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 0x7f04701576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.721+0000 7f046f7fe640 1 -- 172.21.15.71:0/2360115705 >> v1:172.21.15.164:6789/0 conn(0x7f04701542d0 legacy=0x7f04700c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.721+0000 7f046f7fe640 1 -- 172.21.15.71:0/2360115705 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04701546b0 con 0x7f04700c4fb0 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.721+0000 7f0467fff640 1 --2- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 0x7f04701576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.722+0000 7f046f7fe640 1 --2- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04700c4fb0 0x7f04700c5410 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0468009d90 tx=0x7f046802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b35b7aa1a9c31d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.723+0000 7f046e7fc640 1 -- 172.21.15.71:0/2360115705 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046802cc50 con 0x7f04700c4fb0 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.723+0000 7f046e7fc640 1 -- 172.21.15.71:0/2360115705 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046802cdb0 con 0x7f04700c4fb0 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.723+0000 7f046e7fc640 1 -- 172.21.15.71:0/2360115705 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468034660 con 0x7f04700c4fb0 2023-08-25T09:52:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.723+0000 7f046ffff640 1 -- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04700c4fb0 msgr2=0x7f04700c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:40.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.723+0000 7f046ffff640 1 --2- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04700c4fb0 0x7f04700c5410 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0468009d90 tx=0x7f046802bea0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.723+0000 7f046ffff640 1 -- 172.21.15.71:0/2360115705 shutdown_connections 2023-08-25T09:52:40.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.724+0000 7f046ffff640 1 --2- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04700c4fb0 0x7f04700c5410 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.724+0000 7f046ffff640 1 --2- 172.21.15.71:0/2360115705 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0470069850 0x7f04701576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.724+0000 7f046ffff640 1 -- 172.21.15.71:0/2360115705 >> 172.21.15.71:0/2360115705 conn(0x7f047006c280 msgr2=0x7f047006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:40.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.724+0000 7f046ffff640 1 -- 172.21.15.71:0/2360115705 shutdown_connections 2023-08-25T09:52:40.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.724+0000 7f046ffff640 1 -- 172.21.15.71:0/2360115705 wait complete. 2023-08-25T09:52:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.725+0000 7f046ffff640 1 Processor -- start 2023-08-25T09:52:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 -- start start 2023-08-25T09:52:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 0x7f047014c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04700c4fb0 0x7f04700703e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04701542d0 0x7f047006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04701ec470 con 0x7f0470069850 2023-08-25T09:52:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0470071190 con 0x7f04701542d0 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.726+0000 7f046ffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04701ec9a0 con 0x7f04700c4fb0 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 0x7f047014c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 0x7f047014c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38710/0 (socket says 172.21.15.71:38710) 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 -- 172.21.15.71:0/2277169470 learned_addr learned my addr 172.21.15.71:0/2277169470 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f0467fff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04700c4fb0 0x7f04700703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046f7fe640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04701542d0 0x7f047006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 -- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04700c4fb0 msgr2=0x7f04700703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04700c4fb0 0x7f04700703e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 -- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04701542d0 msgr2=0x7f047006eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04701542d0 0x7f047006eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.727+0000 7f046effd640 1 -- 172.21.15.71:0/2277169470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580094d0 con 0x7f0470069850 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.728+0000 7f046effd640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 0x7f047014c0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f046000c3e0 tx=0x7f046000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.728+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04600074e0 con 0x7f0470069850 2023-08-25T09:52:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.728+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0460007640 con 0x7f0470069850 2023-08-25T09:52:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.728+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04600049e0 con 0x7f0470069850 2023-08-25T09:52:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.728+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0468009a70 con 0x7f0470069850 2023-08-25T09:52:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.728+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04701ec6c0 con 0x7f0470069850 2023-08-25T09:52:40.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.729+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0470071300 con 0x7f0470069850 2023-08-25T09:52:40.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.730+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0460004b40 con 0x7f0470069850 2023-08-25T09:52:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.731+0000 7f046cff9640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f045006a080 0x7f045006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.731+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f0460088160 con 0x7f0470069850 2023-08-25T09:52:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.731+0000 7f0467fff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f045006a080 0x7f045006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.732+0000 7f0467fff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f045006a080 0x7f045006c540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f04701e95e0 tx=0x7f0458008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:40.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.735+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f04600534c0 con 0x7f0470069850 2023-08-25T09:52:40.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.962+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0470157bf0 con 0x7f0470069850 2023-08-25T09:52:40.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.964+0000 7f046cff9640 1 -- 172.21.15.71:0/2277169470 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f0470157bf0 con 0x7f0470069850 2023-08-25T09:52:40.965 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:40.966 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:36.562122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:resolve","state_seq":48,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:40.966 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 23 2023-08-25T09:52:40.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f045006a080 msgr2=0x7f045006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:40.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f045006a080 0x7f045006c540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f04701e95e0 tx=0x7f0458008040 comp rx=0 tx=0).stop 2023-08-25T09:52:40.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 msgr2=0x7f047014c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:40.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 0x7f047014c0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f046000c3e0 tx=0x7f046000c8a0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 shutdown_connections 2023-08-25T09:52:40.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04701542d0 0x7f047006eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f045006a080 0x7f045006c540 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04700c4fb0 0x7f04700703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 --2- 172.21.15.71:0/2277169470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0470069850 0x7f047014c0c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:40.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 >> 172.21.15.71:0/2277169470 conn(0x7f047006c280 msgr2=0x7f04700c99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:40.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 shutdown_connections 2023-08-25T09:52:40.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:40.966+0000 7f046ffff640 1 -- 172.21.15.71:0/2277169470 wait complete. 2023-08-25T09:52:40.987 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:42.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:43.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.212+0000 7f4dad045640 1 Processor -- start 2023-08-25T09:52:43.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.212+0000 7f4dad045640 1 -- start start 2023-08-25T09:52:43.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.213+0000 7f4dad045640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4da80acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.213+0000 7f4dad045640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80c1060 con 0x7f4da80ad2f0 2023-08-25T09:52:43.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.213+0000 7f4dad045640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80af730 con 0x7f4da80c76d0 2023-08-25T09:52:43.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.213+0000 7f4dad045640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da80adf70 con 0x7f4da80c72f0 2023-08-25T09:52:43.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4dac844640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4da80ad2f0 0x7f4da8155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54564/0 (socket says 172.21.15.71:54564) 2023-08-25T09:52:43.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4dac844640 1 -- 172.21.15.71:0/3350654561 learned_addr learned my addr 172.21.15.71:0/3350654561 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:43.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da7fff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4da80acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582323986 0 0) 0x7f4da80c1060 con 0x7f4da80ad2f0 2023-08-25T09:52:43.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94003520 con 0x7f4da80ad2f0 2023-08-25T09:52:43.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3721534135 0 0) 0x7f4d94003520 con 0x7f4da80ad2f0 2023-08-25T09:52:43.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 msgr2=0x7f4da80acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:43.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4da80acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 >> v1:172.21.15.164:6789/0 conn(0x7f4da80c76d0 legacy=0x7f4da8152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da806b4a0 con 0x7f4da80ad2f0 2023-08-25T09:52:43.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4da00019f0 con 0x7f4da80ad2f0 2023-08-25T09:52:43.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 >> v1:172.21.15.71:6789/0 conn(0x7f4da80ad2f0 legacy=0x7f4da8155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4d94003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.214+0000 7f4da77fe640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94003f90 0x7f4d940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4da77fe640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d94004d00 0x7f4d94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da80adf70 con 0x7f4d94003f90 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9400dd80 con 0x7f4d94004d00 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9400df40 con 0x7f4da80c72f0 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4d9ffff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94003f90 0x7f4d940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4da7fff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4d94003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4dac844640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d94004d00 0x7f4d94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4d9ffff640 1 -- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 msgr2=0x7f4d94003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:43.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4d9ffff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4d94003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4d9ffff640 1 -- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d94004d00 msgr2=0x7f4d94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:43.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4d9ffff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d94004d00 0x7f4d94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4d9ffff640 1 -- 172.21.15.71:0/3350654561 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94003520 con 0x7f4d94003f90 2023-08-25T09:52:43.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4da7fff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4da80c72f0 0x7f4d94003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:43.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.215+0000 7f4dac844640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d94004d00 0x7f4d94005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:43.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.216+0000 7f4d9ffff640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94003f90 0x7f4d940047c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d900015b0 tx=0x7f4d940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49977c0c4a45518d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:43.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d900052d0 con 0x7f4d94003f90 2023-08-25T09:52:43.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9002e050 con 0x7f4d94003f90 2023-08-25T09:52:43.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4da77fe640 1 -- 172.21.15.71:0/3350654561 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90003830 con 0x7f4d94003f90 2023-08-25T09:52:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dad045640 1 -- 172.21.15.71:0/3350654561 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94003f90 msgr2=0x7f4d940047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dad045640 1 --2- 172.21.15.71:0/3350654561 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94003f90 0x7f4d940047c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d900015b0 tx=0x7f4d940057f0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dac844640 1 -- 172.21.15.71:0/3350654561 reap_dead start 2023-08-25T09:52:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dad045640 1 -- 172.21.15.71:0/3350654561 shutdown_connections 2023-08-25T09:52:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dad045640 1 -- 172.21.15.71:0/3350654561 >> 172.21.15.71:0/3350654561 conn(0x7f4da806c280 msgr2=0x7f4da806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:43.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dad045640 1 -- 172.21.15.71:0/3350654561 shutdown_connections 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.217+0000 7f4dad045640 1 -- 172.21.15.71:0/3350654561 wait complete. 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.219+0000 7f4dad045640 1 Processor -- start 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.219+0000 7f4dad045640 1 -- start start 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dad045640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d94003f90 0x7f4da81eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dad045640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da814f310 0x7f4da80aa760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dad045640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 0x7f4da80c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dad045640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81eef60 con 0x7f4da80aaca0 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dad045640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81e0b10 con 0x7f4da814f310 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dad045640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81e18a0 con 0x7f4d94003f90 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.220+0000 7f4dac844640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 0x7f4da80c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 0x7f4da80c4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38736/0 (socket says 172.21.15.71:38736) 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 -- 172.21.15.71:0/1802652242 learned_addr learned my addr 172.21.15.71:0/1802652242 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4d9ffff640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da814f310 0x7f4da80aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4da7fff640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d94003f90 0x7f4da81eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 -- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d94003f90 msgr2=0x7f4da81eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d94003f90 0x7f4da81eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 -- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da814f310 msgr2=0x7f4da80aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da814f310 0x7f4da80aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 -- 172.21.15.71:0/1802652242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da81e1600 con 0x7f4da80aaca0 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4da7fff640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d94003f90 0x7f4da81eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dac844640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 0x7f4da80c4f10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4da806dc70 tx=0x7f4da000b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da000b870 con 0x7f4da80aaca0 2023-08-25T09:52:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da000b9d0 con 0x7f4da80aaca0 2023-08-25T09:52:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da81e01e0 con 0x7f4da80aaca0 2023-08-25T09:52:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.221+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da000bd60 con 0x7f4da80aaca0 2023-08-25T09:52:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.222+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da81e07a0 con 0x7f4da80aaca0 2023-08-25T09:52:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.223+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da804e420 con 0x7f4da80aaca0 2023-08-25T09:52:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.224+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4da0007110 con 0x7f4da80aaca0 2023-08-25T09:52:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.224+0000 7f4da57fa640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d8806a050 0x7f4d8806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.225+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f4da0088cf0 con 0x7f4da80aaca0 2023-08-25T09:52:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.225+0000 7f4da7fff640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d8806a050 0x7f4d8806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.225+0000 7f4da7fff640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d8806a050 0x7f4d8806c510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009aa0 tx=0x7f4d98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:43.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.229+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4da0054050 con 0x7f4da80aaca0 2023-08-25T09:52:43.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.458+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4da81564c0 con 0x7f4da80aaca0 2023-08-25T09:52:43.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.459+0000 7f4da57fa640 1 -- 172.21.15.71:0/1802652242 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f4da0057670 con 0x7f4da80aaca0 2023-08-25T09:52:43.461 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:43.462 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:42.575849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:reconnect","state_seq":50,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:43.462 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 24 2023-08-25T09:52:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d8806a050 msgr2=0x7f4d8806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d8806a050 0x7f4d8806c510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009aa0 tx=0x7f4d98008040 comp rx=0 tx=0).stop 2023-08-25T09:52:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 msgr2=0x7f4da80c4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 0x7f4da80c4f10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4da806dc70 tx=0x7f4da000b0d0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 shutdown_connections 2023-08-25T09:52:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4da814f310 0x7f4da80aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d8806a050 0x7f4d8806c510 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4da80aaca0 0x7f4da80c4f10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 --2- 172.21.15.71:0/1802652242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d94003f90 0x7f4da81eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 >> 172.21.15.71:0/1802652242 conn(0x7f4da806c280 msgr2=0x7f4da806a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 shutdown_connections 2023-08-25T09:52:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:43.462+0000 7f4dad045640 1 -- 172.21.15.71:0/1802652242 wait complete. 2023-08-25T09:52:43.482 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:45.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.707+0000 7fb68be14640 1 Processor -- start 2023-08-25T09:52:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.707+0000 7fb68be14640 1 -- start start 2023-08-25T09:52:45.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb68be14640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb6841510e0 0x7fb6840c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:45.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb68be14640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6840c1dc0 con 0x7fb6840c3a40 2023-08-25T09:52:45.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb68be14640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb684156e00 con 0x7fb6840c3660 2023-08-25T09:52:45.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb68be14640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb684157a80 con 0x7fb6841510e0 2023-08-25T09:52:45.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb6893af640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb6840c3a40 0x7fb6841509d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54566/0 (socket says 172.21.15.71:54566) 2023-08-25T09:52:45.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb6893af640 1 -- 172.21.15.71:0/1617471092 learned_addr learned my addr 172.21.15.71:0/1617471092 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:45.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.709+0000 7fb68a3b1640 1 --2- 172.21.15.71:0/1617471092 >> v2:172.21.15.71:3300/0 conn(0x7fb6841510e0 0x7fb6840c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:45.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.710+0000 7fb688bae640 1 -- 172.21.15.71:0/1617471092 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361386646 0 0) 0x7fb6840c1dc0 con 0x7fb6840c3a40 2023-08-25T09:52:45.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.710+0000 7fb688bae640 1 -- 172.21.15.71:0/1617471092 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb670003520 con 0x7fb6840c3a40 2023-08-25T09:52:45.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.710+0000 7fb68a3b1640 1 -- 172.21.15.71:0/1617471092 >> v1:172.21.15.164:6789/0 conn(0x7fb6840c3660 legacy=0x7fb68406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.710+0000 7fb68a3b1640 1 -- 172.21.15.71:0/1617471092 >> v1:172.21.15.71:6789/0 conn(0x7fb6840c3a40 legacy=0x7fb6841509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.710+0000 7fb68a3b1640 1 -- 172.21.15.71:0/1617471092 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6841570e0 con 0x7fb6841510e0 2023-08-25T09:52:45.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.711+0000 7fb68a3b1640 1 --2- 172.21.15.71:0/1617471092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6840c1710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009d90 tx=0x7fb67c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e231bfded7940bf8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:45.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.711+0000 7fb688bae640 1 -- 172.21.15.71:0/1617471092 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c02cd80 con 0x7fb6841510e0 2023-08-25T09:52:45.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.711+0000 7fb688bae640 1 -- 172.21.15.71:0/1617471092 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c034440 con 0x7fb6841510e0 2023-08-25T09:52:45.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.711+0000 7fb688bae640 1 -- 172.21.15.71:0/1617471092 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0347b0 con 0x7fb6841510e0 2023-08-25T09:52:45.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 -- 172.21.15.71:0/1617471092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 msgr2=0x7fb6840c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 --2- 172.21.15.71:0/1617471092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6840c1710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009d90 tx=0x7fb67c02bfd0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 -- 172.21.15.71:0/1617471092 shutdown_connections 2023-08-25T09:52:45.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 --2- 172.21.15.71:0/1617471092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6840c1710 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 -- 172.21.15.71:0/1617471092 >> 172.21.15.71:0/1617471092 conn(0x7fb6840654f0 msgr2=0x7fb684065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:45.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 -- 172.21.15.71:0/1617471092 shutdown_connections 2023-08-25T09:52:45.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.712+0000 7fb68be14640 1 -- 172.21.15.71:0/1617471092 wait complete. 2023-08-25T09:52:45.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.714+0000 7fb68be14640 1 Processor -- start 2023-08-25T09:52:45.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.714+0000 7fb68be14640 1 -- start start 2023-08-25T09:52:45.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.715+0000 7fb68be14640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6840c3660 0x7fb684157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:45.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.715+0000 7fb68be14640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6840c3a40 0x7fb68406fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:45.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.715+0000 7fb68be14640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6841f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:45.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.715+0000 7fb68be14640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68415aa00 con 0x7fb6841510e0 2023-08-25T09:52:45.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.715+0000 7fb68be14640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6841589b0 con 0x7fb6840c3660 2023-08-25T09:52:45.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.715+0000 7fb68be14640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6841586d0 con 0x7fb6840c3a40 2023-08-25T09:52:45.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6841f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:45.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6841f1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38758/0 (socket says 172.21.15.71:38758) 2023-08-25T09:52:45.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 -- 172.21.15.71:0/2605649573 learned_addr learned my addr 172.21.15.71:0/2605649573 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:45.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb6893af640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6840c3a40 0x7fb68406fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:45.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb689bb0640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6840c3660 0x7fb684157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:45.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 -- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6840c3a40 msgr2=0x7fb68406fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6840c3a40 0x7fb68406fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 -- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6840c3660 msgr2=0x7fb684157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6840c3660 0x7fb684157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 -- 172.21.15.71:0/2605649573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c009a70 con 0x7fb6841510e0 2023-08-25T09:52:45.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.716+0000 7fb68a3b1640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6841f1ad0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009d90 tx=0x7fb67c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.717+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c003b90 con 0x7fb6841510e0 2023-08-25T09:52:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.717+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c003d10 con 0x7fb6841510e0 2023-08-25T09:52:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.717+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c004270 con 0x7fb6841510e0 2023-08-25T09:52:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.717+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68415a0d0 con 0x7fb6841510e0 2023-08-25T09:52:45.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.717+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68415a690 con 0x7fb6841510e0 2023-08-25T09:52:45.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.718+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68415a820 con 0x7fb6841510e0 2023-08-25T09:52:45.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.719+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb67c03b050 con 0x7fb6841510e0 2023-08-25T09:52:45.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.720+0000 7fb67a7fc640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb66806a050 0x7fb66806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:45.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.720+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fb67c0aaf60 con 0x7fb6841510e0 2023-08-25T09:52:45.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.720+0000 7fb689bb0640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb66806a050 0x7fb66806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:45.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.721+0000 7fb689bb0640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb66806a050 0x7fb66806c510 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb680007fd0 tx=0x7fb680009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:45.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.724+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb67c076230 con 0x7fb6841510e0 2023-08-25T09:52:45.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.951+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb684069160 con 0x7fb6841510e0 2023-08-25T09:52:45.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.952+0000 7fb67a7fc640 1 -- 172.21.15.71:0/2605649573 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fb67c009450 con 0x7fb6841510e0 2023-08-25T09:52:45.954 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:45.955 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:42.575849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:reconnect","state_seq":50,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:45.955 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 24 2023-08-25T09:52:45.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb66806a050 msgr2=0x7fb66806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb66806a050 0x7fb66806c510 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb680007fd0 tx=0x7fb680009040 comp rx=0 tx=0).stop 2023-08-25T09:52:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 msgr2=0x7fb6841f1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:45.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6841f1ad0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009d90 tx=0x7fb67c004a40 comp rx=0 tx=0).stop 2023-08-25T09:52:45.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 shutdown_connections 2023-08-25T09:52:45.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb66806a050 0x7fb66806c510 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6841510e0 0x7fb6841f1ad0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6840c3a40 0x7fb68406fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 --2- 172.21.15.71:0/2605649573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6840c3660 0x7fb684157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:45.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 >> 172.21.15.71:0/2605649573 conn(0x7fb6840654f0 msgr2=0x7fb6840c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:45.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 shutdown_connections 2023-08-25T09:52:45.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:45.954+0000 7fb68be14640 1 -- 172.21.15.71:0/2605649573 wait complete. 2023-08-25T09:52:45.976 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4235": { "addr": "172.21.15.164:6833/2402500458", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 2402500458, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 2402500458, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4235, "incarnation": 22, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:reconnect", "state_seq": 50 }, "gid_4287": { "addr": "172.21.15.71:6836/1556118065", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 1556118065, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 1556118065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4287, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_4290": { "addr": "172.21.15.71:6837/3404832976", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3404832976, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3404832976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:52:42.575849+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 4290, "mds_3": 4287, "mds_4": 4235 } } } ], "standbys": [] } 2023-08-25T09:52:47.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:48.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.198+0000 7fc5d2383640 1 Processor -- start 2023-08-25T09:52:48.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.199+0000 7fc5d2383640 1 -- start start 2023-08-25T09:52:48.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.200+0000 7fc5d2383640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:48.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.200+0000 7fc5d2383640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5cc0c3a40 0x7fc5cc06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:48.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.200+0000 7fc5d2383640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0c1dc0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.200+0000 7fc5d2383640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5cc157a80 con 0x7fc5cc0c3a40 2023-08-25T09:52:48.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.200+0000 7fc5d2383640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc06cf70 con 0x7fc5cc0c3660 2023-08-25T09:52:48.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5c3fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5cc0c3a40 0x7fc5cc06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:48.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5d0920640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc5cc1509d0 0x7fc5cc150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54576/0 (socket says 172.21.15.71:54576) 2023-08-25T09:52:48.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5d0920640 1 -- 172.21.15.71:0/3887299691 learned_addr learned my addr 172.21.15.71:0/3887299691 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:48.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5c3fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5cc0c3a40 0x7fc5cc06ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38774/0 (socket says 172.21.15.71:38774) 2023-08-25T09:52:48.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/3887299691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:48.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5c3fff640 1 -- 172.21.15.71:0/3887299691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 msgr2=0x7fc5cc0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:48.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5c3fff640 1 --2- 172.21.15.71:0/3887299691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5c3fff640 1 -- 172.21.15.71:0/3887299691 >> v1:172.21.15.71:6789/0 conn(0x7fc5cc1509d0 legacy=0x7fc5cc150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:48.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.201+0000 7fc5c3fff640 1 -- 172.21.15.71:0/3887299691 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc156e00 con 0x7fc5cc0c3a40 2023-08-25T09:52:48.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.202+0000 7fc5c3fff640 1 --2- 172.21.15.71:0/3887299691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc06ca30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40097a0 tx=0x7fc5b402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a8e2cd66e3cd64 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:48.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5cb7fe640 1 -- 172.21.15.71:0/3887299691 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4030800 con 0x7fc5cc0c3a40 2023-08-25T09:52:48.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5cb7fe640 1 -- 172.21.15.71:0/3887299691 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b4030960 con 0x7fc5cc0c3a40 2023-08-25T09:52:48.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5cb7fe640 1 -- 172.21.15.71:0/3887299691 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4030cf0 con 0x7fc5cc0c3a40 2023-08-25T09:52:48.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 -- 172.21.15.71:0/3887299691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc0c3a40 msgr2=0x7fc5cc06ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:48.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 --2- 172.21.15.71:0/3887299691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc06ca30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40097a0 tx=0x7fc5b402b940 comp rx=0 tx=0).stop 2023-08-25T09:52:48.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 -- 172.21.15.71:0/3887299691 shutdown_connections 2023-08-25T09:52:48.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 --2- 172.21.15.71:0/3887299691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc06ca30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 --2- 172.21.15.71:0/3887299691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 -- 172.21.15.71:0/3887299691 >> 172.21.15.71:0/3887299691 conn(0x7fc5cc0654f0 msgr2=0x7fc5cc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:48.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 -- 172.21.15.71:0/3887299691 shutdown_connections 2023-08-25T09:52:48.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.203+0000 7fc5d2383640 1 -- 172.21.15.71:0/3887299691 wait complete. 2023-08-25T09:52:48.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.205+0000 7fc5d2383640 1 Processor -- start 2023-08-25T09:52:48.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.205+0000 7fc5d2383640 1 -- start start 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.206+0000 7fc5d2383640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc06f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.206+0000 7fc5d2383640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc14f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.206+0000 7fc5d2383640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 0x7fc5cc14f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.206+0000 7fc5d2383640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc06ac30 con 0x7fc5cc1509d0 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.206+0000 7fc5d2383640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc06ae90 con 0x7fc5cc0c3a40 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.206+0000 7fc5d2383640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc149830 con 0x7fc5cc0c3660 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 0x7fc5cc14f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:48.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 0x7fc5cc14f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38780/0 (socket says 172.21.15.71:38780) 2023-08-25T09:52:48.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 -- 172.21.15.71:0/948898997 learned_addr learned my addr 172.21.15.71:0/948898997 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:48.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:48.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5c3fff640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc14f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:48.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 -- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 msgr2=0x7fc5cc06f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:48.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc06f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 -- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5cc0c3a40 msgr2=0x7fc5cc14f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:48.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc14f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 -- 172.21.15.71:0/948898997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b4009400 con 0x7fc5cc1509d0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.207+0000 7fc5d0920640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 0x7fc5cc14f940 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400cbc0 tx=0x7fc5c400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.208+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400ad40 con 0x7fc5cc1509d0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.208+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4012400 con 0x7fc5cc1509d0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.208+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400aea0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.208+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc1ef0f0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.208+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc06e7a0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.209+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc0c29c0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.210+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc5c4007070 con 0x7fc5cc1509d0 2023-08-25T09:52:48.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.211+0000 7fc5c97fa640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ac06a050 0x7fc5ac06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:48.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.211+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc5c4088820 con 0x7fc5cc1509d0 2023-08-25T09:52:48.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.211+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ac06a050 0x7fc5ac06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:48.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.212+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ac06a050 0x7fc5ac06c510 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006fd0 tx=0x7fc5bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:48.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.215+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc5c4053ba0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.440+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5cc069160 con 0x7fc5cc1509d0 2023-08-25T09:52:48.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.441+0000 7fc5c97fa640 1 -- 172.21.15.71:0/948898997 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5739 (secure 0 0 0) 0x7fc5c40571c0 con 0x7fc5cc1509d0 2023-08-25T09:52:48.444 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:48.445 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:47.784663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:48.446 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 25 2023-08-25T09:52:48.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ac06a050 msgr2=0x7fc5ac06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:48.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ac06a050 0x7fc5ac06c510 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006fd0 tx=0x7fc5bc008040 comp rx=0 tx=0).stop 2023-08-25T09:52:48.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 msgr2=0x7fc5cc14f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:48.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 0x7fc5cc14f940 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400cbc0 tx=0x7fc5c400a430 comp rx=0 tx=0).stop 2023-08-25T09:52:48.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 shutdown_connections 2023-08-25T09:52:48.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5cc0c3a40 0x7fc5cc14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5ac06a050 0x7fc5ac06c510 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5cc1509d0 0x7fc5cc14f940 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 --2- 172.21.15.71:0/948898997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5cc0c3660 0x7fc5cc06f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:48.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 >> 172.21.15.71:0/948898997 conn(0x7fc5cc0654f0 msgr2=0x7fc5cc0658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:48.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 shutdown_connections 2023-08-25T09:52:48.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:48.444+0000 7fc5d2383640 1 -- 172.21.15.71:0/948898997 wait complete. 2023-08-25T09:52:48.466 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:50.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:50.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.693+0000 7f3fced12640 1 Processor -- start 2023-08-25T09:52:50.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.693+0000 7f3fced12640 1 -- start start 2023-08-25T09:52:50.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.695+0000 7f3fced12640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cc6b0 0x7f3fc8156f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:50.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.695+0000 7f3fced12640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3fc80cca90 0x7f3fc8070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:50.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.695+0000 7f3fced12640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80c58a0 con 0x7f3fc80c6060 2023-08-25T09:52:50.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.695+0000 7f3fced12640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc8071350 con 0x7f3fc80cca90 2023-08-25T09:52:50.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.695+0000 7f3fced12640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8157540 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.695+0000 7f3fbffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3fc80cca90 0x7f3fc8070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:50.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fbffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3fc80cca90 0x7f3fc8070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41430/0 (socket says 172.21.15.71:41430) 2023-08-25T09:52:50.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fbffff640 1 -- 172.21.15.71:0/1192690449 learned_addr learned my addr 172.21.15.71:0/1192690449 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:50.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fccaae640 1 --2- 172.21.15.71:0/1192690449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cc6b0 0x7f3fc8156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:50.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fbffff640 1 -- 172.21.15.71:0/1192690449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cc6b0 msgr2=0x7f3fc8156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:50.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fbffff640 1 --2- 172.21.15.71:0/1192690449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cc6b0 0x7f3fc8156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fbffff640 1 -- 172.21.15.71:0/1192690449 >> v1:172.21.15.164:6789/0 conn(0x7f3fc80c6060 legacy=0x7f3fc8150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.696+0000 7f3fbffff640 1 -- 172.21.15.71:0/1192690449 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8157820 con 0x7f3fc80cca90 2023-08-25T09:52:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.697+0000 7f3fbffff640 1 --2- 172.21.15.71:0/1192690449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cca90 0x7f3fc8070e10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009870 tx=0x7f3fb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c763756687d6c024 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fbf7fe640 1 -- 172.21.15.71:0/1192690449 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8030840 con 0x7f3fc80cca90 2023-08-25T09:52:50.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fbf7fe640 1 -- 172.21.15.71:0/1192690449 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb80309a0 con 0x7f3fc80cca90 2023-08-25T09:52:50.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fbf7fe640 1 -- 172.21.15.71:0/1192690449 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8030d30 con 0x7f3fc80cca90 2023-08-25T09:52:50.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fced12640 1 -- 172.21.15.71:0/1192690449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cca90 msgr2=0x7f3fc8070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:50.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fced12640 1 --2- 172.21.15.71:0/1192690449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cca90 0x7f3fc8070e10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009870 tx=0x7f3fb802b980 comp rx=0 tx=0).stop 2023-08-25T09:52:50.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fced12640 1 -- 172.21.15.71:0/1192690449 shutdown_connections 2023-08-25T09:52:50.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fced12640 1 --2- 172.21.15.71:0/1192690449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cca90 0x7f3fc8070e10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.698+0000 7f3fced12640 1 --2- 172.21.15.71:0/1192690449 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cc6b0 0x7f3fc8156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.699+0000 7f3fced12640 1 -- 172.21.15.71:0/1192690449 >> 172.21.15.71:0/1192690449 conn(0x7f3fc8069e80 msgr2=0x7f3fc80654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:50.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.699+0000 7f3fced12640 1 -- 172.21.15.71:0/1192690449 shutdown_connections 2023-08-25T09:52:50.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.699+0000 7f3fced12640 1 -- 172.21.15.71:0/1192690449 wait complete. 2023-08-25T09:52:50.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.700+0000 7f3fced12640 1 Processor -- start 2023-08-25T09:52:50.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.701+0000 7f3fced12640 1 -- start start 2023-08-25T09:52:50.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.701+0000 7f3fced12640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3fc80c6060 0x7f3fc81ebf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:50.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.701+0000 7f3fced12640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 0x7f3fc81494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:50.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fced12640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cca90 0x7f3fc814dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:50.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fced12640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8155e60 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fced12640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc80c9c20 con 0x7f3fc80c6060 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fced12640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc81ec550 con 0x7f3fc80cca90 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 0x7f3fc81494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 0x7f3fc81494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41442/0 (socket says 172.21.15.71:41442) 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 -- 172.21.15.71:0/3187667186 learned_addr learned my addr 172.21.15.71:0/3187667186 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 -- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cca90 msgr2=0x7f3fc814dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fcd2af640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cca90 0x7f3fc814dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:50.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fccaae640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3fc80c6060 0x7f3fc81ebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:50.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cca90 0x7f3fc814dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 -- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3fc80c6060 msgr2=0x7f3fc81ebf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:50.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3fc80c6060 0x7f3fc81ebf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.702+0000 7f3fbffff640 1 -- 172.21.15.71:0/3187667186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb80094d0 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.703+0000 7f3fbffff640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 0x7f3fc81494f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80090b0 tx=0x7f3fb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:50.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.703+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb803e070 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.703+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb8039800 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.703+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8039bb0 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.703+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc81ec830 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.703+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc8156100 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.704+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc8069160 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.705+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3fb8042070 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.706+0000 7f3fbd7fa640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3fb406a050 0x7f3fb406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:50.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.706+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3fb80aa6c0 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.706+0000 7f3fccaae640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3fb406a050 0x7f3fb406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:50.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.707+0000 7f3fccaae640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3fb406a050 0x7f3fb406c510 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009f30 tx=0x7f3fc400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:50.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.711+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3fb8075990 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.941+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc80c7800 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.942+0000 7f3fbd7fa640 1 -- 172.21.15.71:0/3187667186 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f3fb8078fb0 con 0x7f3fc80cc6b0 2023-08-25T09:52:50.944 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:50.945 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:47.784663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:50.945 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 25 2023-08-25T09:52:50.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3fb406a050 msgr2=0x7f3fb406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:50.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3fb406a050 0x7f3fb406c510 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4009f30 tx=0x7f3fc400b040 comp rx=0 tx=0).stop 2023-08-25T09:52:50.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 msgr2=0x7f3fc81494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:50.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 0x7f3fc81494f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80090b0 tx=0x7f3fb80046c0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 shutdown_connections 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3fb406a050 0x7f3fb406c510 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fc80cca90 0x7f3fc814dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3fc80cc6b0 0x7f3fc81494f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 --2- 172.21.15.71:0/3187667186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3fc80c6060 0x7f3fc81ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 >> 172.21.15.71:0/3187667186 conn(0x7f3fc8069e80 msgr2=0x7f3fc80654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 shutdown_connections 2023-08-25T09:52:50.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:50.945+0000 7f3fced12640 1 -- 172.21.15.71:0/3187667186 wait complete. 2023-08-25T09:52:50.966 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:52.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:52:52.652 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:52:52.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:53.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.193+0000 7f25a9406640 1 Processor -- start 2023-08-25T09:52:53.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.193+0000 7f25a9406640 1 -- start start 2023-08-25T09:52:53.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.194+0000 7f25a9406640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f25a41594d0 0x7f25a40c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:53.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.194+0000 7f25a9406640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f25a40c3940 0x7f25a41598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:53.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.194+0000 7f25a9406640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a406a960 con 0x7f25a406cc70 2023-08-25T09:52:53.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.194+0000 7f25a9406640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a406ac20 con 0x7f25a41594d0 2023-08-25T09:52:53.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.194+0000 7f25a9406640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a406af00 con 0x7f25a40c3940 2023-08-25T09:52:53.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a2ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f25a41594d0 0x7f25a40c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:53.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f259bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f25a406cc70 0x7f25a414f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37088/0 (socket says 172.21.15.71:37088) 2023-08-25T09:52:53.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f259bfff640 1 -- 172.21.15.71:0/1255030336 learned_addr learned my addr 172.21.15.71:0/1255030336 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:53.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a37fe640 1 --2- 172.21.15.71:0/1255030336 >> v2:172.21.15.164:3300/0 conn(0x7f25a40c3940 0x7f25a41598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:53.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a2ffd640 1 -- 172.21.15.71:0/1255030336 >> v2:172.21.15.164:3300/0 conn(0x7f25a40c3940 msgr2=0x7f25a41598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:53.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a2ffd640 1 --2- 172.21.15.71:0/1255030336 >> v2:172.21.15.164:3300/0 conn(0x7f25a40c3940 0x7f25a41598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a2ffd640 1 -- 172.21.15.71:0/1255030336 >> v1:172.21.15.71:6789/0 conn(0x7f25a406cc70 legacy=0x7f25a414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:53.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a2ffd640 1 -- 172.21.15.71:0/1255030336 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a40c2e60 con 0x7f25a41594d0 2023-08-25T09:52:53.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.195+0000 7f25a27fc640 1 -- 172.21.15.71:0/1255030336 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885281975 0 0) 0x7f25a406a960 con 0x7f25a406cc70 2023-08-25T09:52:53.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.196+0000 7f25a2ffd640 1 --2- 172.21.15.71:0/1255030336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a41594d0 0x7f25a40c4c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f25a406db90 tx=0x7f259402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe4c25887968ac1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:53.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a27fc640 1 -- 172.21.15.71:0/1255030336 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594031830 con 0x7f25a41594d0 2023-08-25T09:52:53.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a27fc640 1 -- 172.21.15.71:0/1255030336 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594031990 con 0x7f25a41594d0 2023-08-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a27fc640 1 -- 172.21.15.71:0/1255030336 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594031d20 con 0x7f25a41594d0 2023-08-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 -- 172.21.15.71:0/1255030336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a41594d0 msgr2=0x7f25a40c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 --2- 172.21.15.71:0/1255030336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a41594d0 0x7f25a40c4c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f25a406db90 tx=0x7f259402b8d0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 -- 172.21.15.71:0/1255030336 shutdown_connections 2023-08-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 --2- 172.21.15.71:0/1255030336 >> v2:172.21.15.164:3300/0 conn(0x7f25a40c3940 0x7f25a41598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 --2- 172.21.15.71:0/1255030336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a41594d0 0x7f25a40c4c90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 -- 172.21.15.71:0/1255030336 >> 172.21.15.71:0/1255030336 conn(0x7f25a40654f0 msgr2=0x7f25a4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:53.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 -- 172.21.15.71:0/1255030336 shutdown_connections 2023-08-25T09:52:53.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.197+0000 7f25a9406640 1 -- 172.21.15.71:0/1255030336 wait complete. 2023-08-25T09:52:53.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.199+0000 7f25a9406640 1 Processor -- start 2023-08-25T09:52:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.199+0000 7f25a9406640 1 -- start start 2023-08-25T09:52:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f25a9406640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25a406cc70 0x7f25a4140b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f25a9406640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 0x7f25a41463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f25a9406640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a41594d0 0x7f25a41ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f25a9406640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4142be0 con 0x7f25a40c3940 2023-08-25T09:52:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f25a9406640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4141100 con 0x7f25a41594d0 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f25a9406640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41455c0 con 0x7f25a406cc70 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f259bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 0x7f25a41463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f259bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 0x7f25a41463b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41460/0 (socket says 172.21.15.71:41460) 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.200+0000 7f259bfff640 1 -- 172.21.15.71:0/1807682141 learned_addr learned my addr 172.21.15.71:0/1807682141 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f25a2ffd640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25a406cc70 0x7f25a4140b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f25a37fe640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a41594d0 0x7f25a41ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:53.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259bfff640 1 -- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25a406cc70 msgr2=0x7f25a4140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259bfff640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25a406cc70 0x7f25a4140b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259bfff640 1 -- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a41594d0 msgr2=0x7f25a41ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259bfff640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a41594d0 0x7f25a41ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259bfff640 1 -- 172.21.15.71:0/1807682141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594009400 con 0x7f25a40c3940 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f25a2ffd640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25a406cc70 0x7f25a4140b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259bfff640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 0x7f25a41463b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f258c00c430 tx=0x7f258c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c0074e0 con 0x7f25a40c3940 2023-08-25T09:52:53.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c007660 con 0x7f25a40c3940 2023-08-25T09:52:53.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c004a20 con 0x7f25a40c3940 2023-08-25T09:52:53.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.201+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a4145880 con 0x7f25a40c3940 2023-08-25T09:52:53.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.202+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a4142870 con 0x7f25a40c3940 2023-08-25T09:52:53.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.203+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4142a30 con 0x7f25a40c3940 2023-08-25T09:52:53.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.204+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f258c004b80 con 0x7f25a40c3940 2023-08-25T09:52:53.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.205+0000 7f259b7fe640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f258406a050 0x7f258406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:53.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.205+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f258c0887c0 con 0x7f25a40c3940 2023-08-25T09:52:53.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.205+0000 7f25a2ffd640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f258406a050 0x7f258406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:53.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.206+0000 7f25a2ffd640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f258406a050 0x7f258406c510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2594036000 tx=0x7f2594008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:53.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.209+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f258c053aa0 con 0x7f25a40c3940 2023-08-25T09:52:53.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.435+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a406a780 con 0x7f25a40c3940 2023-08-25T09:52:53.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.436+0000 7f259b7fe640 1 -- 172.21.15.71:0/1807682141 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f258c0570c0 con 0x7f25a40c3940 2023-08-25T09:52:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:53.439 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:47.784663+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:rejoin","state_seq":52,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:53.439 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 25 2023-08-25T09:52:53.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f258406a050 msgr2=0x7f258406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f258406a050 0x7f258406c510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2594036000 tx=0x7f2594008030 comp rx=0 tx=0).stop 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 msgr2=0x7f25a41463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 0x7f25a41463b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f258c00c430 tx=0x7f258c00c8f0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 shutdown_connections 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a41594d0 0x7f25a41ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f258406a050 0x7f258406c510 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f25a40c3940 0x7f25a41463b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 --2- 172.21.15.71:0/1807682141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f25a406cc70 0x7f25a4140b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:53.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 >> 172.21.15.71:0/1807682141 conn(0x7f25a40654f0 msgr2=0x7f25a40c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:53.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 shutdown_connections 2023-08-25T09:52:53.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:53.439+0000 7f25a9406640 1 -- 172.21.15.71:0/1807682141 wait complete. 2023-08-25T09:52:53.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:52:55.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:52:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.678+0000 7f9a5bfff640 1 Processor -- start 2023-08-25T09:52:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.678+0000 7f9a5bfff640 1 -- start start 2023-08-25T09:52:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.680+0000 7f9a5bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a5c0cad40 0x7f9a5c0cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.680+0000 7f9a5bfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c15c6c0 con 0x7f9a5c158d60 2023-08-25T09:52:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.680+0000 7f9a5bfff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c15d260 con 0x7f9a5c152fc0 2023-08-25T09:52:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.680+0000 7f9a5bfff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c1592c0 con 0x7f9a5c0cad40 2023-08-25T09:52:55.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.680+0000 7f9a53fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9a5c158d60 0x7f9a5c0c97a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37100/0 (socket says 172.21.15.71:37100) 2023-08-25T09:52:55.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.680+0000 7f9a53fff640 1 -- 172.21.15.71:0/303398198 learned_addr learned my addr 172.21.15.71:0/303398198 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:52:55.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5b7fe640 1 --2- 172.21.15.71:0/303398198 >> v2:172.21.15.164:3300/0 conn(0x7f9a5c0cad40 0x7f9a5c0cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732010142 0 0) 0x7f9a5c15c6c0 con 0x7f9a5c158d60 2023-08-25T09:52:55.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a48003520 con 0x7f9a5c158d60 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028745836 0 0) 0x7f9a48003520 con 0x7f9a5c158d60 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 >> v2:172.21.15.164:3300/0 conn(0x7f9a5c0cad40 msgr2=0x7f9a5c0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 --2- 172.21.15.71:0/303398198 >> v2:172.21.15.164:3300/0 conn(0x7f9a5c0cad40 0x7f9a5c0cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 >> v1:172.21.15.164:6789/0 conn(0x7f9a5c152fc0 legacy=0x7f9a5c15cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c1595a0 con 0x7f9a5c158d60 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f9a44001670 con 0x7f9a5c158d60 2023-08-25T09:52:55.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 >> v1:172.21.15.71:6789/0 conn(0x7f9a5c158d60 legacy=0x7f9a5c0c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a48003a10 0x7f9a48003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a48005180 0x7f9a4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1592c0 con 0x7f9a480047c0 2023-08-25T09:52:55.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800c100 con 0x7f9a48003a10 2023-08-25T09:52:55.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.681+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4800c2c0 con 0x7f9a48005180 2023-08-25T09:52:55.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a53fff640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a5affd640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a48003a10 0x7f9a48003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a5b7fe640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a48005180 0x7f9a4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a53fff640 1 -- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a48005180 msgr2=0x7f9a4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:55.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a53fff640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a48005180 0x7f9a4800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a53fff640 1 -- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a48003a10 msgr2=0x7f9a48003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:52:55.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a53fff640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a48003a10 0x7f9a48003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a53fff640 1 -- 172.21.15.71:0/303398198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48003520 con 0x7f9a480047c0 2023-08-25T09:52:55.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.682+0000 7f9a5b7fe640 1 -- 172.21.15.71:0/303398198 reap_dead start 2023-08-25T09:52:55.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.683+0000 7f9a53fff640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400c190 tx=0x7f9a480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64abbc7dfb1aa5ee server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:55.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a44005ad0 con 0x7f9a480047c0 2023-08-25T09:52:55.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a44005c30 con 0x7f9a480047c0 2023-08-25T09:52:55.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5a7fc640 1 -- 172.21.15.71:0/303398198 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4402ca00 con 0x7f9a480047c0 2023-08-25T09:52:55.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5bfff640 1 -- 172.21.15.71:0/303398198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 msgr2=0x7f9a48004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a4400c190 tx=0x7f9a480057f0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5bfff640 1 -- 172.21.15.71:0/303398198 shutdown_connections 2023-08-25T09:52:55.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/303398198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a48004c40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.684+0000 7f9a5bfff640 1 -- 172.21.15.71:0/303398198 >> 172.21.15.71:0/303398198 conn(0x7f9a5c06a3a0 msgr2=0x7f9a5c070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:55.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.685+0000 7f9a5bfff640 1 -- 172.21.15.71:0/303398198 shutdown_connections 2023-08-25T09:52:55.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.685+0000 7f9a5bfff640 1 -- 172.21.15.71:0/303398198 wait complete. 2023-08-25T09:52:55.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.686+0000 7f9a5bfff640 1 Processor -- start 2023-08-25T09:52:55.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.686+0000 7f9a5bfff640 1 -- start start 2023-08-25T09:52:55.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.687+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a5c0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.687+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a5c1564e0 0x7f9a5c0c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.687+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a5c1e81d0 0x7f9a5c1e8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.687+0000 7f9a5bfff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c147d30 con 0x7f9a480047c0 2023-08-25T09:52:55.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.687+0000 7f9a5bfff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1536d0 con 0x7f9a5c1e81d0 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.687+0000 7f9a5bfff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1544f0 con 0x7f9a5c1564e0 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a5c0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a5c0c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41464/0 (socket says 172.21.15.71:41464) 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 -- 172.21.15.71:0/1759009566 learned_addr learned my addr 172.21.15.71:0/1759009566 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5b7fe640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a5c1e81d0 0x7f9a5c1e8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a53fff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a5c1564e0 0x7f9a5c0c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 -- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a5c1564e0 msgr2=0x7f9a5c0c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a5c1564e0 0x7f9a5c0c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 -- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a5c1e81d0 msgr2=0x7f9a5c1e8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a5c1e81d0 0x7f9a5c1e8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5affd640 1 -- 172.21.15.71:0/1759009566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c1538e0 con 0x7f9a480047c0 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a5b7fe640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a5c1e81d0 0x7f9a5c1e8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.688+0000 7f9a53fff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a5c1564e0 0x7f9a5c0c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:52:55.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.689+0000 7f9a5affd640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a5c0c7c20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00e410 tx=0x7f9a4c00e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:55.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.689+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0052e0 con 0x7f9a480047c0 2023-08-25T09:52:55.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.689+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c005440 con 0x7f9a480047c0 2023-08-25T09:52:55.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.689+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c003810 con 0x7f9a480047c0 2023-08-25T09:52:55.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.689+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c153bc0 con 0x7f9a480047c0 2023-08-25T09:52:55.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.689+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c154180 con 0x7f9a480047c0 2023-08-25T09:52:55.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.690+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c154340 con 0x7f9a480047c0 2023-08-25T09:52:55.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.691+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9a4c003970 con 0x7f9a480047c0 2023-08-25T09:52:55.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.692+0000 7f9a537fe640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a3c06a050 0x7f9a3c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:52:55.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.692+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f9a4c0881c0 con 0x7f9a480047c0 2023-08-25T09:52:55.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.693+0000 7f9a53fff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a3c06a050 0x7f9a3c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:52:55.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.693+0000 7f9a53fff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a3c06a050 0x7f9a3c06c510 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a44034000 tx=0x7f9a44009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:52:55.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.697+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9a4c0534b0 con 0x7f9a480047c0 2023-08-25T09:52:55.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.924+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a5c04e420 con 0x7f9a480047c0 2023-08-25T09:52:55.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.925+0000 7f9a537fe640 1 -- 172.21.15.71:0/1759009566 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f9a4c019090 con 0x7f9a480047c0 2023-08-25T09:52:55.927 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:52:55.928 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:53.771532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:52:55.928 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 26 2023-08-25T09:52:55.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a3c06a050 msgr2=0x7f9a3c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a3c06a050 0x7f9a3c06c510 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a44034000 tx=0x7f9a44009030 comp rx=0 tx=0).stop 2023-08-25T09:52:55.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 msgr2=0x7f9a5c0c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:52:55.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a5c0c7c20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00e410 tx=0x7f9a4c00e8d0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 shutdown_connections 2023-08-25T09:52:55.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a5c1e81d0 0x7f9a5c1e8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a3c06a050 0x7f9a3c06c510 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a5c1564e0 0x7f9a5c0c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 --2- 172.21.15.71:0/1759009566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a480047c0 0x7f9a5c0c7c20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:52:55.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 >> 172.21.15.71:0/1759009566 conn(0x7f9a5c06a3a0 msgr2=0x7f9a5c0c97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:52:55.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 shutdown_connections 2023-08-25T09:52:55.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:52:55.928+0000 7f9a5bfff640 1 -- 172.21.15.71:0/1759009566 wait complete. 2023-08-25T09:52:55.948 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=4, replacing gid=4238 2023-08-25T09:52:55.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.d 2023-08-25T09:52:59.504 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/30 file after the snapshot 2023-08-25T09:52:59.517 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/31 file after the snapshot 2023-08-25T09:52:59.526 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/32 file after the snapshot 2023-08-25T09:52:59.536 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/33 file after the snapshot 2023-08-25T09:52:59.547 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/34 file after the snapshot 2023-08-25T09:52:59.557 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/35 file after the snapshot 2023-08-25T09:52:59.567 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/36 file after the snapshot 2023-08-25T09:52:59.578 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/37 file after the snapshot 2023-08-25T09:52:59.588 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/38 file after the snapshot 2023-08-25T09:52:59.599 INFO:tasks.workunit.client.0.smithi071.stdout: create 138/39 file after the snapshot 2023-08-25T09:52:59.608 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/.snap/snap-subdir-test 2023-08-25T09:52:59.643 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/30 file after the snapshot 2023-08-25T09:52:59.648 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/31 file after the snapshot 2023-08-25T09:53:00.271 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/32 file after the snapshot 2023-08-25T09:53:00.279 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/33 file after the snapshot 2023-08-25T09:53:00.285 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/34 file after the snapshot 2023-08-25T09:53:00.292 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/35 file after the snapshot 2023-08-25T09:53:00.298 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/36 file after the snapshot 2023-08-25T09:53:00.304 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/37 file after the snapshot 2023-08-25T09:53:00.312 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/38 file after the snapshot 2023-08-25T09:53:00.319 INFO:tasks.workunit.client.0.smithi071.stdout: create 139/39 file after the snapshot 2023-08-25T09:53:00.326 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/.snap/snap-subdir-test 2023-08-25T09:53:00.366 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/30 file after the snapshot 2023-08-25T09:53:00.372 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/31 file after the snapshot 2023-08-25T09:53:00.377 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/32 file after the snapshot 2023-08-25T09:53:00.385 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/33 file after the snapshot 2023-08-25T09:53:00.392 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/34 file after the snapshot 2023-08-25T09:53:00.397 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/35 file after the snapshot 2023-08-25T09:53:00.405 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/36 file after the snapshot 2023-08-25T09:53:00.411 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/37 file after the snapshot 2023-08-25T09:53:00.416 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/38 file after the snapshot 2023-08-25T09:53:00.424 INFO:tasks.workunit.client.0.smithi071.stdout: create 140/39 file after the snapshot 2023-08-25T09:53:00.431 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/.snap/snap-subdir-test 2023-08-25T09:53:00.474 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/30 file after the snapshot 2023-08-25T09:53:00.479 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/31 file after the snapshot 2023-08-25T09:53:00.486 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/32 file after the snapshot 2023-08-25T09:53:00.492 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/33 file after the snapshot 2023-08-25T09:53:00.498 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/34 file after the snapshot 2023-08-25T09:53:00.505 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/35 file after the snapshot 2023-08-25T09:53:00.512 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/36 file after the snapshot 2023-08-25T09:53:00.524 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/37 file after the snapshot 2023-08-25T09:53:00.530 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/38 file after the snapshot 2023-08-25T09:53:00.537 INFO:tasks.workunit.client.0.smithi071.stdout: create 141/39 file after the snapshot 2023-08-25T09:53:00.544 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/.snap/snap-subdir-test 2023-08-25T09:53:00.949 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-08-25T09:53:00.950 INFO:tasks.ceph.mds.d:Restarting daemon 2023-08-25T09:53:00.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-08-25T09:53:00.960 INFO:tasks.ceph.mds.d:Started 2023-08-25T09:53:00.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:53:01.151 INFO:tasks.ceph.mds.d.smithi164.stdout:starting mds.d at 2023-08-25T09:53:01.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.188+0000 7f15df59e640 1 Processor -- start 2023-08-25T09:53:01.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.189+0000 7f15df59e640 1 -- start start 2023-08-25T09:53:01.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.190+0000 7f15df59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f15e00c72f0 0x7f15e00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:01.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.190+0000 7f15df59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e0152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:01.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.190+0000 7f15df59e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e00c1060 con 0x7f15e00ad2f0 2023-08-25T09:53:01.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.190+0000 7f15df59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e00adf70 con 0x7f15e00c72f0 2023-08-25T09:53:01.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.190+0000 7f15df59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00af730 con 0x7f15e00c76d0 2023-08-25T09:53:01.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e0152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:01.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15de59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f15e00c72f0 0x7f15e00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:01.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ded9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f15e00ad2f0 0x7f15e0155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:45706/0 (socket says 172.21.15.71:45706) 2023-08-25T09:53:01.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e0152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:60168/0 (socket says 172.21.15.71:60168) 2023-08-25T09:53:01.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ded9d640 1 -- 172.21.15.71:0/1849756794 learned_addr learned my addr 172.21.15.71:0/1849756794 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:53:01.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ddd9b640 1 -- 172.21.15.71:0/1849756794 >> v2:172.21.15.164:3300/0 conn(0x7f15e00c72f0 msgr2=0x7f15e00acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:53:01.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15dd59a640 1 -- 172.21.15.71:0/1849756794 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503340984 0 0) 0x7f15e00c1060 con 0x7f15e00ad2f0 2023-08-25T09:53:01.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ddd9b640 1 --2- 172.21.15.71:0/1849756794 >> v2:172.21.15.164:3300/0 conn(0x7f15e00c72f0 0x7f15e00acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ddd9b640 1 -- 172.21.15.71:0/1849756794 >> v1:172.21.15.164:6789/0 conn(0x7f15e00ad2f0 legacy=0x7f15e0155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:01.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.191+0000 7f15ddd9b640 1 -- 172.21.15.71:0/1849756794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e006b4a0 con 0x7f15e00c76d0 2023-08-25T09:53:01.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.193+0000 7f15ddd9b640 1 --2- 172.21.15.71:0/1849756794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e0152da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f15c80097a0 tx=0x7f15c802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f52a186ee8afaab3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:53:01.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.193+0000 7f15dd59a640 1 -- 172.21.15.71:0/1849756794 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c8030800 con 0x7f15e00c76d0 2023-08-25T09:53:01.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.193+0000 7f15dd59a640 1 -- 172.21.15.71:0/1849756794 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c8030960 con 0x7f15e00c76d0 2023-08-25T09:53:01.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.193+0000 7f15dd59a640 1 -- 172.21.15.71:0/1849756794 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c8030cf0 con 0x7f15e00c76d0 2023-08-25T09:53:01.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 -- 172.21.15.71:0/1849756794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 msgr2=0x7f15e0152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:01.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 --2- 172.21.15.71:0/1849756794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e0152da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f15c80097a0 tx=0x7f15c802b940 comp rx=0 tx=0).stop 2023-08-25T09:53:01.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 -- 172.21.15.71:0/1849756794 shutdown_connections 2023-08-25T09:53:01.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 --2- 172.21.15.71:0/1849756794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e0152da0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 --2- 172.21.15.71:0/1849756794 >> v2:172.21.15.164:3300/0 conn(0x7f15e00c72f0 0x7f15e00acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 -- 172.21.15.71:0/1849756794 >> 172.21.15.71:0/1849756794 conn(0x7f15e006c280 msgr2=0x7f15e006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:53:01.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 -- 172.21.15.71:0/1849756794 shutdown_connections 2023-08-25T09:53:01.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.194+0000 7f15df59e640 1 -- 172.21.15.71:0/1849756794 wait complete. 2023-08-25T09:53:01.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.195+0000 7f15df59e640 1 Processor -- start 2023-08-25T09:53:01.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.196+0000 7f15df59e640 1 -- start start 2023-08-25T09:53:01.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.196+0000 7f15df59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e00ad2f0 0x7f15e00a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:01.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.196+0000 7f15df59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 0x7f15e00aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:01.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.196+0000 7f15df59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e00ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:01.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.196+0000 7f15df59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e006b1f0 con 0x7f15e00c72f0 2023-08-25T09:53:01.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.196+0000 7f15df59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00c9f20 con 0x7f15e00ad2f0 2023-08-25T09:53:01.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15df59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00c8c20 con 0x7f15e00c76d0 2023-08-25T09:53:01.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 0x7f15e00aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:01.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 0x7f15e00aaba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53596/0 (socket says 172.21.15.71:53596) 2023-08-25T09:53:01.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 -- 172.21.15.71:0/350751475 learned_addr learned my addr 172.21.15.71:0/350751475 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:53:01.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ded9d640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e00ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:01.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15de59c640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e00ad2f0 0x7f15e00a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:01.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 -- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 msgr2=0x7f15e00ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e00ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 -- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e00ad2f0 msgr2=0x7f15e00a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e00ad2f0 0x7f15e00a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ddd9b640 1 -- 172.21.15.71:0/350751475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8009400 con 0x7f15e00c72f0 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.197+0000 7f15ded9d640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e00ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.198+0000 7f15ddd9b640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 0x7f15e00aaba0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f15c8000c00 tx=0x7f15c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.198+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c803f070 con 0x7f15e00c72f0 2023-08-25T09:53:01.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.198+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c8004030 con 0x7f15e00c72f0 2023-08-25T09:53:01.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.198+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c80043c0 con 0x7f15e00c72f0 2023-08-25T09:53:01.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.198+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e00c8f00 con 0x7f15e00c72f0 2023-08-25T09:53:01.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.198+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e00c94c0 con 0x7f15e00c72f0 2023-08-25T09:53:01.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.199+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e00ae320 con 0x7f15e00c72f0 2023-08-25T09:53:01.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.200+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f15c8043070 con 0x7f15e00c72f0 2023-08-25T09:53:01.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.201+0000 7f15d6ffd640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f15c006a050 0x7f15c006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:01.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.201+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f15c80aa700 con 0x7f15e00c72f0 2023-08-25T09:53:01.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.201+0000 7f15de59c640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f15c006a050 0x7f15c006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:01.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.202+0000 7f15de59c640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f15c006a050 0x7f15c006c510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f15d00074a0 tx=0x7f15d000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:53:01.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.206+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f15c80759d0 con 0x7f15e00c72f0 2023-08-25T09:53:01.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.432+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15e004e420 con 0x7f15e00c72f0 2023-08-25T09:53:01.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.433+0000 7f15d6ffd640 1 -- 172.21.15.71:0/350751475 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f15c8078ff0 con 0x7f15e00c72f0 2023-08-25T09:53:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:53:01.435 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:53.771532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:53:01.436 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 26 2023-08-25T09:53:01.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f15c006a050 msgr2=0x7f15c006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:01.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f15c006a050 0x7f15c006c510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f15d00074a0 tx=0x7f15d000b040 comp rx=0 tx=0).stop 2023-08-25T09:53:01.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 msgr2=0x7f15e00aaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:01.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 0x7f15e00aaba0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f15c8000c00 tx=0x7f15c80046c0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 shutdown_connections 2023-08-25T09:53:01.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f15c006a050 0x7f15c006c510 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15e00c76d0 0x7f15e00ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15e00c72f0 0x7f15e00aaba0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 --2- 172.21.15.71:0/350751475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15e00ad2f0 0x7f15e00a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:01.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 >> 172.21.15.71:0/350751475 conn(0x7f15e006c280 msgr2=0x7f15e0150330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:53:01.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 shutdown_connections 2023-08-25T09:53:01.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:01.436+0000 7f15df59e640 1 -- 172.21.15.71:0/350751475 wait complete. 2023-08-25T09:53:01.459 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-08-25T09:53:01.462 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/30 file after the snapshot 2023-08-25T09:53:01.466 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/31 file after the snapshot 2023-08-25T09:53:01.469 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/32 file after the snapshot 2023-08-25T09:53:01.473 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/33 file after the snapshot 2023-08-25T09:53:01.476 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/34 file after the snapshot 2023-08-25T09:53:01.480 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/35 file after the snapshot 2023-08-25T09:53:01.484 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/36 file after the snapshot 2023-08-25T09:53:01.487 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/37 file after the snapshot 2023-08-25T09:53:01.491 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/38 file after the snapshot 2023-08-25T09:53:01.494 INFO:tasks.workunit.client.0.smithi071.stdout: create 142/39 file after the snapshot 2023-08-25T09:53:01.497 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/.snap/snap-subdir-test 2023-08-25T09:53:01.531 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/30 file after the snapshot 2023-08-25T09:53:01.535 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/31 file after the snapshot 2023-08-25T09:53:01.540 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/32 file after the snapshot 2023-08-25T09:53:01.544 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/33 file after the snapshot 2023-08-25T09:53:01.550 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/34 file after the snapshot 2023-08-25T09:53:01.554 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/35 file after the snapshot 2023-08-25T09:53:01.561 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/36 file after the snapshot 2023-08-25T09:53:01.567 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/37 file after the snapshot 2023-08-25T09:53:01.573 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/38 file after the snapshot 2023-08-25T09:53:01.580 INFO:tasks.workunit.client.0.smithi071.stdout: create 143/39 file after the snapshot 2023-08-25T09:53:01.588 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/.snap/snap-subdir-test 2023-08-25T09:53:01.623 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/30 file after the snapshot 2023-08-25T09:53:01.628 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/31 file after the snapshot 2023-08-25T09:53:01.632 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/32 file after the snapshot 2023-08-25T09:53:01.638 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/33 file after the snapshot 2023-08-25T09:53:01.642 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/34 file after the snapshot 2023-08-25T09:53:01.648 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/35 file after the snapshot 2023-08-25T09:53:01.652 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/36 file after the snapshot 2023-08-25T09:53:01.658 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/37 file after the snapshot 2023-08-25T09:53:01.664 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/38 file after the snapshot 2023-08-25T09:53:01.669 INFO:tasks.workunit.client.0.smithi071.stdout: create 144/39 file after the snapshot 2023-08-25T09:53:01.676 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/.snap/snap-subdir-test 2023-08-25T09:53:01.720 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/30 file after the snapshot 2023-08-25T09:53:01.726 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/31 file after the snapshot 2023-08-25T09:53:01.732 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/32 file after the snapshot 2023-08-25T09:53:01.738 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/33 file after the snapshot 2023-08-25T09:53:01.742 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/34 file after the snapshot 2023-08-25T09:53:01.747 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/35 file after the snapshot 2023-08-25T09:53:01.753 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/36 file after the snapshot 2023-08-25T09:53:01.757 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/37 file after the snapshot 2023-08-25T09:53:01.762 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/38 file after the snapshot 2023-08-25T09:53:01.766 INFO:tasks.workunit.client.0.smithi071.stdout: create 145/39 file after the snapshot 2023-08-25T09:53:01.771 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/.snap/snap-subdir-test 2023-08-25T09:53:01.803 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/30 file after the snapshot 2023-08-25T09:53:01.809 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/31 file after the snapshot 2023-08-25T09:53:01.814 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/32 file after the snapshot 2023-08-25T09:53:01.818 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/33 file after the snapshot 2023-08-25T09:53:01.824 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/34 file after the snapshot 2023-08-25T09:53:01.830 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/35 file after the snapshot 2023-08-25T09:53:01.836 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/36 file after the snapshot 2023-08-25T09:53:01.843 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/37 file after the snapshot 2023-08-25T09:53:01.850 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/38 file after the snapshot 2023-08-25T09:53:01.856 INFO:tasks.workunit.client.0.smithi071.stdout: create 146/39 file after the snapshot 2023-08-25T09:53:01.862 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/.snap/snap-subdir-test 2023-08-25T09:53:01.914 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/30 file after the snapshot 2023-08-25T09:53:01.919 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/31 file after the snapshot 2023-08-25T09:53:01.924 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/32 file after the snapshot 2023-08-25T09:53:01.929 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/33 file after the snapshot 2023-08-25T09:53:01.934 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/34 file after the snapshot 2023-08-25T09:53:01.939 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/35 file after the snapshot 2023-08-25T09:53:01.945 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/36 file after the snapshot 2023-08-25T09:53:01.950 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/37 file after the snapshot 2023-08-25T09:53:01.955 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/38 file after the snapshot 2023-08-25T09:53:01.960 INFO:tasks.workunit.client.0.smithi071.stdout: create 147/39 file after the snapshot 2023-08-25T09:53:01.966 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/.snap/snap-subdir-test 2023-08-25T09:53:02.001 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/30 file after the snapshot 2023-08-25T09:53:02.008 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/31 file after the snapshot 2023-08-25T09:53:02.014 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/32 file after the snapshot 2023-08-25T09:53:02.020 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/33 file after the snapshot 2023-08-25T09:53:02.026 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/34 file after the snapshot 2023-08-25T09:53:02.033 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/35 file after the snapshot 2023-08-25T09:53:02.040 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/36 file after the snapshot 2023-08-25T09:53:02.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/37 file after the snapshot 2023-08-25T09:53:02.052 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/38 file after the snapshot 2023-08-25T09:53:02.059 INFO:tasks.workunit.client.0.smithi071.stdout: create 148/39 file after the snapshot 2023-08-25T09:53:02.064 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/.snap/snap-subdir-test 2023-08-25T09:53:02.111 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/30 file after the snapshot 2023-08-25T09:53:02.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/31 file after the snapshot 2023-08-25T09:53:02.157 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/32 file after the snapshot 2023-08-25T09:53:02.185 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/33 file after the snapshot 2023-08-25T09:53:02.211 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/34 file after the snapshot 2023-08-25T09:53:02.240 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/35 file after the snapshot 2023-08-25T09:53:02.346 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/36 file after the snapshot 2023-08-25T09:53:02.373 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/37 file after the snapshot 2023-08-25T09:53:02.403 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/38 file after the snapshot 2023-08-25T09:53:02.431 INFO:tasks.workunit.client.0.smithi071.stdout: create 149/39 file after the snapshot 2023-08-25T09:53:02.460 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/.snap/snap-subdir-test 2023-08-25T09:53:02.502 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/30 file after the snapshot 2023-08-25T09:53:02.512 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/31 file after the snapshot 2023-08-25T09:53:02.539 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/32 file after the snapshot 2023-08-25T09:53:02.566 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/33 file after the snapshot 2023-08-25T09:53:02.594 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/34 file after the snapshot 2023-08-25T09:53:03.185 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/35 file after the snapshot 2023-08-25T09:53:03.215 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/36 file after the snapshot 2023-08-25T09:53:03.242 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/37 file after the snapshot 2023-08-25T09:53:03.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:53:03.571 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/38 file after the snapshot 2023-08-25T09:53:03.592 INFO:tasks.workunit.client.0.smithi071.stdout: create 150/39 file after the snapshot 2023-08-25T09:53:03.618 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/.snap/snap-subdir-test 2023-08-25T09:53:03.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.690+0000 7fcd0d7c8640 1 Processor -- start 2023-08-25T09:53:03.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.690+0000 7fcd0d7c8640 1 -- start start 2023-08-25T09:53:03.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd0d7c8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcd08152d80 0x7fcd0806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:03.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd0d7c8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd08070cf0 0x7fcd081637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:03.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd0d7c8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0806eb60 con 0x7fcd080c4560 2023-08-25T09:53:03.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd0d7c8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0806ca30 con 0x7fcd08070cf0 2023-08-25T09:53:03.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd0d7c8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0806cf90 con 0x7fcd08152d80 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd077fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd08070cf0 0x7fcd081637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd077fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcd08070cf0 0x7fcd081637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53604/0 (socket says 172.21.15.71:53604) 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd077fe640 1 -- 172.21.15.71:0/3941854872 learned_addr learned my addr 172.21.15.71:0/3941854872 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.692+0000 7fcd06ffd640 1 --2- 172.21.15.71:0/3941854872 >> v2:172.21.15.164:3300/0 conn(0x7fcd08152d80 0x7fcd0806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.693+0000 7fcd077fe640 1 -- 172.21.15.71:0/3941854872 >> v2:172.21.15.164:3300/0 conn(0x7fcd08152d80 msgr2=0x7fcd0806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.693+0000 7fcd077fe640 1 --2- 172.21.15.71:0/3941854872 >> v2:172.21.15.164:3300/0 conn(0x7fcd08152d80 0x7fcd0806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.693+0000 7fcd077fe640 1 -- 172.21.15.71:0/3941854872 >> v1:172.21.15.164:6789/0 conn(0x7fcd080c4560 legacy=0x7fcd080c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:53:03.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.693+0000 7fcd077fe640 1 -- 172.21.15.71:0/3941854872 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08158fd0 con 0x7fcd08070cf0 2023-08-25T09:53:03.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.693+0000 7fcd06ffd640 1 --2- 172.21.15.71:0/3941854872 >> v2:172.21.15.164:3300/0 conn(0x7fcd08152d80 0x7fcd0806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:53:03.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.694+0000 7fcd077fe640 1 --2- 172.21.15.71:0/3941854872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd08070cf0 0x7fcd081637b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcd0806dee0 tx=0x7fcd0002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafb138e87996645 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:53:03.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.694+0000 7fcd067fc640 1 -- 172.21.15.71:0/3941854872 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0002c9f0 con 0x7fcd08070cf0 2023-08-25T09:53:03.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.694+0000 7fcd067fc640 1 -- 172.21.15.71:0/3941854872 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0002cb50 con 0x7fcd08070cf0 2023-08-25T09:53:03.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.694+0000 7fcd067fc640 1 -- 172.21.15.71:0/3941854872 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd000343f0 con 0x7fcd08070cf0 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/3941854872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd08070cf0 msgr2=0x7fcd081637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/3941854872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd08070cf0 0x7fcd081637b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcd0806dee0 tx=0x7fcd0002be50 comp rx=0 tx=0).stop 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/3941854872 shutdown_connections 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/3941854872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd08070cf0 0x7fcd081637b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/3941854872 >> v2:172.21.15.164:3300/0 conn(0x7fcd08152d80 0x7fcd0806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/3941854872 >> 172.21.15.71:0/3941854872 conn(0x7fcd080654f0 msgr2=0x7fcd08065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:53:03.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/3941854872 shutdown_connections 2023-08-25T09:53:03.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.695+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/3941854872 wait complete. 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.696+0000 7fcd0d7c8640 1 Processor -- start 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.697+0000 7fcd0d7c8640 1 -- start start 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fcd0d7c8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd08070cf0 0x7fcd081eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fcd0d7c8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 0x7fcd08069ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fcd0d7c8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd08152d80 0x7fcd08149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fcd0d7c8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd080c17b0 con 0x7fcd080c4560 2023-08-25T09:53:03.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fcd0d7c8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd081ef120 con 0x7fcd08152d80 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fcd0d7c8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0806a710 con 0x7fcd08070cf0 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 0x7fcd08069ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 0x7fcd08069ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:53620/0 (socket says 172.21.15.71:53620) 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 -- 172.21.15.71:0/578836244 learned_addr learned my addr 172.21.15.71:0/578836244 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 -- 172.21.15.71:0/578836244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd08070cf0 msgr2=0x7fcd081eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd08070cf0 0x7fcd081eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 -- 172.21.15.71:0/578836244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd08152d80 msgr2=0x7fcd08149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:03.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd08152d80 0x7fcd08149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.698+0000 7fccfffff640 1 -- 172.21.15.71:0/578836244 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd00009b30 con 0x7fcd080c4560 2023-08-25T09:53:03.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.699+0000 7fccfffff640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 0x7fcd08069ce0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b3e0 tx=0x7fccf000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:53:03.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.699+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0011830 con 0x7fcd080c4560 2023-08-25T09:53:03.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.699+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf00119b0 con 0x7fcd080c4560 2023-08-25T09:53:03.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.699+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0011d60 con 0x7fcd080c4560 2023-08-25T09:53:03.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.699+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd080c2d80 con 0x7fcd080c4560 2023-08-25T09:53:03.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.699+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd080c3340 con 0x7fcd080c4560 2023-08-25T09:53:03.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.700+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd08069160 con 0x7fcd080c4560 2023-08-25T09:53:03.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.701+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fccf001a400 con 0x7fcd080c4560 2023-08-25T09:53:03.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.702+0000 7fccff7fe640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcce806a050 0x7fcce806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:53:03.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.702+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fccf0088af0 con 0x7fcd080c4560 2023-08-25T09:53:03.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.702+0000 7fcd06ffd640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcce806a050 0x7fcce806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:53:03.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.703+0000 7fcd06ffd640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcce806a050 0x7fcce806c510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fccf8006fd0 tx=0x7fccf8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:53:03.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.706+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fccf0053dc0 con 0x7fcd080c4560 2023-08-25T09:53:03.888 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/30 file after the snapshot 2023-08-25T09:53:03.900 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/31 file after the snapshot 2023-08-25T09:53:03.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.947+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd080c18f0 con 0x7fcd080c4560 2023-08-25T09:53:03.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.949+0000 7fccff7fe640 1 -- 172.21.15.71:0/578836244 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+6529 (secure 0 0 0) 0x7fccf00573e0 con 0x7fcd080c4560 2023-08-25T09:53:03.951 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:53:03.952 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:52:53.771532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:53:03.953 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 27 2023-08-25T09:53:03.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcce806a050 msgr2=0x7fcce806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:03.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcce806a050 0x7fcce806c510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fccf8006fd0 tx=0x7fccf8008040 comp rx=0 tx=0).stop 2023-08-25T09:53:03.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 msgr2=0x7fcd08069ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:53:03.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 0x7fcd08069ce0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b3e0 tx=0x7fccf000b8a0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 shutdown_connections 2023-08-25T09:53:03.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd08152d80 0x7fcd08149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcce806a050 0x7fcce806c510 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcd080c4560 0x7fcd08069ce0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 --2- 172.21.15.71:0/578836244 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd08070cf0 0x7fcd081eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:53:03.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 >> 172.21.15.71:0/578836244 conn(0x7fcd080654f0 msgr2=0x7fcd080c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:53:03.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 shutdown_connections 2023-08-25T09:53:03.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:53:03.951+0000 7fcd0d7c8640 1 -- 172.21.15.71:0/578836244 wait complete. 2023-08-25T09:53:03.973 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-08-25T09:53:03.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 108 secs before thrashing 2023-08-25T09:53:04.682 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/32 file after the snapshot 2023-08-25T09:53:04.714 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/33 file after the snapshot 2023-08-25T09:53:04.742 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/34 file after the snapshot 2023-08-25T09:53:04.769 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/35 file after the snapshot 2023-08-25T09:53:04.801 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/36 file after the snapshot 2023-08-25T09:53:05.097 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/37 file after the snapshot 2023-08-25T09:53:05.125 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/38 file after the snapshot 2023-08-25T09:53:05.154 INFO:tasks.workunit.client.0.smithi071.stdout: create 151/39 file after the snapshot 2023-08-25T09:53:05.185 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/.snap/snap-subdir-test 2023-08-25T09:53:05.223 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/30 file after the snapshot 2023-08-25T09:53:05.228 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/31 file after the snapshot 2023-08-25T09:53:05.236 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/32 file after the snapshot 2023-08-25T09:53:05.244 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/33 file after the snapshot 2023-08-25T09:53:05.252 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/34 file after the snapshot 2023-08-25T09:53:05.262 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/35 file after the snapshot 2023-08-25T09:53:05.271 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/36 file after the snapshot 2023-08-25T09:53:05.281 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/37 file after the snapshot 2023-08-25T09:53:05.291 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/38 file after the snapshot 2023-08-25T09:53:05.301 INFO:tasks.workunit.client.0.smithi071.stdout: create 152/39 file after the snapshot 2023-08-25T09:53:06.016 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/.snap/snap-subdir-test 2023-08-25T09:53:06.069 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/30 file after the snapshot 2023-08-25T09:53:06.076 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/31 file after the snapshot 2023-08-25T09:53:06.085 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/32 file after the snapshot 2023-08-25T09:53:06.092 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/33 file after the snapshot 2023-08-25T09:53:06.101 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/34 file after the snapshot 2023-08-25T09:53:06.110 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/35 file after the snapshot 2023-08-25T09:53:06.119 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/36 file after the snapshot 2023-08-25T09:53:06.130 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/37 file after the snapshot 2023-08-25T09:53:06.140 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/38 file after the snapshot 2023-08-25T09:53:06.149 INFO:tasks.workunit.client.0.smithi071.stdout: create 153/39 file after the snapshot 2023-08-25T09:53:06.159 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/.snap/snap-subdir-test 2023-08-25T09:53:06.203 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/30 file after the snapshot 2023-08-25T09:53:06.209 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/31 file after the snapshot 2023-08-25T09:53:06.217 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/32 file after the snapshot 2023-08-25T09:53:06.224 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/33 file after the snapshot 2023-08-25T09:53:06.232 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/34 file after the snapshot 2023-08-25T09:53:06.240 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/35 file after the snapshot 2023-08-25T09:53:06.247 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/36 file after the snapshot 2023-08-25T09:53:06.253 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/37 file after the snapshot 2023-08-25T09:53:06.258 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/38 file after the snapshot 2023-08-25T09:53:06.265 INFO:tasks.workunit.client.0.smithi071.stdout: create 154/39 file after the snapshot 2023-08-25T09:53:06.270 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/.snap/snap-subdir-test 2023-08-25T09:53:06.323 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/30 file after the snapshot 2023-08-25T09:53:06.680 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/31 file after the snapshot 2023-08-25T09:53:06.730 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/32 file after the snapshot 2023-08-25T09:53:07.543 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/33 file after the snapshot 2023-08-25T09:53:07.572 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/34 file after the snapshot 2023-08-25T09:53:07.603 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/35 file after the snapshot 2023-08-25T09:53:07.651 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/36 file after the snapshot 2023-08-25T09:53:07.701 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/37 file after the snapshot 2023-08-25T09:53:07.735 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/38 file after the snapshot 2023-08-25T09:53:07.766 INFO:tasks.workunit.client.0.smithi071.stdout: create 155/39 file after the snapshot 2023-08-25T09:53:08.683 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/.snap/snap-subdir-test 2023-08-25T09:53:08.746 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/30 file after the snapshot 2023-08-25T09:53:08.763 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/31 file after the snapshot 2023-08-25T09:53:08.796 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/32 file after the snapshot 2023-08-25T09:53:08.825 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/33 file after the snapshot 2023-08-25T09:53:08.860 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/34 file after the snapshot 2023-08-25T09:53:08.889 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/35 file after the snapshot 2023-08-25T09:53:08.917 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/36 file after the snapshot 2023-08-25T09:53:08.946 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/37 file after the snapshot 2023-08-25T09:53:08.976 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/38 file after the snapshot 2023-08-25T09:53:09.007 INFO:tasks.workunit.client.0.smithi071.stdout: create 156/39 file after the snapshot 2023-08-25T09:53:09.036 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/.snap/snap-subdir-test 2023-08-25T09:53:09.074 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/30 file after the snapshot 2023-08-25T09:53:09.086 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/31 file after the snapshot 2023-08-25T09:53:09.115 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/32 file after the snapshot 2023-08-25T09:53:09.154 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/33 file after the snapshot 2023-08-25T09:53:09.174 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/34 file after the snapshot 2023-08-25T09:53:09.203 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/35 file after the snapshot 2023-08-25T09:53:09.230 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/36 file after the snapshot 2023-08-25T09:53:09.263 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/37 file after the snapshot 2023-08-25T09:53:09.797 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/38 file after the snapshot 2023-08-25T09:53:09.825 INFO:tasks.workunit.client.0.smithi071.stdout: create 157/39 file after the snapshot 2023-08-25T09:53:09.852 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/.snap/snap-subdir-test 2023-08-25T09:53:09.892 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/30 file after the snapshot 2023-08-25T09:53:09.906 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/31 file after the snapshot 2023-08-25T09:53:09.938 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/32 file after the snapshot 2023-08-25T09:53:10.307 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/33 file after the snapshot 2023-08-25T09:53:10.337 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/34 file after the snapshot 2023-08-25T09:53:10.367 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/35 file after the snapshot 2023-08-25T09:53:10.398 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/36 file after the snapshot 2023-08-25T09:53:10.426 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/37 file after the snapshot 2023-08-25T09:53:10.456 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/38 file after the snapshot 2023-08-25T09:53:10.485 INFO:tasks.workunit.client.0.smithi071.stdout: create 158/39 file after the snapshot 2023-08-25T09:53:10.516 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/.snap/snap-subdir-test 2023-08-25T09:53:10.558 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/30 file after the snapshot 2023-08-25T09:53:11.063 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/31 file after the snapshot 2023-08-25T09:53:11.111 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/32 file after the snapshot 2023-08-25T09:53:11.139 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/33 file after the snapshot 2023-08-25T09:53:11.170 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/34 file after the snapshot 2023-08-25T09:53:11.201 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/35 file after the snapshot 2023-08-25T09:53:11.231 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/36 file after the snapshot 2023-08-25T09:53:11.262 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/37 file after the snapshot 2023-08-25T09:53:11.291 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/38 file after the snapshot 2023-08-25T09:53:11.321 INFO:tasks.workunit.client.0.smithi071.stdout: create 159/39 file after the snapshot 2023-08-25T09:53:11.348 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/.snap/snap-subdir-test 2023-08-25T09:53:11.390 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/30 file after the snapshot 2023-08-25T09:53:11.556 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/31 file after the snapshot 2023-08-25T09:53:12.550 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/32 file after the snapshot 2023-08-25T09:53:13.048 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/33 file after the snapshot 2023-08-25T09:53:13.079 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/34 file after the snapshot 2023-08-25T09:53:13.109 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/35 file after the snapshot 2023-08-25T09:53:13.139 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/36 file after the snapshot 2023-08-25T09:53:13.172 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/37 file after the snapshot 2023-08-25T09:53:13.201 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/38 file after the snapshot 2023-08-25T09:53:13.234 INFO:tasks.workunit.client.0.smithi071.stdout: create 160/39 file after the snapshot 2023-08-25T09:53:13.267 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/.snap/snap-subdir-test 2023-08-25T09:53:14.311 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/30 file after the snapshot 2023-08-25T09:53:14.327 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/31 file after the snapshot 2023-08-25T09:53:14.358 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/32 file after the snapshot 2023-08-25T09:53:14.386 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/33 file after the snapshot 2023-08-25T09:53:14.412 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/34 file after the snapshot 2023-08-25T09:53:14.441 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/35 file after the snapshot 2023-08-25T09:53:14.469 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/36 file after the snapshot 2023-08-25T09:53:14.498 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/37 file after the snapshot 2023-08-25T09:53:14.526 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/38 file after the snapshot 2023-08-25T09:53:16.142 INFO:tasks.workunit.client.0.smithi071.stdout: create 161/39 file after the snapshot 2023-08-25T09:53:16.174 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/.snap/snap-subdir-test 2023-08-25T09:53:16.222 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/30 file after the snapshot 2023-08-25T09:53:16.469 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/31 file after the snapshot 2023-08-25T09:53:16.505 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/32 file after the snapshot 2023-08-25T09:53:16.536 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/33 file after the snapshot 2023-08-25T09:53:16.568 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/34 file after the snapshot 2023-08-25T09:53:16.600 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/35 file after the snapshot 2023-08-25T09:53:16.629 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/36 file after the snapshot 2023-08-25T09:53:16.658 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/37 file after the snapshot 2023-08-25T09:53:16.697 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/38 file after the snapshot 2023-08-25T09:53:16.728 INFO:tasks.workunit.client.0.smithi071.stdout: create 162/39 file after the snapshot 2023-08-25T09:53:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/.snap/snap-subdir-test 2023-08-25T09:53:16.804 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/30 file after the snapshot 2023-08-25T09:53:17.191 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/31 file after the snapshot 2023-08-25T09:53:17.228 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/32 file after the snapshot 2023-08-25T09:53:17.261 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/33 file after the snapshot 2023-08-25T09:53:17.291 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/34 file after the snapshot 2023-08-25T09:53:17.320 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/35 file after the snapshot 2023-08-25T09:53:17.350 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/36 file after the snapshot 2023-08-25T09:53:17.379 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/37 file after the snapshot 2023-08-25T09:53:17.409 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/38 file after the snapshot 2023-08-25T09:53:17.437 INFO:tasks.workunit.client.0.smithi071.stdout: create 163/39 file after the snapshot 2023-08-25T09:53:17.465 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/.snap/snap-subdir-test 2023-08-25T09:53:17.504 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/30 file after the snapshot 2023-08-25T09:53:17.518 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/31 file after the snapshot 2023-08-25T09:53:17.549 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/32 file after the snapshot 2023-08-25T09:53:17.578 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/33 file after the snapshot 2023-08-25T09:53:17.609 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/34 file after the snapshot 2023-08-25T09:53:17.642 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/35 file after the snapshot 2023-08-25T09:53:17.674 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/36 file after the snapshot 2023-08-25T09:53:17.701 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/37 file after the snapshot 2023-08-25T09:53:17.730 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/38 file after the snapshot 2023-08-25T09:53:17.771 INFO:tasks.workunit.client.0.smithi071.stdout: create 164/39 file after the snapshot 2023-08-25T09:53:17.800 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/.snap/snap-subdir-test 2023-08-25T09:53:17.843 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/30 file after the snapshot 2023-08-25T09:53:17.859 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/31 file after the snapshot 2023-08-25T09:53:17.891 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/32 file after the snapshot 2023-08-25T09:53:17.920 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/33 file after the snapshot 2023-08-25T09:53:17.949 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/34 file after the snapshot 2023-08-25T09:53:17.981 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/35 file after the snapshot 2023-08-25T09:53:18.014 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/36 file after the snapshot 2023-08-25T09:53:18.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/37 file after the snapshot 2023-08-25T09:53:18.077 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/38 file after the snapshot 2023-08-25T09:53:18.106 INFO:tasks.workunit.client.0.smithi071.stdout: create 165/39 file after the snapshot 2023-08-25T09:53:18.137 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/.snap/snap-subdir-test 2023-08-25T09:53:18.176 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/30 file after the snapshot 2023-08-25T09:53:18.181 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/31 file after the snapshot 2023-08-25T09:53:18.188 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/32 file after the snapshot 2023-08-25T09:53:18.195 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/33 file after the snapshot 2023-08-25T09:53:18.204 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/34 file after the snapshot 2023-08-25T09:53:18.214 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/35 file after the snapshot 2023-08-25T09:53:18.225 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/36 file after the snapshot 2023-08-25T09:53:18.235 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/37 file after the snapshot 2023-08-25T09:53:18.243 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/38 file after the snapshot 2023-08-25T09:53:18.252 INFO:tasks.workunit.client.0.smithi071.stdout: create 166/39 file after the snapshot 2023-08-25T09:53:18.262 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/.snap/snap-subdir-test 2023-08-25T09:53:18.305 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/30 file after the snapshot 2023-08-25T09:53:18.311 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/31 file after the snapshot 2023-08-25T09:53:18.319 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/32 file after the snapshot 2023-08-25T09:53:18.328 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/33 file after the snapshot 2023-08-25T09:53:18.336 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/34 file after the snapshot 2023-08-25T09:53:18.344 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/35 file after the snapshot 2023-08-25T09:53:18.351 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/36 file after the snapshot 2023-08-25T09:53:18.359 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/37 file after the snapshot 2023-08-25T09:53:18.365 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/38 file after the snapshot 2023-08-25T09:53:18.372 INFO:tasks.workunit.client.0.smithi071.stdout: create 167/39 file after the snapshot 2023-08-25T09:53:18.379 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/.snap/snap-subdir-test 2023-08-25T09:53:18.423 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/30 file after the snapshot 2023-08-25T09:53:18.439 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/31 file after the snapshot 2023-08-25T09:53:19.308 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/32 file after the snapshot 2023-08-25T09:53:19.344 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/33 file after the snapshot 2023-08-25T09:53:19.371 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/34 file after the snapshot 2023-08-25T09:53:19.400 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/35 file after the snapshot 2023-08-25T09:53:19.430 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/36 file after the snapshot 2023-08-25T09:53:19.461 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/37 file after the snapshot 2023-08-25T09:53:19.492 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/38 file after the snapshot 2023-08-25T09:53:19.521 INFO:tasks.workunit.client.0.smithi071.stdout: create 168/39 file after the snapshot 2023-08-25T09:53:19.552 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/.snap/snap-subdir-test 2023-08-25T09:53:19.592 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/30 file after the snapshot 2023-08-25T09:53:19.603 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/31 file after the snapshot 2023-08-25T09:53:19.630 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/32 file after the snapshot 2023-08-25T09:53:20.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/33 file after the snapshot 2023-08-25T09:53:20.175 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/34 file after the snapshot 2023-08-25T09:53:20.205 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/35 file after the snapshot 2023-08-25T09:53:20.233 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/36 file after the snapshot 2023-08-25T09:53:20.260 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/37 file after the snapshot 2023-08-25T09:53:20.288 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/38 file after the snapshot 2023-08-25T09:53:20.316 INFO:tasks.workunit.client.0.smithi071.stdout: create 169/39 file after the snapshot 2023-08-25T09:53:20.344 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/.snap/snap-subdir-test 2023-08-25T09:53:20.387 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/30 file after the snapshot 2023-08-25T09:53:20.394 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/31 file after the snapshot 2023-08-25T09:53:20.403 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/32 file after the snapshot 2023-08-25T09:53:20.411 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/33 file after the snapshot 2023-08-25T09:53:20.420 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/34 file after the snapshot 2023-08-25T09:53:20.429 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/35 file after the snapshot 2023-08-25T09:53:20.437 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/36 file after the snapshot 2023-08-25T09:53:20.447 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/37 file after the snapshot 2023-08-25T09:53:20.455 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/38 file after the snapshot 2023-08-25T09:53:20.463 INFO:tasks.workunit.client.0.smithi071.stdout: create 170/39 file after the snapshot 2023-08-25T09:53:20.470 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/.snap/snap-subdir-test 2023-08-25T09:53:20.508 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/30 file after the snapshot 2023-08-25T09:53:21.470 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/31 file after the snapshot 2023-08-25T09:53:21.508 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/32 file after the snapshot 2023-08-25T09:53:21.538 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/33 file after the snapshot 2023-08-25T09:53:21.567 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/34 file after the snapshot 2023-08-25T09:53:21.595 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/35 file after the snapshot 2023-08-25T09:53:21.626 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/36 file after the snapshot 2023-08-25T09:53:21.657 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/37 file after the snapshot 2023-08-25T09:53:21.687 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/38 file after the snapshot 2023-08-25T09:53:21.716 INFO:tasks.workunit.client.0.smithi071.stdout: create 171/39 file after the snapshot 2023-08-25T09:53:21.744 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/.snap/snap-subdir-test 2023-08-25T09:53:21.783 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/30 file after the snapshot 2023-08-25T09:53:21.796 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/31 file after the snapshot 2023-08-25T09:53:21.826 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/32 file after the snapshot 2023-08-25T09:53:21.859 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/33 file after the snapshot 2023-08-25T09:53:21.887 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/34 file after the snapshot 2023-08-25T09:53:21.917 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/35 file after the snapshot 2023-08-25T09:53:21.948 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/36 file after the snapshot 2023-08-25T09:53:21.977 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/37 file after the snapshot 2023-08-25T09:53:22.005 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/38 file after the snapshot 2023-08-25T09:53:22.032 INFO:tasks.workunit.client.0.smithi071.stdout: create 172/39 file after the snapshot 2023-08-25T09:53:22.064 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/.snap/snap-subdir-test 2023-08-25T09:53:22.099 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/30 file after the snapshot 2023-08-25T09:53:22.107 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/31 file after the snapshot 2023-08-25T09:53:22.115 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/32 file after the snapshot 2023-08-25T09:53:22.121 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/33 file after the snapshot 2023-08-25T09:53:22.128 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/34 file after the snapshot 2023-08-25T09:53:22.135 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/35 file after the snapshot 2023-08-25T09:53:22.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/36 file after the snapshot 2023-08-25T09:53:22.147 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/37 file after the snapshot 2023-08-25T09:53:22.153 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/38 file after the snapshot 2023-08-25T09:53:22.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:53:22.792 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:53:23.017 INFO:tasks.workunit.client.0.smithi071.stdout: create 173/39 file after the snapshot 2023-08-25T09:53:23.029 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/.snap/snap-subdir-test 2023-08-25T09:53:23.068 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/30 file after the snapshot 2023-08-25T09:53:23.076 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/31 file after the snapshot 2023-08-25T09:53:23.083 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/32 file after the snapshot 2023-08-25T09:53:23.090 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/33 file after the snapshot 2023-08-25T09:53:23.100 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/34 file after the snapshot 2023-08-25T09:53:23.109 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/35 file after the snapshot 2023-08-25T09:53:23.121 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/36 file after the snapshot 2023-08-25T09:53:23.131 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/37 file after the snapshot 2023-08-25T09:53:23.140 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/38 file after the snapshot 2023-08-25T09:53:23.150 INFO:tasks.workunit.client.0.smithi071.stdout: create 174/39 file after the snapshot 2023-08-25T09:53:23.158 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/.snap/snap-subdir-test 2023-08-25T09:53:23.196 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/30 file after the snapshot 2023-08-25T09:53:23.204 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/31 file after the snapshot 2023-08-25T09:53:23.234 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/32 file after the snapshot 2023-08-25T09:53:23.264 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/33 file after the snapshot 2023-08-25T09:53:23.293 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/34 file after the snapshot 2023-08-25T09:53:23.322 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/35 file after the snapshot 2023-08-25T09:53:23.351 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/36 file after the snapshot 2023-08-25T09:53:23.378 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/37 file after the snapshot 2023-08-25T09:53:23.889 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/38 file after the snapshot 2023-08-25T09:53:23.917 INFO:tasks.workunit.client.0.smithi071.stdout: create 175/39 file after the snapshot 2023-08-25T09:53:23.944 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/.snap/snap-subdir-test 2023-08-25T09:53:24.177 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/30 file after the snapshot 2023-08-25T09:53:24.197 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/31 file after the snapshot 2023-08-25T09:53:24.229 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/32 file after the snapshot 2023-08-25T09:53:24.259 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/33 file after the snapshot 2023-08-25T09:53:24.871 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/34 file after the snapshot 2023-08-25T09:53:24.903 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/35 file after the snapshot 2023-08-25T09:53:24.931 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/36 file after the snapshot 2023-08-25T09:53:24.961 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/37 file after the snapshot 2023-08-25T09:53:24.992 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/38 file after the snapshot 2023-08-25T09:53:25.021 INFO:tasks.workunit.client.0.smithi071.stdout: create 176/39 file after the snapshot 2023-08-25T09:53:25.050 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/.snap/snap-subdir-test 2023-08-25T09:53:25.093 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/30 file after the snapshot 2023-08-25T09:53:25.104 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/31 file after the snapshot 2023-08-25T09:53:25.134 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/32 file after the snapshot 2023-08-25T09:53:25.160 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/33 file after the snapshot 2023-08-25T09:53:25.191 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/34 file after the snapshot 2023-08-25T09:53:26.001 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/35 file after the snapshot 2023-08-25T09:53:26.030 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/36 file after the snapshot 2023-08-25T09:53:26.056 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/37 file after the snapshot 2023-08-25T09:53:26.083 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/38 file after the snapshot 2023-08-25T09:53:26.111 INFO:tasks.workunit.client.0.smithi071.stdout: create 177/39 file after the snapshot 2023-08-25T09:53:26.140 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/.snap/snap-subdir-test 2023-08-25T09:53:26.186 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/30 file after the snapshot 2023-08-25T09:53:26.202 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/31 file after the snapshot 2023-08-25T09:53:26.235 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/32 file after the snapshot 2023-08-25T09:53:26.265 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/33 file after the snapshot 2023-08-25T09:53:26.297 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/34 file after the snapshot 2023-08-25T09:53:26.327 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/35 file after the snapshot 2023-08-25T09:53:26.357 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/36 file after the snapshot 2023-08-25T09:53:26.388 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/37 file after the snapshot 2023-08-25T09:53:26.420 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/38 file after the snapshot 2023-08-25T09:53:26.448 INFO:tasks.workunit.client.0.smithi071.stdout: create 178/39 file after the snapshot 2023-08-25T09:53:26.780 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/.snap/snap-subdir-test 2023-08-25T09:53:26.820 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/30 file after the snapshot 2023-08-25T09:53:26.839 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/31 file after the snapshot 2023-08-25T09:53:27.677 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/32 file after the snapshot 2023-08-25T09:53:28.532 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/33 file after the snapshot 2023-08-25T09:53:28.563 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/34 file after the snapshot 2023-08-25T09:53:28.590 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/35 file after the snapshot 2023-08-25T09:53:28.710 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/36 file after the snapshot 2023-08-25T09:53:28.755 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/37 file after the snapshot 2023-08-25T09:53:28.786 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/38 file after the snapshot 2023-08-25T09:53:28.816 INFO:tasks.workunit.client.0.smithi071.stdout: create 179/39 file after the snapshot 2023-08-25T09:53:28.851 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/.snap/snap-subdir-test 2023-08-25T09:53:28.892 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/30 file after the snapshot 2023-08-25T09:53:28.908 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/31 file after the snapshot 2023-08-25T09:53:28.939 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/32 file after the snapshot 2023-08-25T09:53:28.968 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/33 file after the snapshot 2023-08-25T09:53:29.080 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/34 file after the snapshot 2023-08-25T09:53:29.104 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/35 file after the snapshot 2023-08-25T09:53:29.134 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/36 file after the snapshot 2023-08-25T09:53:29.165 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/37 file after the snapshot 2023-08-25T09:53:29.196 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/38 file after the snapshot 2023-08-25T09:53:29.223 INFO:tasks.workunit.client.0.smithi071.stdout: create 180/39 file after the snapshot 2023-08-25T09:53:29.250 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/.snap/snap-subdir-test 2023-08-25T09:53:29.285 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/30 file after the snapshot 2023-08-25T09:53:29.298 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/31 file after the snapshot 2023-08-25T09:53:29.328 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/32 file after the snapshot 2023-08-25T09:53:29.357 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/33 file after the snapshot 2023-08-25T09:53:29.385 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/34 file after the snapshot 2023-08-25T09:53:29.414 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/35 file after the snapshot 2023-08-25T09:53:29.444 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/36 file after the snapshot 2023-08-25T09:53:29.470 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/37 file after the snapshot 2023-08-25T09:53:29.499 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/38 file after the snapshot 2023-08-25T09:53:29.528 INFO:tasks.workunit.client.0.smithi071.stdout: create 181/39 file after the snapshot 2023-08-25T09:53:29.556 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/.snap/snap-subdir-test 2023-08-25T09:53:29.593 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/30 file after the snapshot 2023-08-25T09:53:29.599 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/31 file after the snapshot 2023-08-25T09:53:29.605 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/32 file after the snapshot 2023-08-25T09:53:29.611 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/33 file after the snapshot 2023-08-25T09:53:29.617 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/34 file after the snapshot 2023-08-25T09:53:29.624 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/35 file after the snapshot 2023-08-25T09:53:29.631 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/36 file after the snapshot 2023-08-25T09:53:29.638 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/37 file after the snapshot 2023-08-25T09:53:29.644 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/38 file after the snapshot 2023-08-25T09:53:29.651 INFO:tasks.workunit.client.0.smithi071.stdout: create 182/39 file after the snapshot 2023-08-25T09:53:29.657 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/.snap/snap-subdir-test 2023-08-25T09:53:29.700 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/30 file after the snapshot 2023-08-25T09:53:29.705 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/31 file after the snapshot 2023-08-25T09:53:29.711 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/32 file after the snapshot 2023-08-25T09:53:29.719 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/33 file after the snapshot 2023-08-25T09:53:29.722 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/34 file after the snapshot 2023-08-25T09:53:29.727 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/35 file after the snapshot 2023-08-25T09:53:29.732 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/36 file after the snapshot 2023-08-25T09:53:29.739 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/37 file after the snapshot 2023-08-25T09:53:29.745 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/38 file after the snapshot 2023-08-25T09:53:29.752 INFO:tasks.workunit.client.0.smithi071.stdout: create 183/39 file after the snapshot 2023-08-25T09:53:29.756 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/.snap/snap-subdir-test 2023-08-25T09:53:29.796 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/30 file after the snapshot 2023-08-25T09:53:29.802 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/31 file after the snapshot 2023-08-25T09:53:29.808 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/32 file after the snapshot 2023-08-25T09:53:29.817 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/33 file after the snapshot 2023-08-25T09:53:29.824 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/34 file after the snapshot 2023-08-25T09:53:29.831 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/35 file after the snapshot 2023-08-25T09:53:29.840 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/36 file after the snapshot 2023-08-25T09:53:29.847 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/37 file after the snapshot 2023-08-25T09:53:29.854 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/38 file after the snapshot 2023-08-25T09:53:29.861 INFO:tasks.workunit.client.0.smithi071.stdout: create 184/39 file after the snapshot 2023-08-25T09:53:29.868 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/.snap/snap-subdir-test 2023-08-25T09:53:29.921 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/30 file after the snapshot 2023-08-25T09:53:29.926 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/31 file after the snapshot 2023-08-25T09:53:29.932 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/32 file after the snapshot 2023-08-25T09:53:29.938 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/33 file after the snapshot 2023-08-25T09:53:29.944 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/34 file after the snapshot 2023-08-25T09:53:29.950 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/35 file after the snapshot 2023-08-25T09:53:29.954 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/36 file after the snapshot 2023-08-25T09:53:29.958 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/37 file after the snapshot 2023-08-25T09:53:29.963 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/38 file after the snapshot 2023-08-25T09:53:29.968 INFO:tasks.workunit.client.0.smithi071.stdout: create 185/39 file after the snapshot 2023-08-25T09:53:29.973 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/.snap/snap-subdir-test 2023-08-25T09:53:30.438 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/30 file after the snapshot 2023-08-25T09:53:30.447 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/31 file after the snapshot 2023-08-25T09:53:30.456 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/32 file after the snapshot 2023-08-25T09:53:30.466 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/33 file after the snapshot 2023-08-25T09:53:30.476 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/34 file after the snapshot 2023-08-25T09:53:30.485 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/35 file after the snapshot 2023-08-25T09:53:30.496 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/36 file after the snapshot 2023-08-25T09:53:30.505 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/37 file after the snapshot 2023-08-25T09:53:30.516 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/38 file after the snapshot 2023-08-25T09:53:30.529 INFO:tasks.workunit.client.0.smithi071.stdout: create 186/39 file after the snapshot 2023-08-25T09:53:30.541 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/.snap/snap-subdir-test 2023-08-25T09:53:30.588 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/30 file after the snapshot 2023-08-25T09:53:30.595 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/31 file after the snapshot 2023-08-25T09:53:30.599 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/32 file after the snapshot 2023-08-25T09:53:30.605 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/33 file after the snapshot 2023-08-25T09:53:30.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/34 file after the snapshot 2023-08-25T09:53:30.617 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/35 file after the snapshot 2023-08-25T09:53:30.624 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/36 file after the snapshot 2023-08-25T09:53:30.630 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/37 file after the snapshot 2023-08-25T09:53:30.638 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/38 file after the snapshot 2023-08-25T09:53:30.644 INFO:tasks.workunit.client.0.smithi071.stdout: create 187/39 file after the snapshot 2023-08-25T09:53:30.651 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/.snap/snap-subdir-test 2023-08-25T09:53:30.701 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/30 file after the snapshot 2023-08-25T09:53:30.705 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/31 file after the snapshot 2023-08-25T09:53:30.710 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/32 file after the snapshot 2023-08-25T09:53:30.713 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/33 file after the snapshot 2023-08-25T09:53:30.717 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/34 file after the snapshot 2023-08-25T09:53:30.721 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/35 file after the snapshot 2023-08-25T09:53:30.726 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/36 file after the snapshot 2023-08-25T09:53:30.730 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/37 file after the snapshot 2023-08-25T09:53:30.734 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/38 file after the snapshot 2023-08-25T09:53:30.737 INFO:tasks.workunit.client.0.smithi071.stdout: create 188/39 file after the snapshot 2023-08-25T09:53:30.741 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/.snap/snap-subdir-test 2023-08-25T09:53:30.770 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/30 file after the snapshot 2023-08-25T09:53:30.774 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/31 file after the snapshot 2023-08-25T09:53:30.779 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/32 file after the snapshot 2023-08-25T09:53:30.783 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/33 file after the snapshot 2023-08-25T09:53:30.786 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/34 file after the snapshot 2023-08-25T09:53:30.790 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/35 file after the snapshot 2023-08-25T09:53:30.794 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/36 file after the snapshot 2023-08-25T09:53:30.798 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/37 file after the snapshot 2023-08-25T09:53:30.803 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/38 file after the snapshot 2023-08-25T09:53:30.807 INFO:tasks.workunit.client.0.smithi071.stdout: create 189/39 file after the snapshot 2023-08-25T09:53:30.811 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/.snap/snap-subdir-test 2023-08-25T09:53:31.456 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/30 file after the snapshot 2023-08-25T09:53:31.465 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/31 file after the snapshot 2023-08-25T09:53:31.475 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/32 file after the snapshot 2023-08-25T09:53:31.484 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/33 file after the snapshot 2023-08-25T09:53:31.494 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/34 file after the snapshot 2023-08-25T09:53:32.111 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/35 file after the snapshot 2023-08-25T09:53:32.121 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/36 file after the snapshot 2023-08-25T09:53:32.132 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/37 file after the snapshot 2023-08-25T09:53:32.142 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/38 file after the snapshot 2023-08-25T09:53:32.152 INFO:tasks.workunit.client.0.smithi071.stdout: create 190/39 file after the snapshot 2023-08-25T09:53:32.162 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/.snap/snap-subdir-test 2023-08-25T09:53:32.202 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/30 file after the snapshot 2023-08-25T09:53:32.208 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/31 file after the snapshot 2023-08-25T09:53:32.214 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/32 file after the snapshot 2023-08-25T09:53:32.221 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/33 file after the snapshot 2023-08-25T09:53:32.227 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/34 file after the snapshot 2023-08-25T09:53:32.234 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/35 file after the snapshot 2023-08-25T09:53:32.242 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/36 file after the snapshot 2023-08-25T09:53:32.250 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/37 file after the snapshot 2023-08-25T09:53:32.257 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/38 file after the snapshot 2023-08-25T09:53:32.264 INFO:tasks.workunit.client.0.smithi071.stdout: create 191/39 file after the snapshot 2023-08-25T09:53:32.271 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/.snap/snap-subdir-test 2023-08-25T09:53:32.317 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/30 file after the snapshot 2023-08-25T09:53:32.322 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/31 file after the snapshot 2023-08-25T09:53:32.329 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/32 file after the snapshot 2023-08-25T09:53:32.335 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/33 file after the snapshot 2023-08-25T09:53:32.342 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/34 file after the snapshot 2023-08-25T09:53:32.347 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/35 file after the snapshot 2023-08-25T09:53:32.353 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/36 file after the snapshot 2023-08-25T09:53:32.360 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/37 file after the snapshot 2023-08-25T09:53:32.367 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/38 file after the snapshot 2023-08-25T09:53:32.374 INFO:tasks.workunit.client.0.smithi071.stdout: create 192/39 file after the snapshot 2023-08-25T09:53:32.381 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/.snap/snap-subdir-test 2023-08-25T09:53:32.428 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/30 file after the snapshot 2023-08-25T09:53:32.433 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/31 file after the snapshot 2023-08-25T09:53:32.438 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/32 file after the snapshot 2023-08-25T09:53:32.442 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/33 file after the snapshot 2023-08-25T09:53:32.447 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/34 file after the snapshot 2023-08-25T09:53:32.454 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/35 file after the snapshot 2023-08-25T09:53:32.460 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/36 file after the snapshot 2023-08-25T09:53:32.467 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/37 file after the snapshot 2023-08-25T09:53:32.473 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/38 file after the snapshot 2023-08-25T09:53:32.480 INFO:tasks.workunit.client.0.smithi071.stdout: create 193/39 file after the snapshot 2023-08-25T09:53:32.486 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/.snap/snap-subdir-test 2023-08-25T09:53:32.528 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/30 file after the snapshot 2023-08-25T09:53:32.533 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/31 file after the snapshot 2023-08-25T09:53:32.541 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/32 file after the snapshot 2023-08-25T09:53:32.545 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/33 file after the snapshot 2023-08-25T09:53:32.553 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/34 file after the snapshot 2023-08-25T09:53:32.561 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/35 file after the snapshot 2023-08-25T09:53:32.566 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/36 file after the snapshot 2023-08-25T09:53:32.569 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/37 file after the snapshot 2023-08-25T09:53:32.573 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/38 file after the snapshot 2023-08-25T09:53:32.576 INFO:tasks.workunit.client.0.smithi071.stdout: create 194/39 file after the snapshot 2023-08-25T09:53:32.580 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/.snap/snap-subdir-test 2023-08-25T09:53:32.609 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/30 file after the snapshot 2023-08-25T09:53:32.615 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/31 file after the snapshot 2023-08-25T09:53:32.627 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/32 file after the snapshot 2023-08-25T09:53:32.631 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/33 file after the snapshot 2023-08-25T09:53:32.640 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/34 file after the snapshot 2023-08-25T09:53:32.650 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/35 file after the snapshot 2023-08-25T09:53:32.659 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/36 file after the snapshot 2023-08-25T09:53:32.668 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/37 file after the snapshot 2023-08-25T09:53:32.678 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/38 file after the snapshot 2023-08-25T09:53:32.688 INFO:tasks.workunit.client.0.smithi071.stdout: create 195/39 file after the snapshot 2023-08-25T09:53:32.697 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/.snap/snap-subdir-test 2023-08-25T09:53:32.730 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/30 file after the snapshot 2023-08-25T09:53:32.734 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/31 file after the snapshot 2023-08-25T09:53:32.741 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/32 file after the snapshot 2023-08-25T09:53:32.746 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/33 file after the snapshot 2023-08-25T09:53:32.752 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/34 file after the snapshot 2023-08-25T09:53:32.757 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/35 file after the snapshot 2023-08-25T09:53:32.762 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/36 file after the snapshot 2023-08-25T09:53:32.768 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/37 file after the snapshot 2023-08-25T09:53:32.773 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/38 file after the snapshot 2023-08-25T09:53:32.777 INFO:tasks.workunit.client.0.smithi071.stdout: create 196/39 file after the snapshot 2023-08-25T09:53:32.781 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/.snap/snap-subdir-test 2023-08-25T09:53:33.432 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/30 file after the snapshot 2023-08-25T09:53:33.437 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/31 file after the snapshot 2023-08-25T09:53:33.441 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/32 file after the snapshot 2023-08-25T09:53:33.449 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/33 file after the snapshot 2023-08-25T09:53:33.455 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/34 file after the snapshot 2023-08-25T09:53:33.461 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/35 file after the snapshot 2023-08-25T09:53:33.470 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/36 file after the snapshot 2023-08-25T09:53:33.477 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/37 file after the snapshot 2023-08-25T09:53:33.483 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/38 file after the snapshot 2023-08-25T09:53:33.489 INFO:tasks.workunit.client.0.smithi071.stdout: create 197/39 file after the snapshot 2023-08-25T09:53:33.495 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/.snap/snap-subdir-test 2023-08-25T09:53:33.535 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/30 file after the snapshot 2023-08-25T09:53:33.541 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/31 file after the snapshot 2023-08-25T09:53:33.546 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/32 file after the snapshot 2023-08-25T09:53:33.551 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/33 file after the snapshot 2023-08-25T09:53:33.556 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/34 file after the snapshot 2023-08-25T09:53:33.560 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/35 file after the snapshot 2023-08-25T09:53:33.565 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/36 file after the snapshot 2023-08-25T09:53:33.569 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/37 file after the snapshot 2023-08-25T09:53:33.573 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/38 file after the snapshot 2023-08-25T09:53:33.578 INFO:tasks.workunit.client.0.smithi071.stdout: create 198/39 file after the snapshot 2023-08-25T09:53:33.583 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/.snap/snap-subdir-test 2023-08-25T09:53:33.620 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/30 file after the snapshot 2023-08-25T09:53:33.627 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/31 file after the snapshot 2023-08-25T09:53:33.634 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/32 file after the snapshot 2023-08-25T09:53:33.641 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/33 file after the snapshot 2023-08-25T09:53:33.647 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/34 file after the snapshot 2023-08-25T09:53:33.653 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/35 file after the snapshot 2023-08-25T09:53:33.659 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/36 file after the snapshot 2023-08-25T09:53:33.666 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/37 file after the snapshot 2023-08-25T09:53:33.672 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/38 file after the snapshot 2023-08-25T09:53:33.678 INFO:tasks.workunit.client.0.smithi071.stdout: create 199/39 file after the snapshot 2023-08-25T09:53:33.685 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/.snap/snap-subdir-test 2023-08-25T09:53:33.736 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/30 file after the snapshot 2023-08-25T09:53:33.743 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/31 file after the snapshot 2023-08-25T09:53:33.752 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/32 file after the snapshot 2023-08-25T09:53:33.761 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/33 file after the snapshot 2023-08-25T09:53:33.771 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/34 file after the snapshot 2023-08-25T09:53:33.781 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/35 file after the snapshot 2023-08-25T09:53:33.792 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/36 file after the snapshot 2023-08-25T09:53:33.805 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/37 file after the snapshot 2023-08-25T09:53:33.816 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/38 file after the snapshot 2023-08-25T09:53:33.826 INFO:tasks.workunit.client.0.smithi071.stdout: create 200/39 file after the snapshot 2023-08-25T09:53:33.838 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/.snap/snap-subdir-test 2023-08-25T09:53:33.879 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/30 file after the snapshot 2023-08-25T09:53:33.885 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/31 file after the snapshot 2023-08-25T09:53:33.892 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/32 file after the snapshot 2023-08-25T09:53:33.898 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/33 file after the snapshot 2023-08-25T09:53:33.905 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/34 file after the snapshot 2023-08-25T09:53:33.912 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/35 file after the snapshot 2023-08-25T09:53:33.919 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/36 file after the snapshot 2023-08-25T09:53:33.925 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/37 file after the snapshot 2023-08-25T09:53:33.933 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/38 file after the snapshot 2023-08-25T09:53:33.939 INFO:tasks.workunit.client.0.smithi071.stdout: create 201/39 file after the snapshot 2023-08-25T09:53:33.946 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/.snap/snap-subdir-test 2023-08-25T09:53:33.999 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/30 file after the snapshot 2023-08-25T09:53:34.004 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/31 file after the snapshot 2023-08-25T09:53:34.009 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/32 file after the snapshot 2023-08-25T09:53:34.013 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/33 file after the snapshot 2023-08-25T09:53:34.020 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/34 file after the snapshot 2023-08-25T09:53:34.027 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/35 file after the snapshot 2023-08-25T09:53:34.033 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/36 file after the snapshot 2023-08-25T09:53:34.040 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/37 file after the snapshot 2023-08-25T09:53:34.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/38 file after the snapshot 2023-08-25T09:53:34.053 INFO:tasks.workunit.client.0.smithi071.stdout: create 202/39 file after the snapshot 2023-08-25T09:53:34.059 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/.snap/snap-subdir-test 2023-08-25T09:53:34.101 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/30 file after the snapshot 2023-08-25T09:53:34.107 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/31 file after the snapshot 2023-08-25T09:53:34.112 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/32 file after the snapshot 2023-08-25T09:53:34.117 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/33 file after the snapshot 2023-08-25T09:53:34.123 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/34 file after the snapshot 2023-08-25T09:53:34.130 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/35 file after the snapshot 2023-08-25T09:53:34.136 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/36 file after the snapshot 2023-08-25T09:53:34.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/37 file after the snapshot 2023-08-25T09:53:34.145 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/38 file after the snapshot 2023-08-25T09:53:34.149 INFO:tasks.workunit.client.0.smithi071.stdout: create 203/39 file after the snapshot 2023-08-25T09:53:34.155 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/.snap/snap-subdir-test 2023-08-25T09:53:34.191 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/30 file after the snapshot 2023-08-25T09:53:34.196 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/31 file after the snapshot 2023-08-25T09:53:34.203 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/32 file after the snapshot 2023-08-25T09:53:34.209 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/33 file after the snapshot 2023-08-25T09:53:34.215 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/34 file after the snapshot 2023-08-25T09:53:34.221 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/35 file after the snapshot 2023-08-25T09:53:34.228 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/36 file after the snapshot 2023-08-25T09:53:34.234 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/37 file after the snapshot 2023-08-25T09:53:34.242 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/38 file after the snapshot 2023-08-25T09:53:34.248 INFO:tasks.workunit.client.0.smithi071.stdout: create 204/39 file after the snapshot 2023-08-25T09:53:34.254 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/.snap/snap-subdir-test 2023-08-25T09:53:34.296 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/30 file after the snapshot 2023-08-25T09:53:34.301 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/31 file after the snapshot 2023-08-25T09:53:34.556 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/32 file after the snapshot 2023-08-25T09:53:34.563 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/33 file after the snapshot 2023-08-25T09:53:34.571 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/34 file after the snapshot 2023-08-25T09:53:34.578 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/35 file after the snapshot 2023-08-25T09:53:34.585 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/36 file after the snapshot 2023-08-25T09:53:34.592 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/37 file after the snapshot 2023-08-25T09:53:34.696 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/38 file after the snapshot 2023-08-25T09:53:34.979 INFO:tasks.workunit.client.0.smithi071.stdout: create 205/39 file after the snapshot 2023-08-25T09:53:34.986 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/.snap/snap-subdir-test 2023-08-25T09:53:35.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/30 file after the snapshot 2023-08-25T09:53:35.049 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/31 file after the snapshot 2023-08-25T09:53:35.054 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/32 file after the snapshot 2023-08-25T09:53:35.058 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/33 file after the snapshot 2023-08-25T09:53:35.062 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/34 file after the snapshot 2023-08-25T09:53:35.069 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/35 file after the snapshot 2023-08-25T09:53:35.075 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/36 file after the snapshot 2023-08-25T09:53:35.082 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/37 file after the snapshot 2023-08-25T09:53:35.088 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/38 file after the snapshot 2023-08-25T09:53:35.094 INFO:tasks.workunit.client.0.smithi071.stdout: create 206/39 file after the snapshot 2023-08-25T09:53:35.099 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/.snap/snap-subdir-test 2023-08-25T09:53:35.132 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/30 file after the snapshot 2023-08-25T09:53:35.135 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/31 file after the snapshot 2023-08-25T09:53:35.139 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/32 file after the snapshot 2023-08-25T09:53:35.142 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/33 file after the snapshot 2023-08-25T09:53:35.147 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/34 file after the snapshot 2023-08-25T09:53:35.152 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/35 file after the snapshot 2023-08-25T09:53:35.158 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/36 file after the snapshot 2023-08-25T09:53:35.161 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/37 file after the snapshot 2023-08-25T09:53:35.166 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/38 file after the snapshot 2023-08-25T09:53:35.173 INFO:tasks.workunit.client.0.smithi071.stdout: create 207/39 file after the snapshot 2023-08-25T09:53:35.177 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/.snap/snap-subdir-test 2023-08-25T09:53:35.566 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/30 file after the snapshot 2023-08-25T09:53:35.572 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/31 file after the snapshot 2023-08-25T09:53:35.577 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/32 file after the snapshot 2023-08-25T09:53:35.583 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/33 file after the snapshot 2023-08-25T09:53:35.591 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/34 file after the snapshot 2023-08-25T09:53:35.598 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/35 file after the snapshot 2023-08-25T09:53:35.603 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/36 file after the snapshot 2023-08-25T09:53:35.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/37 file after the snapshot 2023-08-25T09:53:35.618 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/38 file after the snapshot 2023-08-25T09:53:35.623 INFO:tasks.workunit.client.0.smithi071.stdout: create 208/39 file after the snapshot 2023-08-25T09:53:35.628 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/.snap/snap-subdir-test 2023-08-25T09:53:35.710 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/30 file after the snapshot 2023-08-25T09:53:35.715 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/31 file after the snapshot 2023-08-25T09:53:35.719 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/32 file after the snapshot 2023-08-25T09:53:35.724 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/33 file after the snapshot 2023-08-25T09:53:35.729 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/34 file after the snapshot 2023-08-25T09:53:35.734 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/35 file after the snapshot 2023-08-25T09:53:35.742 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/36 file after the snapshot 2023-08-25T09:53:35.746 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/37 file after the snapshot 2023-08-25T09:53:35.751 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/38 file after the snapshot 2023-08-25T09:53:35.756 INFO:tasks.workunit.client.0.smithi071.stdout: create 209/39 file after the snapshot 2023-08-25T09:53:35.761 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/.snap/snap-subdir-test 2023-08-25T09:53:35.817 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/30 file after the snapshot 2023-08-25T09:53:35.823 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/31 file after the snapshot 2023-08-25T09:53:35.828 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/32 file after the snapshot 2023-08-25T09:53:35.835 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/33 file after the snapshot 2023-08-25T09:53:35.841 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/34 file after the snapshot 2023-08-25T09:53:35.848 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/35 file after the snapshot 2023-08-25T09:53:35.854 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/36 file after the snapshot 2023-08-25T09:53:35.859 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/37 file after the snapshot 2023-08-25T09:53:35.864 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/38 file after the snapshot 2023-08-25T09:53:35.870 INFO:tasks.workunit.client.0.smithi071.stdout: create 210/39 file after the snapshot 2023-08-25T09:53:35.874 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/.snap/snap-subdir-test 2023-08-25T09:53:35.908 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/30 file after the snapshot 2023-08-25T09:53:35.913 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/31 file after the snapshot 2023-08-25T09:53:36.036 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/32 file after the snapshot 2023-08-25T09:53:36.043 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/33 file after the snapshot 2023-08-25T09:53:36.051 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/34 file after the snapshot 2023-08-25T09:53:36.059 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/35 file after the snapshot 2023-08-25T09:53:36.067 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/36 file after the snapshot 2023-08-25T09:53:36.074 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/37 file after the snapshot 2023-08-25T09:53:36.082 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/38 file after the snapshot 2023-08-25T09:53:36.089 INFO:tasks.workunit.client.0.smithi071.stdout: create 211/39 file after the snapshot 2023-08-25T09:53:36.097 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/.snap/snap-subdir-test 2023-08-25T09:53:36.147 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/30 file after the snapshot 2023-08-25T09:53:36.152 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/31 file after the snapshot 2023-08-25T09:53:36.159 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/32 file after the snapshot 2023-08-25T09:53:36.165 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/33 file after the snapshot 2023-08-25T09:53:36.171 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/34 file after the snapshot 2023-08-25T09:53:36.178 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/35 file after the snapshot 2023-08-25T09:53:36.183 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/36 file after the snapshot 2023-08-25T09:53:36.189 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/37 file after the snapshot 2023-08-25T09:53:36.195 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/38 file after the snapshot 2023-08-25T09:53:36.202 INFO:tasks.workunit.client.0.smithi071.stdout: create 212/39 file after the snapshot 2023-08-25T09:53:36.208 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/.snap/snap-subdir-test 2023-08-25T09:53:36.248 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/30 file after the snapshot 2023-08-25T09:53:36.256 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/31 file after the snapshot 2023-08-25T09:53:36.264 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/32 file after the snapshot 2023-08-25T09:53:36.274 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/33 file after the snapshot 2023-08-25T09:53:36.283 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/34 file after the snapshot 2023-08-25T09:53:36.293 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/35 file after the snapshot 2023-08-25T09:53:36.303 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/36 file after the snapshot 2023-08-25T09:53:36.313 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/37 file after the snapshot 2023-08-25T09:53:36.324 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/38 file after the snapshot 2023-08-25T09:53:36.348 INFO:tasks.workunit.client.0.smithi071.stdout: create 213/39 file after the snapshot 2023-08-25T09:53:36.357 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/.snap/snap-subdir-test 2023-08-25T09:53:36.399 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/30 file after the snapshot 2023-08-25T09:53:36.404 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/31 file after the snapshot 2023-08-25T09:53:36.409 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/32 file after the snapshot 2023-08-25T09:53:36.414 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/33 file after the snapshot 2023-08-25T09:53:36.420 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/34 file after the snapshot 2023-08-25T09:53:36.425 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/35 file after the snapshot 2023-08-25T09:53:36.430 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/36 file after the snapshot 2023-08-25T09:53:36.435 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/37 file after the snapshot 2023-08-25T09:53:36.440 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/38 file after the snapshot 2023-08-25T09:53:36.445 INFO:tasks.workunit.client.0.smithi071.stdout: create 214/39 file after the snapshot 2023-08-25T09:53:36.449 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/.snap/snap-subdir-test 2023-08-25T09:53:36.481 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/30 file after the snapshot 2023-08-25T09:53:36.487 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/31 file after the snapshot 2023-08-25T09:53:36.493 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/32 file after the snapshot 2023-08-25T09:53:36.499 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/33 file after the snapshot 2023-08-25T09:53:36.506 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/34 file after the snapshot 2023-08-25T09:53:36.513 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/35 file after the snapshot 2023-08-25T09:53:36.518 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/36 file after the snapshot 2023-08-25T09:53:36.523 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/37 file after the snapshot 2023-08-25T09:53:36.528 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/38 file after the snapshot 2023-08-25T09:53:36.534 INFO:tasks.workunit.client.0.smithi071.stdout: create 215/39 file after the snapshot 2023-08-25T09:53:36.539 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/.snap/snap-subdir-test 2023-08-25T09:53:36.579 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/30 file after the snapshot 2023-08-25T09:53:36.585 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/31 file after the snapshot 2023-08-25T09:53:36.591 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/32 file after the snapshot 2023-08-25T09:53:36.596 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/33 file after the snapshot 2023-08-25T09:53:36.601 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/34 file after the snapshot 2023-08-25T09:53:36.606 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/35 file after the snapshot 2023-08-25T09:53:36.611 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/36 file after the snapshot 2023-08-25T09:53:36.618 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/37 file after the snapshot 2023-08-25T09:53:36.622 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/38 file after the snapshot 2023-08-25T09:53:36.627 INFO:tasks.workunit.client.0.smithi071.stdout: create 216/39 file after the snapshot 2023-08-25T09:53:36.630 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/.snap/snap-subdir-test 2023-08-25T09:53:36.761 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/30 file after the snapshot 2023-08-25T09:53:37.457 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/31 file after the snapshot 2023-08-25T09:53:37.465 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/32 file after the snapshot 2023-08-25T09:53:37.473 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/33 file after the snapshot 2023-08-25T09:53:37.479 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/34 file after the snapshot 2023-08-25T09:53:37.486 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/35 file after the snapshot 2023-08-25T09:53:37.494 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/36 file after the snapshot 2023-08-25T09:53:37.500 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/37 file after the snapshot 2023-08-25T09:53:37.509 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/38 file after the snapshot 2023-08-25T09:53:37.516 INFO:tasks.workunit.client.0.smithi071.stdout: create 217/39 file after the snapshot 2023-08-25T09:53:37.522 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/.snap/snap-subdir-test 2023-08-25T09:53:37.571 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/30 file after the snapshot 2023-08-25T09:53:37.576 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/31 file after the snapshot 2023-08-25T09:53:37.581 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/32 file after the snapshot 2023-08-25T09:53:37.586 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/33 file after the snapshot 2023-08-25T09:53:37.590 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/34 file after the snapshot 2023-08-25T09:53:37.595 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/35 file after the snapshot 2023-08-25T09:53:37.600 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/36 file after the snapshot 2023-08-25T09:53:37.607 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/37 file after the snapshot 2023-08-25T09:53:37.611 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/38 file after the snapshot 2023-08-25T09:53:37.616 INFO:tasks.workunit.client.0.smithi071.stdout: create 218/39 file after the snapshot 2023-08-25T09:53:37.621 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/.snap/snap-subdir-test 2023-08-25T09:53:37.664 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/30 file after the snapshot 2023-08-25T09:53:37.670 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/31 file after the snapshot 2023-08-25T09:53:37.674 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/32 file after the snapshot 2023-08-25T09:53:37.681 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/33 file after the snapshot 2023-08-25T09:53:37.687 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/34 file after the snapshot 2023-08-25T09:53:37.691 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/35 file after the snapshot 2023-08-25T09:53:37.696 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/36 file after the snapshot 2023-08-25T09:53:37.701 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/37 file after the snapshot 2023-08-25T09:53:37.709 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/38 file after the snapshot 2023-08-25T09:53:37.722 INFO:tasks.workunit.client.0.smithi071.stdout: create 219/39 file after the snapshot 2023-08-25T09:53:38.375 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/.snap/snap-subdir-test 2023-08-25T09:53:38.424 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/30 file after the snapshot 2023-08-25T09:53:38.428 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/31 file after the snapshot 2023-08-25T09:53:38.433 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/32 file after the snapshot 2023-08-25T09:53:38.440 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/33 file after the snapshot 2023-08-25T09:53:38.445 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/34 file after the snapshot 2023-08-25T09:53:38.451 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/35 file after the snapshot 2023-08-25T09:53:38.460 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/36 file after the snapshot 2023-08-25T09:53:38.467 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/37 file after the snapshot 2023-08-25T09:53:38.475 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/38 file after the snapshot 2023-08-25T09:53:38.482 INFO:tasks.workunit.client.0.smithi071.stdout: create 220/39 file after the snapshot 2023-08-25T09:53:38.487 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/.snap/snap-subdir-test 2023-08-25T09:53:38.523 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/30 file after the snapshot 2023-08-25T09:53:38.527 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/31 file after the snapshot 2023-08-25T09:53:38.532 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/32 file after the snapshot 2023-08-25T09:53:38.540 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/33 file after the snapshot 2023-08-25T09:53:38.546 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/34 file after the snapshot 2023-08-25T09:53:38.552 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/35 file after the snapshot 2023-08-25T09:53:38.557 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/36 file after the snapshot 2023-08-25T09:53:38.564 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/37 file after the snapshot 2023-08-25T09:53:38.569 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/38 file after the snapshot 2023-08-25T09:53:38.576 INFO:tasks.workunit.client.0.smithi071.stdout: create 221/39 file after the snapshot 2023-08-25T09:53:38.580 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/.snap/snap-subdir-test 2023-08-25T09:53:38.620 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/30 file after the snapshot 2023-08-25T09:53:38.625 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/31 file after the snapshot 2023-08-25T09:53:38.632 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/32 file after the snapshot 2023-08-25T09:53:38.638 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/33 file after the snapshot 2023-08-25T09:53:38.644 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/34 file after the snapshot 2023-08-25T09:53:38.650 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/35 file after the snapshot 2023-08-25T09:53:38.657 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/36 file after the snapshot 2023-08-25T09:53:38.662 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/37 file after the snapshot 2023-08-25T09:53:38.668 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/38 file after the snapshot 2023-08-25T09:53:38.672 INFO:tasks.workunit.client.0.smithi071.stdout: create 222/39 file after the snapshot 2023-08-25T09:53:38.678 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/.snap/snap-subdir-test 2023-08-25T09:53:38.717 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/30 file after the snapshot 2023-08-25T09:53:38.720 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/31 file after the snapshot 2023-08-25T09:53:38.727 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/32 file after the snapshot 2023-08-25T09:53:38.743 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/33 file after the snapshot 2023-08-25T09:53:38.747 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/34 file after the snapshot 2023-08-25T09:53:38.752 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/35 file after the snapshot 2023-08-25T09:53:38.756 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/36 file after the snapshot 2023-08-25T09:53:38.760 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/37 file after the snapshot 2023-08-25T09:53:38.763 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/38 file after the snapshot 2023-08-25T09:53:38.767 INFO:tasks.workunit.client.0.smithi071.stdout: create 223/39 file after the snapshot 2023-08-25T09:53:38.771 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/.snap/snap-subdir-test 2023-08-25T09:53:38.826 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/30 file after the snapshot 2023-08-25T09:53:38.831 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/31 file after the snapshot 2023-08-25T09:53:38.837 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/32 file after the snapshot 2023-08-25T09:53:38.843 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/33 file after the snapshot 2023-08-25T09:53:38.851 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/34 file after the snapshot 2023-08-25T09:53:38.855 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/35 file after the snapshot 2023-08-25T09:53:38.859 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/36 file after the snapshot 2023-08-25T09:53:38.864 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/37 file after the snapshot 2023-08-25T09:53:38.868 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/38 file after the snapshot 2023-08-25T09:53:38.872 INFO:tasks.workunit.client.0.smithi071.stdout: create 224/39 file after the snapshot 2023-08-25T09:53:38.876 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/.snap/snap-subdir-test 2023-08-25T09:53:38.913 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/30 file after the snapshot 2023-08-25T09:53:38.918 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/31 file after the snapshot 2023-08-25T09:53:38.924 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/32 file after the snapshot 2023-08-25T09:53:38.930 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/33 file after the snapshot 2023-08-25T09:53:38.935 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/34 file after the snapshot 2023-08-25T09:53:38.942 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/35 file after the snapshot 2023-08-25T09:53:38.948 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/36 file after the snapshot 2023-08-25T09:53:38.953 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/37 file after the snapshot 2023-08-25T09:53:38.960 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/38 file after the snapshot 2023-08-25T09:53:38.967 INFO:tasks.workunit.client.0.smithi071.stdout: create 225/39 file after the snapshot 2023-08-25T09:53:38.971 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/.snap/snap-subdir-test 2023-08-25T09:53:39.027 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/30 file after the snapshot 2023-08-25T09:53:39.034 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/31 file after the snapshot 2023-08-25T09:53:39.039 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/32 file after the snapshot 2023-08-25T09:53:39.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/33 file after the snapshot 2023-08-25T09:53:39.051 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/34 file after the snapshot 2023-08-25T09:53:39.058 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/35 file after the snapshot 2023-08-25T09:53:39.064 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/36 file after the snapshot 2023-08-25T09:53:39.070 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/37 file after the snapshot 2023-08-25T09:53:39.076 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/38 file after the snapshot 2023-08-25T09:53:39.081 INFO:tasks.workunit.client.0.smithi071.stdout: create 226/39 file after the snapshot 2023-08-25T09:53:39.085 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/.snap/snap-subdir-test 2023-08-25T09:53:39.118 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/30 file after the snapshot 2023-08-25T09:53:39.125 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/31 file after the snapshot 2023-08-25T09:53:39.131 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/32 file after the snapshot 2023-08-25T09:53:39.136 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/33 file after the snapshot 2023-08-25T09:53:39.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/34 file after the snapshot 2023-08-25T09:53:39.147 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/35 file after the snapshot 2023-08-25T09:53:39.153 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/36 file after the snapshot 2023-08-25T09:53:39.158 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/37 file after the snapshot 2023-08-25T09:53:39.164 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/38 file after the snapshot 2023-08-25T09:53:39.169 INFO:tasks.workunit.client.0.smithi071.stdout: create 227/39 file after the snapshot 2023-08-25T09:53:39.176 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/.snap/snap-subdir-test 2023-08-25T09:53:39.246 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/30 file after the snapshot 2023-08-25T09:53:39.737 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/31 file after the snapshot 2023-08-25T09:53:39.770 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/32 file after the snapshot 2023-08-25T09:53:39.777 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/33 file after the snapshot 2023-08-25T09:53:39.783 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/34 file after the snapshot 2023-08-25T09:53:39.790 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/35 file after the snapshot 2023-08-25T09:53:39.797 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/36 file after the snapshot 2023-08-25T09:53:39.805 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/37 file after the snapshot 2023-08-25T09:53:39.813 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/38 file after the snapshot 2023-08-25T09:53:39.821 INFO:tasks.workunit.client.0.smithi071.stdout: create 228/39 file after the snapshot 2023-08-25T09:53:39.828 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/.snap/snap-subdir-test 2023-08-25T09:53:39.951 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/30 file after the snapshot 2023-08-25T09:53:39.958 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/31 file after the snapshot 2023-08-25T09:53:39.964 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/32 file after the snapshot 2023-08-25T09:53:39.971 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/33 file after the snapshot 2023-08-25T09:53:39.977 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/34 file after the snapshot 2023-08-25T09:53:39.986 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/35 file after the snapshot 2023-08-25T09:53:39.992 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/36 file after the snapshot 2023-08-25T09:53:40.000 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/37 file after the snapshot 2023-08-25T09:53:40.010 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/38 file after the snapshot 2023-08-25T09:53:40.017 INFO:tasks.workunit.client.0.smithi071.stdout: create 229/39 file after the snapshot 2023-08-25T09:53:40.026 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/.snap/snap-subdir-test 2023-08-25T09:53:40.082 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/30 file after the snapshot 2023-08-25T09:53:40.087 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/31 file after the snapshot 2023-08-25T09:53:40.095 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/32 file after the snapshot 2023-08-25T09:53:40.101 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/33 file after the snapshot 2023-08-25T09:53:40.106 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/34 file after the snapshot 2023-08-25T09:53:40.111 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/35 file after the snapshot 2023-08-25T09:53:40.116 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/36 file after the snapshot 2023-08-25T09:53:40.121 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/37 file after the snapshot 2023-08-25T09:53:40.128 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/38 file after the snapshot 2023-08-25T09:53:40.134 INFO:tasks.workunit.client.0.smithi071.stdout: create 230/39 file after the snapshot 2023-08-25T09:53:40.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/.snap/snap-subdir-test 2023-08-25T09:53:40.184 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/30 file after the snapshot 2023-08-25T09:53:40.188 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/31 file after the snapshot 2023-08-25T09:53:40.192 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/32 file after the snapshot 2023-08-25T09:53:40.197 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/33 file after the snapshot 2023-08-25T09:53:40.202 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/34 file after the snapshot 2023-08-25T09:53:40.207 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/35 file after the snapshot 2023-08-25T09:53:40.213 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/36 file after the snapshot 2023-08-25T09:53:40.218 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/37 file after the snapshot 2023-08-25T09:53:40.224 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/38 file after the snapshot 2023-08-25T09:53:40.229 INFO:tasks.workunit.client.0.smithi071.stdout: create 231/39 file after the snapshot 2023-08-25T09:53:40.233 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/.snap/snap-subdir-test 2023-08-25T09:53:40.266 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/30 file after the snapshot 2023-08-25T09:53:40.271 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/31 file after the snapshot 2023-08-25T09:53:40.275 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/32 file after the snapshot 2023-08-25T09:53:40.280 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/33 file after the snapshot 2023-08-25T09:53:40.284 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/34 file after the snapshot 2023-08-25T09:53:40.287 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/35 file after the snapshot 2023-08-25T09:53:40.293 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/36 file after the snapshot 2023-08-25T09:53:40.299 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/37 file after the snapshot 2023-08-25T09:53:40.304 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/38 file after the snapshot 2023-08-25T09:53:40.310 INFO:tasks.workunit.client.0.smithi071.stdout: create 232/39 file after the snapshot 2023-08-25T09:53:40.316 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/.snap/snap-subdir-test 2023-08-25T09:53:40.350 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/30 file after the snapshot 2023-08-25T09:53:40.354 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/31 file after the snapshot 2023-08-25T09:53:40.359 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/32 file after the snapshot 2023-08-25T09:53:40.363 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/33 file after the snapshot 2023-08-25T09:53:40.367 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/34 file after the snapshot 2023-08-25T09:53:40.372 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/35 file after the snapshot 2023-08-25T09:53:40.376 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/36 file after the snapshot 2023-08-25T09:53:40.380 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/37 file after the snapshot 2023-08-25T09:53:40.384 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/38 file after the snapshot 2023-08-25T09:53:40.389 INFO:tasks.workunit.client.0.smithi071.stdout: create 233/39 file after the snapshot 2023-08-25T09:53:40.393 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/.snap/snap-subdir-test 2023-08-25T09:53:40.425 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/30 file after the snapshot 2023-08-25T09:53:40.430 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/31 file after the snapshot 2023-08-25T09:53:40.437 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/32 file after the snapshot 2023-08-25T09:53:40.442 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/33 file after the snapshot 2023-08-25T09:53:40.449 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/34 file after the snapshot 2023-08-25T09:53:40.455 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/35 file after the snapshot 2023-08-25T09:53:40.460 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/36 file after the snapshot 2023-08-25T09:53:40.468 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/37 file after the snapshot 2023-08-25T09:53:40.474 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/38 file after the snapshot 2023-08-25T09:53:40.481 INFO:tasks.workunit.client.0.smithi071.stdout: create 234/39 file after the snapshot 2023-08-25T09:53:40.487 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/.snap/snap-subdir-test 2023-08-25T09:53:40.547 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/30 file after the snapshot 2023-08-25T09:53:40.549 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/31 file after the snapshot 2023-08-25T09:53:40.555 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/32 file after the snapshot 2023-08-25T09:53:40.560 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/33 file after the snapshot 2023-08-25T09:53:40.567 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/34 file after the snapshot 2023-08-25T09:53:40.571 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/35 file after the snapshot 2023-08-25T09:53:40.577 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/36 file after the snapshot 2023-08-25T09:53:40.581 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/37 file after the snapshot 2023-08-25T09:53:40.585 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/38 file after the snapshot 2023-08-25T09:53:40.590 INFO:tasks.workunit.client.0.smithi071.stdout: create 235/39 file after the snapshot 2023-08-25T09:53:40.595 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/.snap/snap-subdir-test 2023-08-25T09:53:41.502 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/30 file after the snapshot 2023-08-25T09:53:41.508 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/31 file after the snapshot 2023-08-25T09:53:41.514 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/32 file after the snapshot 2023-08-25T09:53:41.519 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/33 file after the snapshot 2023-08-25T09:53:41.525 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/34 file after the snapshot 2023-08-25T09:53:41.531 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/35 file after the snapshot 2023-08-25T09:53:41.538 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/36 file after the snapshot 2023-08-25T09:53:41.544 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/37 file after the snapshot 2023-08-25T09:53:41.549 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/38 file after the snapshot 2023-08-25T09:53:41.556 INFO:tasks.workunit.client.0.smithi071.stdout: create 236/39 file after the snapshot 2023-08-25T09:53:41.563 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/.snap/snap-subdir-test 2023-08-25T09:53:41.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/30 file after the snapshot 2023-08-25T09:53:41.619 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/31 file after the snapshot 2023-08-25T09:53:41.630 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/32 file after the snapshot 2023-08-25T09:53:41.925 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/33 file after the snapshot 2023-08-25T09:53:42.497 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/34 file after the snapshot 2023-08-25T09:53:42.508 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/35 file after the snapshot 2023-08-25T09:53:42.519 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/36 file after the snapshot 2023-08-25T09:53:42.528 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/37 file after the snapshot 2023-08-25T09:53:42.539 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/38 file after the snapshot 2023-08-25T09:53:42.552 INFO:tasks.workunit.client.0.smithi071.stdout: create 237/39 file after the snapshot 2023-08-25T09:53:42.563 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/.snap/snap-subdir-test 2023-08-25T09:53:42.601 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/30 file after the snapshot 2023-08-25T09:53:42.608 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/31 file after the snapshot 2023-08-25T09:53:42.615 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/32 file after the snapshot 2023-08-25T09:53:42.626 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/33 file after the snapshot 2023-08-25T09:53:42.637 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/34 file after the snapshot 2023-08-25T09:53:42.645 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/35 file after the snapshot 2023-08-25T09:53:42.651 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/36 file after the snapshot 2023-08-25T09:53:42.660 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/37 file after the snapshot 2023-08-25T09:53:42.665 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/38 file after the snapshot 2023-08-25T09:53:42.670 INFO:tasks.workunit.client.0.smithi071.stdout: create 238/39 file after the snapshot 2023-08-25T09:53:42.675 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/.snap/snap-subdir-test 2023-08-25T09:53:42.722 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/30 file after the snapshot 2023-08-25T09:53:42.726 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/31 file after the snapshot 2023-08-25T09:53:42.730 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/32 file after the snapshot 2023-08-25T09:53:42.736 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/33 file after the snapshot 2023-08-25T09:53:42.739 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/34 file after the snapshot 2023-08-25T09:53:42.743 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/35 file after the snapshot 2023-08-25T09:53:42.747 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/36 file after the snapshot 2023-08-25T09:53:42.751 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/37 file after the snapshot 2023-08-25T09:53:42.754 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/38 file after the snapshot 2023-08-25T09:53:42.759 INFO:tasks.workunit.client.0.smithi071.stdout: create 239/39 file after the snapshot 2023-08-25T09:53:42.768 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/.snap/snap-subdir-test 2023-08-25T09:53:42.797 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/30 file after the snapshot 2023-08-25T09:53:42.802 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/31 file after the snapshot 2023-08-25T09:53:42.809 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/32 file after the snapshot 2023-08-25T09:53:42.814 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/33 file after the snapshot 2023-08-25T09:53:42.820 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/34 file after the snapshot 2023-08-25T09:53:42.826 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/35 file after the snapshot 2023-08-25T09:53:42.830 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/36 file after the snapshot 2023-08-25T09:53:42.837 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/37 file after the snapshot 2023-08-25T09:53:42.843 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/38 file after the snapshot 2023-08-25T09:53:42.848 INFO:tasks.workunit.client.0.smithi071.stdout: create 240/39 file after the snapshot 2023-08-25T09:53:42.854 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/.snap/snap-subdir-test 2023-08-25T09:53:42.894 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/30 file after the snapshot 2023-08-25T09:53:42.900 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/31 file after the snapshot 2023-08-25T09:53:42.906 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/32 file after the snapshot 2023-08-25T09:53:42.913 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/33 file after the snapshot 2023-08-25T09:53:42.918 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/34 file after the snapshot 2023-08-25T09:53:42.925 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/35 file after the snapshot 2023-08-25T09:53:42.931 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/36 file after the snapshot 2023-08-25T09:53:42.937 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/37 file after the snapshot 2023-08-25T09:53:42.942 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/38 file after the snapshot 2023-08-25T09:53:42.950 INFO:tasks.workunit.client.0.smithi071.stdout: create 241/39 file after the snapshot 2023-08-25T09:53:42.956 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/.snap/snap-subdir-test 2023-08-25T09:53:43.012 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/30 file after the snapshot 2023-08-25T09:53:43.016 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/31 file after the snapshot 2023-08-25T09:53:43.020 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/32 file after the snapshot 2023-08-25T09:53:43.026 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/33 file after the snapshot 2023-08-25T09:53:43.031 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/34 file after the snapshot 2023-08-25T09:53:43.036 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/35 file after the snapshot 2023-08-25T09:53:43.041 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/36 file after the snapshot 2023-08-25T09:53:43.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/37 file after the snapshot 2023-08-25T09:53:43.051 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/38 file after the snapshot 2023-08-25T09:53:43.055 INFO:tasks.workunit.client.0.smithi071.stdout: create 242/39 file after the snapshot 2023-08-25T09:53:43.060 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/.snap/snap-subdir-test 2023-08-25T09:53:43.102 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/30 file after the snapshot 2023-08-25T09:53:43.107 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/31 file after the snapshot 2023-08-25T09:53:43.112 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/32 file after the snapshot 2023-08-25T09:53:43.118 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/33 file after the snapshot 2023-08-25T09:53:43.122 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/34 file after the snapshot 2023-08-25T09:53:43.128 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/35 file after the snapshot 2023-08-25T09:53:43.132 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/36 file after the snapshot 2023-08-25T09:53:43.138 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/37 file after the snapshot 2023-08-25T09:53:43.143 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/38 file after the snapshot 2023-08-25T09:53:43.148 INFO:tasks.workunit.client.0.smithi071.stdout: create 243/39 file after the snapshot 2023-08-25T09:53:43.154 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/.snap/snap-subdir-test 2023-08-25T09:53:43.189 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/30 file after the snapshot 2023-08-25T09:53:43.194 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/31 file after the snapshot 2023-08-25T09:53:43.199 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/32 file after the snapshot 2023-08-25T09:53:43.203 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/33 file after the snapshot 2023-08-25T09:53:43.210 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/34 file after the snapshot 2023-08-25T09:53:43.215 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/35 file after the snapshot 2023-08-25T09:53:43.219 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/36 file after the snapshot 2023-08-25T09:53:43.223 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/37 file after the snapshot 2023-08-25T09:53:43.227 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/38 file after the snapshot 2023-08-25T09:53:43.232 INFO:tasks.workunit.client.0.smithi071.stdout: create 244/39 file after the snapshot 2023-08-25T09:53:43.236 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/.snap/snap-subdir-test 2023-08-25T09:53:43.538 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/30 file after the snapshot 2023-08-25T09:53:43.545 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/31 file after the snapshot 2023-08-25T09:53:43.557 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/32 file after the snapshot 2023-08-25T09:53:43.567 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/33 file after the snapshot 2023-08-25T09:53:43.577 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/34 file after the snapshot 2023-08-25T09:53:43.587 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/35 file after the snapshot 2023-08-25T09:53:43.596 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/36 file after the snapshot 2023-08-25T09:53:44.398 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/37 file after the snapshot 2023-08-25T09:53:44.410 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/38 file after the snapshot 2023-08-25T09:53:44.420 INFO:tasks.workunit.client.0.smithi071.stdout: create 245/39 file after the snapshot 2023-08-25T09:53:44.430 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/.snap/snap-subdir-test 2023-08-25T09:53:44.478 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/30 file after the snapshot 2023-08-25T09:53:44.483 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/31 file after the snapshot 2023-08-25T09:53:44.488 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/32 file after the snapshot 2023-08-25T09:53:44.492 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/33 file after the snapshot 2023-08-25T09:53:44.499 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/34 file after the snapshot 2023-08-25T09:53:44.506 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/35 file after the snapshot 2023-08-25T09:53:44.512 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/36 file after the snapshot 2023-08-25T09:53:44.518 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/37 file after the snapshot 2023-08-25T09:53:44.523 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/38 file after the snapshot 2023-08-25T09:53:44.529 INFO:tasks.workunit.client.0.smithi071.stdout: create 246/39 file after the snapshot 2023-08-25T09:53:44.533 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/.snap/snap-subdir-test 2023-08-25T09:53:44.572 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/30 file after the snapshot 2023-08-25T09:53:44.576 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/31 file after the snapshot 2023-08-25T09:53:44.581 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/32 file after the snapshot 2023-08-25T09:53:44.586 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/33 file after the snapshot 2023-08-25T09:53:44.593 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/34 file after the snapshot 2023-08-25T09:53:44.600 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/35 file after the snapshot 2023-08-25T09:53:44.605 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/36 file after the snapshot 2023-08-25T09:53:44.611 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/37 file after the snapshot 2023-08-25T09:53:44.616 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/38 file after the snapshot 2023-08-25T09:53:44.622 INFO:tasks.workunit.client.0.smithi071.stdout: create 247/39 file after the snapshot 2023-08-25T09:53:44.629 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/.snap/snap-subdir-test 2023-08-25T09:53:44.674 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/30 file after the snapshot 2023-08-25T09:53:44.678 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/31 file after the snapshot 2023-08-25T09:53:44.682 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/32 file after the snapshot 2023-08-25T09:53:44.689 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/33 file after the snapshot 2023-08-25T09:53:44.693 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/34 file after the snapshot 2023-08-25T09:53:44.699 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/35 file after the snapshot 2023-08-25T09:53:44.705 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/36 file after the snapshot 2023-08-25T09:53:44.712 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/37 file after the snapshot 2023-08-25T09:53:44.715 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/38 file after the snapshot 2023-08-25T09:53:44.721 INFO:tasks.workunit.client.0.smithi071.stdout: create 248/39 file after the snapshot 2023-08-25T09:53:44.725 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/.snap/snap-subdir-test 2023-08-25T09:53:44.765 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/30 file after the snapshot 2023-08-25T09:53:44.770 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/31 file after the snapshot 2023-08-25T09:53:44.778 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/32 file after the snapshot 2023-08-25T09:53:44.786 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/33 file after the snapshot 2023-08-25T09:53:44.792 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/34 file after the snapshot 2023-08-25T09:53:44.796 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/35 file after the snapshot 2023-08-25T09:53:44.799 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/36 file after the snapshot 2023-08-25T09:53:44.802 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/37 file after the snapshot 2023-08-25T09:53:44.805 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/38 file after the snapshot 2023-08-25T09:53:44.810 INFO:tasks.workunit.client.0.smithi071.stdout: create 249/39 file after the snapshot 2023-08-25T09:53:44.813 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/.snap/snap-subdir-test 2023-08-25T09:53:44.863 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/30 file after the snapshot 2023-08-25T09:53:44.866 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/31 file after the snapshot 2023-08-25T09:53:44.870 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/32 file after the snapshot 2023-08-25T09:53:44.875 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/33 file after the snapshot 2023-08-25T09:53:44.879 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/34 file after the snapshot 2023-08-25T09:53:44.882 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/35 file after the snapshot 2023-08-25T09:53:44.887 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/36 file after the snapshot 2023-08-25T09:53:44.894 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/37 file after the snapshot 2023-08-25T09:53:44.898 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/38 file after the snapshot 2023-08-25T09:53:44.902 INFO:tasks.workunit.client.0.smithi071.stdout: create 250/39 file after the snapshot 2023-08-25T09:53:44.905 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/.snap/snap-subdir-test 2023-08-25T09:53:45.624 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/30 file after the snapshot 2023-08-25T09:53:45.630 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/31 file after the snapshot 2023-08-25T09:53:45.636 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/32 file after the snapshot 2023-08-25T09:53:45.644 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/33 file after the snapshot 2023-08-25T09:53:45.649 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/34 file after the snapshot 2023-08-25T09:53:45.654 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/35 file after the snapshot 2023-08-25T09:53:45.660 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/36 file after the snapshot 2023-08-25T09:53:45.668 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/37 file after the snapshot 2023-08-25T09:53:45.676 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/38 file after the snapshot 2023-08-25T09:53:45.682 INFO:tasks.workunit.client.0.smithi071.stdout: create 251/39 file after the snapshot 2023-08-25T09:53:45.690 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/.snap/snap-subdir-test 2023-08-25T09:53:45.725 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/30 file after the snapshot 2023-08-25T09:53:45.730 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/31 file after the snapshot 2023-08-25T09:53:45.736 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/32 file after the snapshot 2023-08-25T09:53:45.743 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/33 file after the snapshot 2023-08-25T09:53:45.748 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/34 file after the snapshot 2023-08-25T09:53:45.753 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/35 file after the snapshot 2023-08-25T09:53:45.761 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/36 file after the snapshot 2023-08-25T09:53:45.767 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/37 file after the snapshot 2023-08-25T09:53:45.775 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/38 file after the snapshot 2023-08-25T09:53:45.807 INFO:tasks.workunit.client.0.smithi071.stdout: create 252/39 file after the snapshot 2023-08-25T09:53:45.813 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/.snap/snap-subdir-test 2023-08-25T09:53:45.867 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/30 file after the snapshot 2023-08-25T09:53:45.872 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/31 file after the snapshot 2023-08-25T09:53:45.878 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/32 file after the snapshot 2023-08-25T09:53:45.883 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/33 file after the snapshot 2023-08-25T09:53:45.890 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/34 file after the snapshot 2023-08-25T09:53:45.894 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/35 file after the snapshot 2023-08-25T09:53:45.901 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/36 file after the snapshot 2023-08-25T09:53:45.906 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/37 file after the snapshot 2023-08-25T09:53:45.913 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/38 file after the snapshot 2023-08-25T09:53:45.917 INFO:tasks.workunit.client.0.smithi071.stdout: create 253/39 file after the snapshot 2023-08-25T09:53:45.922 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/.snap/snap-subdir-test 2023-08-25T09:53:45.963 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/30 file after the snapshot 2023-08-25T09:53:45.968 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/31 file after the snapshot 2023-08-25T09:53:45.975 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/32 file after the snapshot 2023-08-25T09:53:45.982 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/33 file after the snapshot 2023-08-25T09:53:45.986 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/34 file after the snapshot 2023-08-25T09:53:45.991 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/35 file after the snapshot 2023-08-25T09:53:45.996 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/36 file after the snapshot 2023-08-25T09:53:46.002 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/37 file after the snapshot 2023-08-25T09:53:46.008 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/38 file after the snapshot 2023-08-25T09:53:46.015 INFO:tasks.workunit.client.0.smithi071.stdout: create 254/39 file after the snapshot 2023-08-25T09:53:46.020 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/.snap/snap-subdir-test 2023-08-25T09:53:46.055 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/30 file after the snapshot 2023-08-25T09:53:46.061 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/31 file after the snapshot 2023-08-25T09:53:46.067 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/32 file after the snapshot 2023-08-25T09:53:46.072 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/33 file after the snapshot 2023-08-25T09:53:46.078 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/34 file after the snapshot 2023-08-25T09:53:46.085 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/35 file after the snapshot 2023-08-25T09:53:46.091 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/36 file after the snapshot 2023-08-25T09:53:46.096 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/37 file after the snapshot 2023-08-25T09:53:46.102 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/38 file after the snapshot 2023-08-25T09:53:46.108 INFO:tasks.workunit.client.0.smithi071.stdout: create 255/39 file after the snapshot 2023-08-25T09:53:46.115 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/.snap/snap-subdir-test 2023-08-25T09:53:46.179 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/30 file after the snapshot 2023-08-25T09:53:46.183 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/31 file after the snapshot 2023-08-25T09:53:46.188 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/32 file after the snapshot 2023-08-25T09:53:46.194 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/33 file after the snapshot 2023-08-25T09:53:46.201 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/34 file after the snapshot 2023-08-25T09:53:46.207 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/35 file after the snapshot 2023-08-25T09:53:46.215 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/36 file after the snapshot 2023-08-25T09:53:46.222 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/37 file after the snapshot 2023-08-25T09:53:46.228 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/38 file after the snapshot 2023-08-25T09:53:46.235 INFO:tasks.workunit.client.0.smithi071.stdout: create 256/39 file after the snapshot 2023-08-25T09:53:46.241 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/.snap/snap-subdir-test 2023-08-25T09:53:46.274 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/30 file after the snapshot 2023-08-25T09:53:46.280 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/31 file after the snapshot 2023-08-25T09:53:46.285 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/32 file after the snapshot 2023-08-25T09:53:46.291 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/33 file after the snapshot 2023-08-25T09:53:46.299 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/34 file after the snapshot 2023-08-25T09:53:46.306 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/35 file after the snapshot 2023-08-25T09:53:46.313 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/36 file after the snapshot 2023-08-25T09:53:46.321 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/37 file after the snapshot 2023-08-25T09:53:46.329 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/38 file after the snapshot 2023-08-25T09:53:46.337 INFO:tasks.workunit.client.0.smithi071.stdout: create 257/39 file after the snapshot 2023-08-25T09:53:46.344 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/.snap/snap-subdir-test 2023-08-25T09:53:46.390 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/30 file after the snapshot 2023-08-25T09:53:46.395 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/31 file after the snapshot 2023-08-25T09:53:46.400 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/32 file after the snapshot 2023-08-25T09:53:46.405 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/33 file after the snapshot 2023-08-25T09:53:46.410 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/34 file after the snapshot 2023-08-25T09:53:46.415 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/35 file after the snapshot 2023-08-25T09:53:46.421 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/36 file after the snapshot 2023-08-25T09:53:46.425 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/37 file after the snapshot 2023-08-25T09:53:46.430 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/38 file after the snapshot 2023-08-25T09:53:46.435 INFO:tasks.workunit.client.0.smithi071.stdout: create 258/39 file after the snapshot 2023-08-25T09:53:46.439 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/.snap/snap-subdir-test 2023-08-25T09:53:46.483 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/30 file after the snapshot 2023-08-25T09:53:46.488 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/31 file after the snapshot 2023-08-25T09:53:46.493 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/32 file after the snapshot 2023-08-25T09:53:46.497 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/33 file after the snapshot 2023-08-25T09:53:46.503 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/34 file after the snapshot 2023-08-25T09:53:46.507 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/35 file after the snapshot 2023-08-25T09:53:46.512 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/36 file after the snapshot 2023-08-25T09:53:46.517 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/37 file after the snapshot 2023-08-25T09:53:46.523 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/38 file after the snapshot 2023-08-25T09:53:46.529 INFO:tasks.workunit.client.0.smithi071.stdout: create 259/39 file after the snapshot 2023-08-25T09:53:46.533 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/.snap/snap-subdir-test 2023-08-25T09:53:46.805 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/30 file after the snapshot 2023-08-25T09:53:46.809 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/31 file after the snapshot 2023-08-25T09:53:46.814 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/32 file after the snapshot 2023-08-25T09:53:46.819 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/33 file after the snapshot 2023-08-25T09:53:46.825 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/34 file after the snapshot 2023-08-25T09:53:46.832 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/35 file after the snapshot 2023-08-25T09:53:46.838 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/36 file after the snapshot 2023-08-25T09:53:46.845 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/37 file after the snapshot 2023-08-25T09:53:46.854 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/38 file after the snapshot 2023-08-25T09:53:46.862 INFO:tasks.workunit.client.0.smithi071.stdout: create 260/39 file after the snapshot 2023-08-25T09:53:46.868 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/.snap/snap-subdir-test 2023-08-25T09:53:47.764 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/30 file after the snapshot 2023-08-25T09:53:47.770 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/31 file after the snapshot 2023-08-25T09:53:47.776 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/32 file after the snapshot 2023-08-25T09:53:47.782 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/33 file after the snapshot 2023-08-25T09:53:47.789 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/34 file after the snapshot 2023-08-25T09:53:47.795 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/35 file after the snapshot 2023-08-25T09:53:47.820 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/36 file after the snapshot 2023-08-25T09:53:47.827 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/37 file after the snapshot 2023-08-25T09:53:47.836 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/38 file after the snapshot 2023-08-25T09:53:47.843 INFO:tasks.workunit.client.0.smithi071.stdout: create 261/39 file after the snapshot 2023-08-25T09:53:47.850 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/.snap/snap-subdir-test 2023-08-25T09:53:47.899 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/30 file after the snapshot 2023-08-25T09:53:47.904 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/31 file after the snapshot 2023-08-25T09:53:47.909 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/32 file after the snapshot 2023-08-25T09:53:47.915 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/33 file after the snapshot 2023-08-25T09:53:47.922 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/34 file after the snapshot 2023-08-25T09:53:47.926 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/35 file after the snapshot 2023-08-25T09:53:47.931 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/36 file after the snapshot 2023-08-25T09:53:47.936 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/37 file after the snapshot 2023-08-25T09:53:47.941 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/38 file after the snapshot 2023-08-25T09:53:47.945 INFO:tasks.workunit.client.0.smithi071.stdout: create 262/39 file after the snapshot 2023-08-25T09:53:47.949 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/.snap/snap-subdir-test 2023-08-25T09:53:47.980 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/30 file after the snapshot 2023-08-25T09:53:47.984 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/31 file after the snapshot 2023-08-25T09:53:47.988 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/32 file after the snapshot 2023-08-25T09:53:47.993 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/33 file after the snapshot 2023-08-25T09:53:47.997 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/34 file after the snapshot 2023-08-25T09:53:48.001 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/35 file after the snapshot 2023-08-25T09:53:48.005 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/36 file after the snapshot 2023-08-25T09:53:48.011 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/37 file after the snapshot 2023-08-25T09:53:48.016 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/38 file after the snapshot 2023-08-25T09:53:48.022 INFO:tasks.workunit.client.0.smithi071.stdout: create 263/39 file after the snapshot 2023-08-25T09:53:48.027 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/.snap/snap-subdir-test 2023-08-25T09:53:48.057 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/30 file after the snapshot 2023-08-25T09:53:48.061 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/31 file after the snapshot 2023-08-25T09:53:48.064 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/32 file after the snapshot 2023-08-25T09:53:48.069 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/33 file after the snapshot 2023-08-25T09:53:48.072 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/34 file after the snapshot 2023-08-25T09:53:48.077 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/35 file after the snapshot 2023-08-25T09:53:48.082 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/36 file after the snapshot 2023-08-25T09:53:48.086 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/37 file after the snapshot 2023-08-25T09:53:48.092 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/38 file after the snapshot 2023-08-25T09:53:48.095 INFO:tasks.workunit.client.0.smithi071.stdout: create 264/39 file after the snapshot 2023-08-25T09:53:48.099 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/.snap/snap-subdir-test 2023-08-25T09:53:48.127 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/30 file after the snapshot 2023-08-25T09:53:48.131 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/31 file after the snapshot 2023-08-25T09:53:48.134 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/32 file after the snapshot 2023-08-25T09:53:48.139 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/33 file after the snapshot 2023-08-25T09:53:48.143 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/34 file after the snapshot 2023-08-25T09:53:48.148 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/35 file after the snapshot 2023-08-25T09:53:48.151 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/36 file after the snapshot 2023-08-25T09:53:48.155 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/37 file after the snapshot 2023-08-25T09:53:48.159 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/38 file after the snapshot 2023-08-25T09:53:48.162 INFO:tasks.workunit.client.0.smithi071.stdout: create 265/39 file after the snapshot 2023-08-25T09:53:48.167 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/.snap/snap-subdir-test 2023-08-25T09:53:48.198 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/30 file after the snapshot 2023-08-25T09:53:48.203 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/31 file after the snapshot 2023-08-25T09:53:48.209 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/32 file after the snapshot 2023-08-25T09:53:48.214 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/33 file after the snapshot 2023-08-25T09:53:48.220 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/34 file after the snapshot 2023-08-25T09:53:48.225 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/35 file after the snapshot 2023-08-25T09:53:48.232 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/36 file after the snapshot 2023-08-25T09:53:48.240 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/37 file after the snapshot 2023-08-25T09:53:48.243 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/38 file after the snapshot 2023-08-25T09:53:48.249 INFO:tasks.workunit.client.0.smithi071.stdout: create 266/39 file after the snapshot 2023-08-25T09:53:48.255 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/.snap/snap-subdir-test 2023-08-25T09:53:48.305 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/30 file after the snapshot 2023-08-25T09:53:48.309 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/31 file after the snapshot 2023-08-25T09:53:48.314 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/32 file after the snapshot 2023-08-25T09:53:48.320 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/33 file after the snapshot 2023-08-25T09:53:48.324 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/34 file after the snapshot 2023-08-25T09:53:48.330 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/35 file after the snapshot 2023-08-25T09:53:48.335 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/36 file after the snapshot 2023-08-25T09:53:48.339 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/37 file after the snapshot 2023-08-25T09:53:48.344 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/38 file after the snapshot 2023-08-25T09:53:48.347 INFO:tasks.workunit.client.0.smithi071.stdout: create 267/39 file after the snapshot 2023-08-25T09:53:48.353 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/.snap/snap-subdir-test 2023-08-25T09:53:48.391 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/30 file after the snapshot 2023-08-25T09:53:48.395 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/31 file after the snapshot 2023-08-25T09:53:48.400 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/32 file after the snapshot 2023-08-25T09:53:48.406 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/33 file after the snapshot 2023-08-25T09:53:48.411 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/34 file after the snapshot 2023-08-25T09:53:48.416 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/35 file after the snapshot 2023-08-25T09:53:48.422 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/36 file after the snapshot 2023-08-25T09:53:48.428 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/37 file after the snapshot 2023-08-25T09:53:48.431 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/38 file after the snapshot 2023-08-25T09:53:48.436 INFO:tasks.workunit.client.0.smithi071.stdout: create 268/39 file after the snapshot 2023-08-25T09:53:48.440 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/.snap/snap-subdir-test 2023-08-25T09:53:48.484 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/30 file after the snapshot 2023-08-25T09:53:48.490 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/31 file after the snapshot 2023-08-25T09:53:48.496 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/32 file after the snapshot 2023-08-25T09:53:48.502 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/33 file after the snapshot 2023-08-25T09:53:48.508 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/34 file after the snapshot 2023-08-25T09:53:48.513 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/35 file after the snapshot 2023-08-25T09:53:48.520 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/36 file after the snapshot 2023-08-25T09:53:48.525 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/37 file after the snapshot 2023-08-25T09:53:48.530 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/38 file after the snapshot 2023-08-25T09:53:48.535 INFO:tasks.workunit.client.0.smithi071.stdout: create 269/39 file after the snapshot 2023-08-25T09:53:48.541 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/.snap/snap-subdir-test 2023-08-25T09:53:48.577 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/30 file after the snapshot 2023-08-25T09:53:48.583 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/31 file after the snapshot 2023-08-25T09:53:48.589 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/32 file after the snapshot 2023-08-25T09:53:48.595 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/33 file after the snapshot 2023-08-25T09:53:48.600 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/34 file after the snapshot 2023-08-25T09:53:48.605 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/35 file after the snapshot 2023-08-25T09:53:48.610 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/36 file after the snapshot 2023-08-25T09:53:48.614 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/37 file after the snapshot 2023-08-25T09:53:48.617 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/38 file after the snapshot 2023-08-25T09:53:48.623 INFO:tasks.workunit.client.0.smithi071.stdout: create 270/39 file after the snapshot 2023-08-25T09:53:48.629 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/.snap/snap-subdir-test 2023-08-25T09:53:48.661 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/30 file after the snapshot 2023-08-25T09:53:48.667 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/31 file after the snapshot 2023-08-25T09:53:48.674 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/32 file after the snapshot 2023-08-25T09:53:48.678 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/33 file after the snapshot 2023-08-25T09:53:48.683 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/34 file after the snapshot 2023-08-25T09:53:48.688 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/35 file after the snapshot 2023-08-25T09:53:48.692 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/36 file after the snapshot 2023-08-25T09:53:48.697 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/37 file after the snapshot 2023-08-25T09:53:48.703 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/38 file after the snapshot 2023-08-25T09:53:48.709 INFO:tasks.workunit.client.0.smithi071.stdout: create 271/39 file after the snapshot 2023-08-25T09:53:48.715 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/.snap/snap-subdir-test 2023-08-25T09:53:48.759 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/30 file after the snapshot 2023-08-25T09:53:48.772 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/31 file after the snapshot 2023-08-25T09:53:48.794 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/32 file after the snapshot 2023-08-25T09:53:48.819 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/33 file after the snapshot 2023-08-25T09:53:48.839 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/34 file after the snapshot 2023-08-25T09:53:48.852 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/35 file after the snapshot 2023-08-25T09:53:48.862 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/36 file after the snapshot 2023-08-25T09:53:48.873 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/37 file after the snapshot 2023-08-25T09:53:48.885 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/38 file after the snapshot 2023-08-25T09:53:48.896 INFO:tasks.workunit.client.0.smithi071.stdout: create 272/39 file after the snapshot 2023-08-25T09:53:48.908 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/.snap/snap-subdir-test 2023-08-25T09:53:49.804 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/30 file after the snapshot 2023-08-25T09:53:49.811 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/31 file after the snapshot 2023-08-25T09:53:49.825 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/32 file after the snapshot 2023-08-25T09:53:49.854 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/33 file after the snapshot 2023-08-25T09:53:49.861 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/34 file after the snapshot 2023-08-25T09:53:49.865 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/35 file after the snapshot 2023-08-25T09:53:49.869 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/36 file after the snapshot 2023-08-25T09:53:49.875 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/37 file after the snapshot 2023-08-25T09:53:49.880 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/38 file after the snapshot 2023-08-25T09:53:49.885 INFO:tasks.workunit.client.0.smithi071.stdout: create 273/39 file after the snapshot 2023-08-25T09:53:49.889 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/.snap/snap-subdir-test 2023-08-25T09:53:49.926 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/30 file after the snapshot 2023-08-25T09:53:49.929 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/31 file after the snapshot 2023-08-25T09:53:49.933 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/32 file after the snapshot 2023-08-25T09:53:49.936 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/33 file after the snapshot 2023-08-25T09:53:49.940 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/34 file after the snapshot 2023-08-25T09:53:49.945 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/35 file after the snapshot 2023-08-25T09:53:49.948 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/36 file after the snapshot 2023-08-25T09:53:49.950 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/37 file after the snapshot 2023-08-25T09:53:49.953 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/38 file after the snapshot 2023-08-25T09:53:49.956 INFO:tasks.workunit.client.0.smithi071.stdout: create 274/39 file after the snapshot 2023-08-25T09:53:49.959 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/.snap/snap-subdir-test 2023-08-25T09:53:50.398 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/30 file after the snapshot 2023-08-25T09:53:50.405 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/31 file after the snapshot 2023-08-25T09:53:50.411 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/32 file after the snapshot 2023-08-25T09:53:50.418 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/33 file after the snapshot 2023-08-25T09:53:50.425 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/34 file after the snapshot 2023-08-25T09:53:50.431 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/35 file after the snapshot 2023-08-25T09:53:50.439 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/36 file after the snapshot 2023-08-25T09:53:50.445 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/37 file after the snapshot 2023-08-25T09:53:50.451 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/38 file after the snapshot 2023-08-25T09:53:50.459 INFO:tasks.workunit.client.0.smithi071.stdout: create 275/39 file after the snapshot 2023-08-25T09:53:50.465 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/.snap/snap-subdir-test 2023-08-25T09:53:50.498 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/30 file after the snapshot 2023-08-25T09:53:50.505 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/31 file after the snapshot 2023-08-25T09:53:50.510 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/32 file after the snapshot 2023-08-25T09:53:50.517 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/33 file after the snapshot 2023-08-25T09:53:50.522 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/34 file after the snapshot 2023-08-25T09:53:50.527 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/35 file after the snapshot 2023-08-25T09:53:50.533 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/36 file after the snapshot 2023-08-25T09:53:50.538 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/37 file after the snapshot 2023-08-25T09:53:50.543 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/38 file after the snapshot 2023-08-25T09:53:50.546 INFO:tasks.workunit.client.0.smithi071.stdout: create 276/39 file after the snapshot 2023-08-25T09:53:50.551 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/.snap/snap-subdir-test 2023-08-25T09:53:50.837 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/30 file after the snapshot 2023-08-25T09:53:51.592 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/31 file after the snapshot 2023-08-25T09:53:51.602 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/32 file after the snapshot 2023-08-25T09:53:51.614 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/33 file after the snapshot 2023-08-25T09:53:51.624 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/34 file after the snapshot 2023-08-25T09:53:51.634 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/35 file after the snapshot 2023-08-25T09:53:51.644 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/36 file after the snapshot 2023-08-25T09:53:51.653 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/37 file after the snapshot 2023-08-25T09:53:51.664 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/38 file after the snapshot 2023-08-25T09:53:51.674 INFO:tasks.workunit.client.0.smithi071.stdout: create 277/39 file after the snapshot 2023-08-25T09:53:51.684 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/.snap/snap-subdir-test 2023-08-25T09:53:51.732 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/30 file after the snapshot 2023-08-25T09:53:51.740 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/31 file after the snapshot 2023-08-25T09:53:51.748 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/32 file after the snapshot 2023-08-25T09:53:51.756 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/33 file after the snapshot 2023-08-25T09:53:51.763 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/34 file after the snapshot 2023-08-25T09:53:51.770 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/35 file after the snapshot 2023-08-25T09:53:51.776 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/36 file after the snapshot 2023-08-25T09:53:51.782 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/37 file after the snapshot 2023-08-25T09:53:51.790 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/38 file after the snapshot 2023-08-25T09:53:51.795 INFO:tasks.workunit.client.0.smithi071.stdout: create 278/39 file after the snapshot 2023-08-25T09:53:51.802 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/.snap/snap-subdir-test 2023-08-25T09:53:51.861 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/30 file after the snapshot 2023-08-25T09:53:51.864 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/31 file after the snapshot 2023-08-25T09:53:51.869 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/32 file after the snapshot 2023-08-25T09:53:51.873 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/33 file after the snapshot 2023-08-25T09:53:51.877 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/34 file after the snapshot 2023-08-25T09:53:51.881 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/35 file after the snapshot 2023-08-25T09:53:51.884 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/36 file after the snapshot 2023-08-25T09:53:51.888 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/37 file after the snapshot 2023-08-25T09:53:51.892 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/38 file after the snapshot 2023-08-25T09:53:51.895 INFO:tasks.workunit.client.0.smithi071.stdout: create 279/39 file after the snapshot 2023-08-25T09:53:51.898 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/.snap/snap-subdir-test 2023-08-25T09:53:51.951 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/30 file after the snapshot 2023-08-25T09:53:51.961 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/31 file after the snapshot 2023-08-25T09:53:51.971 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/32 file after the snapshot 2023-08-25T09:53:51.979 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/33 file after the snapshot 2023-08-25T09:53:51.988 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/34 file after the snapshot 2023-08-25T09:53:51.998 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/35 file after the snapshot 2023-08-25T09:53:52.009 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/36 file after the snapshot 2023-08-25T09:53:52.019 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/37 file after the snapshot 2023-08-25T09:53:52.029 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/38 file after the snapshot 2023-08-25T09:53:52.038 INFO:tasks.workunit.client.0.smithi071.stdout: create 280/39 file after the snapshot 2023-08-25T09:53:52.049 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/.snap/snap-subdir-test 2023-08-25T09:53:52.086 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/30 file after the snapshot 2023-08-25T09:53:52.589 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/31 file after the snapshot 2023-08-25T09:53:52.597 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/32 file after the snapshot 2023-08-25T09:53:52.605 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/33 file after the snapshot 2023-08-25T09:53:52.611 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/34 file after the snapshot 2023-08-25T09:53:52.617 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/35 file after the snapshot 2023-08-25T09:53:52.625 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/36 file after the snapshot 2023-08-25T09:53:52.632 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/37 file after the snapshot 2023-08-25T09:53:52.640 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/38 file after the snapshot 2023-08-25T09:53:52.647 INFO:tasks.workunit.client.0.smithi071.stdout: create 281/39 file after the snapshot 2023-08-25T09:53:52.655 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/.snap/snap-subdir-test 2023-08-25T09:53:52.699 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/30 file after the snapshot 2023-08-25T09:53:52.704 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/31 file after the snapshot 2023-08-25T09:53:52.710 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/32 file after the snapshot 2023-08-25T09:53:52.716 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/33 file after the snapshot 2023-08-25T09:53:52.722 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/34 file after the snapshot 2023-08-25T09:53:52.731 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/35 file after the snapshot 2023-08-25T09:53:52.739 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/36 file after the snapshot 2023-08-25T09:53:52.748 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/37 file after the snapshot 2023-08-25T09:53:52.756 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/38 file after the snapshot 2023-08-25T09:53:52.762 INFO:tasks.workunit.client.0.smithi071.stdout: create 282/39 file after the snapshot 2023-08-25T09:53:52.769 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/.snap/snap-subdir-test 2023-08-25T09:53:52.808 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/30 file after the snapshot 2023-08-25T09:53:52.812 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/31 file after the snapshot 2023-08-25T09:53:52.819 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/32 file after the snapshot 2023-08-25T09:53:52.825 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/33 file after the snapshot 2023-08-25T09:53:52.830 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/34 file after the snapshot 2023-08-25T09:53:52.836 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/35 file after the snapshot 2023-08-25T09:53:52.848 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/36 file after the snapshot 2023-08-25T09:53:52.853 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/37 file after the snapshot 2023-08-25T09:53:52.860 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/38 file after the snapshot 2023-08-25T09:53:52.863 INFO:tasks.workunit.client.0.smithi071.stdout: create 283/39 file after the snapshot 2023-08-25T09:53:52.867 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/.snap/snap-subdir-test 2023-08-25T09:53:52.916 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/30 file after the snapshot 2023-08-25T09:53:52.929 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/31 file after the snapshot 2023-08-25T09:53:52.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:53:52.943 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:53:52.947 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/32 file after the snapshot 2023-08-25T09:53:52.957 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/33 file after the snapshot 2023-08-25T09:53:52.968 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/34 file after the snapshot 2023-08-25T09:53:52.975 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/35 file after the snapshot 2023-08-25T09:53:52.978 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/36 file after the snapshot 2023-08-25T09:53:52.984 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/37 file after the snapshot 2023-08-25T09:53:52.989 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/38 file after the snapshot 2023-08-25T09:53:52.994 INFO:tasks.workunit.client.0.smithi071.stdout: create 284/39 file after the snapshot 2023-08-25T09:53:52.999 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/.snap/snap-subdir-test 2023-08-25T09:53:53.021 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/30 file after the snapshot 2023-08-25T09:53:53.024 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/31 file after the snapshot 2023-08-25T09:53:53.027 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/32 file after the snapshot 2023-08-25T09:53:53.029 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/33 file after the snapshot 2023-08-25T09:53:53.032 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/34 file after the snapshot 2023-08-25T09:53:53.034 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/35 file after the snapshot 2023-08-25T09:53:53.037 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/36 file after the snapshot 2023-08-25T09:53:53.040 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/37 file after the snapshot 2023-08-25T09:53:53.043 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/38 file after the snapshot 2023-08-25T09:53:53.046 INFO:tasks.workunit.client.0.smithi071.stdout: create 285/39 file after the snapshot 2023-08-25T09:53:53.049 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/.snap/snap-subdir-test 2023-08-25T09:53:53.071 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/30 file after the snapshot 2023-08-25T09:53:53.072 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/31 file after the snapshot 2023-08-25T09:53:53.075 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/32 file after the snapshot 2023-08-25T09:53:53.077 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/33 file after the snapshot 2023-08-25T09:53:53.080 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/34 file after the snapshot 2023-08-25T09:53:53.083 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/35 file after the snapshot 2023-08-25T09:53:53.085 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/36 file after the snapshot 2023-08-25T09:53:53.088 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/37 file after the snapshot 2023-08-25T09:53:53.091 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/38 file after the snapshot 2023-08-25T09:53:53.094 INFO:tasks.workunit.client.0.smithi071.stdout: create 286/39 file after the snapshot 2023-08-25T09:53:53.096 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/.snap/snap-subdir-test 2023-08-25T09:53:53.121 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/30 file after the snapshot 2023-08-25T09:53:53.124 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/31 file after the snapshot 2023-08-25T09:53:53.126 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/32 file after the snapshot 2023-08-25T09:53:53.129 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/33 file after the snapshot 2023-08-25T09:53:53.132 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/34 file after the snapshot 2023-08-25T09:53:53.135 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/35 file after the snapshot 2023-08-25T09:53:53.138 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/36 file after the snapshot 2023-08-25T09:53:53.141 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/37 file after the snapshot 2023-08-25T09:53:53.144 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/38 file after the snapshot 2023-08-25T09:53:53.146 INFO:tasks.workunit.client.0.smithi071.stdout: create 287/39 file after the snapshot 2023-08-25T09:53:53.150 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/.snap/snap-subdir-test 2023-08-25T09:53:53.178 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/30 file after the snapshot 2023-08-25T09:53:53.181 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/31 file after the snapshot 2023-08-25T09:53:53.183 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/32 file after the snapshot 2023-08-25T09:53:53.186 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/33 file after the snapshot 2023-08-25T09:53:53.189 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/34 file after the snapshot 2023-08-25T09:53:53.192 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/35 file after the snapshot 2023-08-25T09:53:53.195 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/36 file after the snapshot 2023-08-25T09:53:53.198 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/37 file after the snapshot 2023-08-25T09:53:53.201 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/38 file after the snapshot 2023-08-25T09:53:53.204 INFO:tasks.workunit.client.0.smithi071.stdout: create 288/39 file after the snapshot 2023-08-25T09:53:53.207 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/.snap/snap-subdir-test 2023-08-25T09:53:53.236 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/30 file after the snapshot 2023-08-25T09:53:53.238 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/31 file after the snapshot 2023-08-25T09:53:53.241 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/32 file after the snapshot 2023-08-25T09:53:53.244 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/33 file after the snapshot 2023-08-25T09:53:53.247 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/34 file after the snapshot 2023-08-25T09:53:53.251 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/35 file after the snapshot 2023-08-25T09:53:53.254 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/36 file after the snapshot 2023-08-25T09:53:53.257 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/37 file after the snapshot 2023-08-25T09:53:53.259 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/38 file after the snapshot 2023-08-25T09:53:53.262 INFO:tasks.workunit.client.0.smithi071.stdout: create 289/39 file after the snapshot 2023-08-25T09:53:53.268 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/.snap/snap-subdir-test 2023-08-25T09:53:53.296 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/30 file after the snapshot 2023-08-25T09:53:53.300 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/31 file after the snapshot 2023-08-25T09:53:53.304 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/32 file after the snapshot 2023-08-25T09:53:53.308 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/33 file after the snapshot 2023-08-25T09:53:53.312 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/34 file after the snapshot 2023-08-25T09:53:53.316 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/35 file after the snapshot 2023-08-25T09:53:53.320 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/36 file after the snapshot 2023-08-25T09:53:53.323 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/37 file after the snapshot 2023-08-25T09:53:53.326 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/38 file after the snapshot 2023-08-25T09:53:53.330 INFO:tasks.workunit.client.0.smithi071.stdout: create 290/39 file after the snapshot 2023-08-25T09:53:53.334 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/.snap/snap-subdir-test 2023-08-25T09:53:53.362 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/30 file after the snapshot 2023-08-25T09:53:53.365 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/31 file after the snapshot 2023-08-25T09:53:53.368 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/32 file after the snapshot 2023-08-25T09:53:53.373 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/33 file after the snapshot 2023-08-25T09:53:53.376 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/34 file after the snapshot 2023-08-25T09:53:53.379 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/35 file after the snapshot 2023-08-25T09:53:53.382 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/36 file after the snapshot 2023-08-25T09:53:53.384 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/37 file after the snapshot 2023-08-25T09:53:53.389 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/38 file after the snapshot 2023-08-25T09:53:53.393 INFO:tasks.workunit.client.0.smithi071.stdout: create 291/39 file after the snapshot 2023-08-25T09:53:53.396 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/.snap/snap-subdir-test 2023-08-25T09:53:53.426 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/30 file after the snapshot 2023-08-25T09:53:53.429 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/31 file after the snapshot 2023-08-25T09:53:53.432 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/32 file after the snapshot 2023-08-25T09:53:53.435 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/33 file after the snapshot 2023-08-25T09:53:53.438 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/34 file after the snapshot 2023-08-25T09:53:53.442 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/35 file after the snapshot 2023-08-25T09:53:53.445 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/36 file after the snapshot 2023-08-25T09:53:53.449 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/37 file after the snapshot 2023-08-25T09:53:53.453 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/38 file after the snapshot 2023-08-25T09:53:53.456 INFO:tasks.workunit.client.0.smithi071.stdout: create 292/39 file after the snapshot 2023-08-25T09:53:53.459 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/.snap/snap-subdir-test 2023-08-25T09:53:53.487 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/30 file after the snapshot 2023-08-25T09:53:53.491 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/31 file after the snapshot 2023-08-25T09:53:53.494 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/32 file after the snapshot 2023-08-25T09:53:53.497 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/33 file after the snapshot 2023-08-25T09:53:53.500 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/34 file after the snapshot 2023-08-25T09:53:53.504 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/35 file after the snapshot 2023-08-25T09:53:53.507 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/36 file after the snapshot 2023-08-25T09:53:53.512 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/37 file after the snapshot 2023-08-25T09:53:53.514 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/38 file after the snapshot 2023-08-25T09:53:53.517 INFO:tasks.workunit.client.0.smithi071.stdout: create 293/39 file after the snapshot 2023-08-25T09:53:53.520 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/.snap/snap-subdir-test 2023-08-25T09:53:53.552 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/30 file after the snapshot 2023-08-25T09:53:53.554 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/31 file after the snapshot 2023-08-25T09:53:53.557 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/32 file after the snapshot 2023-08-25T09:53:53.559 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/33 file after the snapshot 2023-08-25T09:53:53.563 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/34 file after the snapshot 2023-08-25T09:53:53.566 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/35 file after the snapshot 2023-08-25T09:53:53.570 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/36 file after the snapshot 2023-08-25T09:53:53.573 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/37 file after the snapshot 2023-08-25T09:53:53.576 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/38 file after the snapshot 2023-08-25T09:53:53.578 INFO:tasks.workunit.client.0.smithi071.stdout: create 294/39 file after the snapshot 2023-08-25T09:53:53.581 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/.snap/snap-subdir-test 2023-08-25T09:53:53.618 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/30 file after the snapshot 2023-08-25T09:53:53.627 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/31 file after the snapshot 2023-08-25T09:53:53.634 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/32 file after the snapshot 2023-08-25T09:53:53.642 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/33 file after the snapshot 2023-08-25T09:53:53.650 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/34 file after the snapshot 2023-08-25T09:53:53.660 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/35 file after the snapshot 2023-08-25T09:53:53.668 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/36 file after the snapshot 2023-08-25T09:53:54.315 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/37 file after the snapshot 2023-08-25T09:53:54.325 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/38 file after the snapshot 2023-08-25T09:53:54.336 INFO:tasks.workunit.client.0.smithi071.stdout: create 295/39 file after the snapshot 2023-08-25T09:53:54.347 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/.snap/snap-subdir-test 2023-08-25T09:53:54.402 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/30 file after the snapshot 2023-08-25T09:53:54.408 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/31 file after the snapshot 2023-08-25T09:53:54.415 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/32 file after the snapshot 2023-08-25T09:53:54.422 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/33 file after the snapshot 2023-08-25T09:53:54.429 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/34 file after the snapshot 2023-08-25T09:53:54.437 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/35 file after the snapshot 2023-08-25T09:53:54.444 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/36 file after the snapshot 2023-08-25T09:53:54.453 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/37 file after the snapshot 2023-08-25T09:53:54.459 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/38 file after the snapshot 2023-08-25T09:53:54.465 INFO:tasks.workunit.client.0.smithi071.stdout: create 296/39 file after the snapshot 2023-08-25T09:53:54.472 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/.snap/snap-subdir-test 2023-08-25T09:53:54.519 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/30 file after the snapshot 2023-08-25T09:53:54.524 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/31 file after the snapshot 2023-08-25T09:53:54.530 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/32 file after the snapshot 2023-08-25T09:53:54.535 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/33 file after the snapshot 2023-08-25T09:53:54.540 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/34 file after the snapshot 2023-08-25T09:53:54.545 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/35 file after the snapshot 2023-08-25T09:53:54.550 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/36 file after the snapshot 2023-08-25T09:53:54.555 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/37 file after the snapshot 2023-08-25T09:53:54.560 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/38 file after the snapshot 2023-08-25T09:53:54.566 INFO:tasks.workunit.client.0.smithi071.stdout: create 297/39 file after the snapshot 2023-08-25T09:53:54.570 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/.snap/snap-subdir-test 2023-08-25T09:53:54.609 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/30 file after the snapshot 2023-08-25T09:53:54.617 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/31 file after the snapshot 2023-08-25T09:53:54.623 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/32 file after the snapshot 2023-08-25T09:53:54.631 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/33 file after the snapshot 2023-08-25T09:53:54.637 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/34 file after the snapshot 2023-08-25T09:53:54.644 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/35 file after the snapshot 2023-08-25T09:53:54.652 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/36 file after the snapshot 2023-08-25T09:53:54.658 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/37 file after the snapshot 2023-08-25T09:53:54.664 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/38 file after the snapshot 2023-08-25T09:53:54.669 INFO:tasks.workunit.client.0.smithi071.stdout: create 298/39 file after the snapshot 2023-08-25T09:53:54.674 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/.snap/snap-subdir-test 2023-08-25T09:53:54.719 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/30 file after the snapshot 2023-08-25T09:53:54.724 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/31 file after the snapshot 2023-08-25T09:53:54.729 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/32 file after the snapshot 2023-08-25T09:53:54.734 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/33 file after the snapshot 2023-08-25T09:53:54.739 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/34 file after the snapshot 2023-08-25T09:53:54.743 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/35 file after the snapshot 2023-08-25T09:53:54.749 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/36 file after the snapshot 2023-08-25T09:53:54.754 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/37 file after the snapshot 2023-08-25T09:53:54.759 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/38 file after the snapshot 2023-08-25T09:53:54.764 INFO:tasks.workunit.client.0.smithi071.stdout: create 299/39 file after the snapshot 2023-08-25T09:53:54.768 INFO:tasks.workunit.client.0.smithi071.stdout:Take second snapshot .snap/test2 2023-08-25T09:53:55.484 INFO:tasks.workunit.client.0.smithi071.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-08-25T09:54:00.395 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-08-25T09:54:00.395 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-08-25T09:54:00.396 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-08-25T09:54:00.396 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-08-25T09:54:00.396 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-08-25T09:54:00.396 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-08-25T09:54:00.396 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-08-25T09:54:00.396 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-08-25T09:54:00.397 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-08-25T09:54:00.398 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-08-25T09:54:00.399 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-08-25T09:54:00.400 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-08-25T09:54:00.400 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-08-25T09:54:00.400 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-08-25T09:54:00.400 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-08-25T09:54:00.400 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-08-25T09:54:00.400 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-08-25T09:54:00.401 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-08-25T09:54:00.402 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-08-25T09:54:00.403 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-08-25T09:54:00.404 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-08-25T09:54:00.405 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-08-25T09:54:00.405 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-08-25T09:54:00.405 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-08-25T09:54:00.405 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-08-25T09:54:00.405 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-08-25T09:54:00.405 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-08-25T09:54:00.406 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-08-25T09:54:00.406 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-08-25T09:54:00.406 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-08-25T09:54:00.406 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-08-25T09:54:00.406 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-08-25T09:54:03.249 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-08-25T09:54:03.250 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-08-25T09:54:03.250 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-08-25T09:54:03.250 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-08-25T09:54:03.250 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-08-25T09:54:03.250 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-08-25T09:54:03.251 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-08-25T09:54:03.251 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-08-25T09:54:03.251 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-08-25T09:54:03.251 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-08-25T09:54:03.251 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-08-25T09:54:03.251 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-08-25T09:54:03.252 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-08-25T09:54:03.252 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-08-25T09:54:03.252 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-08-25T09:54:03.252 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-08-25T09:54:03.252 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-08-25T09:54:03.253 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-08-25T09:54:03.253 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-08-25T09:54:03.253 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-08-25T09:54:03.253 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-08-25T09:54:03.253 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-08-25T09:54:03.254 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-08-25T09:54:03.254 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-08-25T09:54:03.254 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-08-25T09:54:03.254 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-08-25T09:54:03.254 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-08-25T09:54:03.254 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-08-25T09:54:03.255 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-08-25T09:54:03.255 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-08-25T09:54:03.255 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-08-25T09:54:03.255 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-08-25T09:54:03.255 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-08-25T09:54:03.256 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-08-25T09:54:03.256 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-08-25T09:54:03.256 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-08-25T09:54:03.256 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-08-25T09:54:03.256 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-08-25T09:54:03.257 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-08-25T09:54:03.257 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-08-25T09:54:03.257 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-08-25T09:54:03.257 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-08-25T09:54:03.257 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-08-25T09:54:03.258 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-08-25T09:54:03.258 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-08-25T09:54:03.258 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-08-25T09:54:03.258 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-08-25T09:54:03.258 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-08-25T09:54:03.258 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-08-25T09:54:03.259 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-08-25T09:54:03.259 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-08-25T09:54:03.259 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-08-25T09:54:03.259 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-08-25T09:54:03.259 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-08-25T09:54:03.260 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-08-25T09:54:03.260 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-08-25T09:54:03.260 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-08-25T09:54:03.260 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-08-25T09:54:03.260 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-08-25T09:54:03.261 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-08-25T09:54:03.261 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-08-25T09:54:03.261 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-08-25T09:54:03.261 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-08-25T09:54:03.261 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-08-25T09:54:03.261 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-08-25T09:54:03.262 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-08-25T09:54:03.262 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-08-25T09:54:03.262 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-08-25T09:54:03.262 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-08-25T09:54:03.262 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-08-25T09:54:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-08-25T09:54:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-08-25T09:54:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-08-25T09:54:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-08-25T09:54:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-08-25T09:54:03.263 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-08-25T09:54:03.264 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-08-25T09:54:03.264 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-08-25T09:54:03.264 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-08-25T09:54:03.264 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-08-25T09:54:03.264 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-08-25T09:54:07.048 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-08-25T09:54:07.048 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-08-25T09:54:07.048 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-08-25T09:54:07.048 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-08-25T09:54:07.049 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-08-25T09:54:07.050 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-08-25T09:54:07.050 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-08-25T09:54:07.050 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-08-25T09:54:07.050 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-08-25T09:54:07.050 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-08-25T09:54:07.050 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-08-25T09:54:07.051 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-08-25T09:54:07.052 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-08-25T09:54:07.053 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-08-25T09:54:07.053 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-08-25T09:54:07.053 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-08-25T09:54:07.053 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-08-25T09:54:07.053 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-08-25T09:54:07.053 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-08-25T09:54:07.054 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-08-25T09:54:07.055 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-08-25T09:54:07.056 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-08-25T09:54:07.056 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-08-25T09:54:07.056 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-08-25T09:54:07.056 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-08-25T09:54:07.056 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-08-25T09:54:07.056 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-08-25T09:54:07.057 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-08-25T09:54:07.058 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-08-25T09:54:07.058 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-08-25T09:54:07.058 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-08-25T09:54:07.058 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-08-25T09:54:07.058 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-08-25T09:54:07.058 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-08-25T09:54:07.059 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-08-25T09:54:07.059 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-08-25T09:54:07.059 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-08-25T09:54:07.059 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-08-25T09:54:07.059 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-08-25T09:54:07.059 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-08-25T09:54:07.060 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-08-25T09:54:07.060 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-08-25T09:54:09.808 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-08-25T09:54:09.809 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-08-25T09:54:09.809 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-08-25T09:54:09.809 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-08-25T09:54:09.810 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-08-25T09:54:09.810 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-08-25T09:54:09.810 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-08-25T09:54:09.810 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-08-25T09:54:09.811 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-08-25T09:54:09.811 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-08-25T09:54:09.811 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-08-25T09:54:09.811 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-08-25T09:54:09.812 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-08-25T09:54:09.812 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-08-25T09:54:09.812 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-08-25T09:54:09.812 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-08-25T09:54:09.813 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-08-25T09:54:09.813 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-08-25T09:54:09.813 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-08-25T09:54:09.814 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-08-25T09:54:09.814 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-08-25T09:54:09.814 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-08-25T09:54:09.814 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-08-25T09:54:09.815 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-08-25T09:54:09.816 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-08-25T09:54:09.816 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-08-25T09:54:09.816 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-08-25T09:54:09.816 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-08-25T09:54:09.816 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-08-25T09:54:09.816 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-08-25T09:54:09.817 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-08-25T09:54:09.817 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-08-25T09:54:09.817 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-08-25T09:54:09.817 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-08-25T09:54:09.817 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-08-25T09:54:09.817 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-08-25T09:54:09.818 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-08-25T09:54:09.818 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-08-25T09:54:09.818 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-08-25T09:54:09.818 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-08-25T09:54:09.819 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-08-25T09:54:09.819 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-08-25T09:54:09.819 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-08-25T09:54:09.819 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-08-25T09:54:09.820 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-08-25T09:54:09.820 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-08-25T09:54:09.820 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-08-25T09:54:09.820 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-08-25T09:54:09.821 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-08-25T09:54:09.821 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-08-25T09:54:09.821 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-08-25T09:54:09.822 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-08-25T09:54:09.822 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-08-25T09:54:09.822 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-08-25T09:54:09.823 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-08-25T09:54:09.823 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-08-25T09:54:09.823 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-08-25T09:54:09.823 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-08-25T09:54:09.824 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-08-25T09:54:09.824 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-08-25T09:54:09.824 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-08-25T09:54:09.825 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-08-25T09:54:09.825 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-08-25T09:54:09.825 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-08-25T09:54:09.825 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-08-25T09:54:09.825 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-08-25T09:54:09.826 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-08-25T09:54:09.826 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-08-25T09:54:09.826 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-08-25T09:54:09.826 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-08-25T09:54:09.826 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-08-25T09:54:09.827 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-08-25T09:54:09.827 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-08-25T09:54:09.827 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-08-25T09:54:09.828 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-08-25T09:54:13.330 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-08-25T09:54:13.330 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-08-25T09:54:13.331 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-08-25T09:54:13.332 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-08-25T09:54:13.333 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-08-25T09:54:13.334 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-08-25T09:54:13.335 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-08-25T09:54:13.335 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-08-25T09:54:13.335 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-08-25T09:54:13.335 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-08-25T09:54:13.335 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-08-25T09:54:13.335 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-08-25T09:54:13.336 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-08-25T09:54:13.337 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-08-25T09:54:13.338 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-08-25T09:54:13.339 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-08-25T09:54:13.340 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-08-25T09:54:13.340 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-08-25T09:54:13.340 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-08-25T09:54:13.340 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-08-25T09:54:13.340 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-08-25T09:54:13.341 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-08-25T09:54:13.342 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-08-25T09:54:13.342 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-08-25T09:54:13.342 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-08-25T09:54:13.342 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-08-25T09:54:13.342 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-08-25T09:54:18.625 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-08-25T09:54:18.625 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-08-25T09:54:18.625 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-08-25T09:54:18.625 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-08-25T09:54:18.626 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-08-25T09:54:18.627 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-08-25T09:54:18.627 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-08-25T09:54:18.627 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-08-25T09:54:18.627 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-08-25T09:54:18.627 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-08-25T09:54:18.628 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-08-25T09:54:18.628 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-08-25T09:54:18.628 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-08-25T09:54:18.628 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-08-25T09:54:18.628 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-08-25T09:54:18.629 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-08-25T09:54:18.630 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-08-25T09:54:18.631 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-08-25T09:54:18.632 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-08-25T09:54:18.632 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-08-25T09:54:18.632 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-08-25T09:54:18.632 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-08-25T09:54:18.632 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-08-25T09:54:18.632 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-08-25T09:54:18.633 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-08-25T09:54:18.634 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-08-25T09:54:18.634 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-08-25T09:54:18.634 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-08-25T09:54:18.634 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-08-25T09:54:18.634 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-08-25T09:54:18.634 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-08-25T09:54:18.635 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-08-25T09:54:18.636 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-08-25T09:54:18.636 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-08-25T09:54:18.636 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-08-25T09:54:18.636 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-08-25T09:54:18.636 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-08-25T09:54:18.636 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-08-25T09:54:18.637 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-08-25T09:54:18.637 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-08-25T09:54:18.637 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-08-25T09:54:18.637 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-08-25T09:54:18.637 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-08-25T09:54:21.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-08-25T09:54:21.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-08-25T09:54:21.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-08-25T09:54:21.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-08-25T09:54:21.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-08-25T09:54:21.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-08-25T09:54:21.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-08-25T09:54:21.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-08-25T09:54:21.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-08-25T09:54:21.899 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-08-25T09:54:21.900 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-08-25T09:54:21.901 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-08-25T09:54:21.902 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-08-25T09:54:21.903 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-08-25T09:54:21.904 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-08-25T09:54:21.905 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-08-25T09:54:23.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:54:23.090 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:54:24.726 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-08-25T09:54:24.727 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-08-25T09:54:24.727 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-08-25T09:54:24.727 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-08-25T09:54:24.727 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-08-25T09:54:24.727 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-08-25T09:54:24.727 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-08-25T09:54:24.728 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-08-25T09:54:24.729 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-08-25T09:54:24.730 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-08-25T09:54:24.731 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-08-25T09:54:24.732 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-08-25T09:54:24.733 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-08-25T09:54:24.734 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-08-25T09:54:24.735 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-08-25T09:54:24.736 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-08-25T09:54:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-08-25T09:54:24.738 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-08-25T09:54:24.738 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-08-25T09:54:24.738 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-08-25T09:54:28.960 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-08-25T09:54:28.960 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-08-25T09:54:28.961 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-08-25T09:54:28.961 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-08-25T09:54:28.961 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-08-25T09:54:28.962 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-08-25T09:54:28.962 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-08-25T09:54:28.962 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-08-25T09:54:28.962 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-08-25T09:54:28.963 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-08-25T09:54:28.963 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-08-25T09:54:28.963 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-08-25T09:54:28.963 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-08-25T09:54:28.963 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-08-25T09:54:28.963 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-08-25T09:54:28.964 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-08-25T09:54:28.965 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-08-25T09:54:28.965 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-08-25T09:54:28.965 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-08-25T09:54:28.965 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-08-25T09:54:28.965 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-08-25T09:54:28.965 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-08-25T09:54:28.966 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-08-25T09:54:28.967 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-08-25T09:54:28.967 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-08-25T09:54:28.967 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-08-25T09:54:28.967 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-08-25T09:54:28.968 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-08-25T09:54:28.968 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-08-25T09:54:28.968 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-08-25T09:54:28.969 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-08-25T09:54:28.969 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-08-25T09:54:28.969 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-08-25T09:54:28.969 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-08-25T09:54:28.970 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-08-25T09:54:28.970 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-08-25T09:54:28.970 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-08-25T09:54:28.971 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-08-25T09:54:28.971 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-08-25T09:54:28.971 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-08-25T09:54:28.971 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-08-25T09:54:28.972 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-08-25T09:54:28.972 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-08-25T09:54:28.972 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-08-25T09:54:28.973 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-08-25T09:54:28.973 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-08-25T09:54:28.973 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-08-25T09:54:28.974 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-08-25T09:54:28.974 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-08-25T09:54:28.974 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-08-25T09:54:28.974 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-08-25T09:54:28.975 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-08-25T09:54:28.975 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-08-25T09:54:28.975 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-08-25T09:54:28.975 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-08-25T09:54:28.976 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-08-25T09:54:28.976 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-08-25T09:54:28.976 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-08-25T09:54:28.977 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-08-25T09:54:28.977 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-08-25T09:54:28.977 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-08-25T09:54:28.977 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-08-25T09:54:28.978 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-08-25T09:54:28.978 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-08-25T09:54:28.978 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-08-25T09:54:28.978 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-08-25T09:54:28.979 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-08-25T09:54:28.979 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-08-25T09:54:32.588 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-08-25T09:54:32.589 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-08-25T09:54:32.590 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-08-25T09:54:32.590 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-08-25T09:54:32.590 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-08-25T09:54:32.590 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-08-25T09:54:32.590 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-08-25T09:54:32.590 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-08-25T09:54:32.591 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-08-25T09:54:32.592 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-08-25T09:54:32.592 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-08-25T09:54:32.592 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-08-25T09:54:32.592 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-08-25T09:54:32.592 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-08-25T09:54:32.592 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-08-25T09:54:32.593 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-08-25T09:54:32.594 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-08-25T09:54:32.595 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-08-25T09:54:32.596 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-08-25T09:54:32.596 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-08-25T09:54:32.596 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-08-25T09:54:32.596 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-08-25T09:54:32.596 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-08-25T09:54:32.597 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-08-25T09:54:32.598 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-08-25T09:54:32.599 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-08-25T09:54:32.600 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-08-25T09:54:32.600 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-08-25T09:54:34.989 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-08-25T09:54:34.990 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-08-25T09:54:34.990 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-08-25T09:54:34.990 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-08-25T09:54:34.991 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-08-25T09:54:34.991 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-08-25T09:54:34.991 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-08-25T09:54:34.991 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-08-25T09:54:34.991 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-08-25T09:54:34.992 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-08-25T09:54:34.992 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-08-25T09:54:34.993 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-08-25T09:54:34.993 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-08-25T09:54:34.993 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-08-25T09:54:34.993 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-08-25T09:54:34.993 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-08-25T09:54:34.994 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-08-25T09:54:34.995 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-08-25T09:54:34.995 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-08-25T09:54:34.995 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-08-25T09:54:34.995 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-08-25T09:54:34.996 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-08-25T09:54:34.996 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-08-25T09:54:34.996 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-08-25T09:54:34.996 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-08-25T09:54:34.997 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-08-25T09:54:34.997 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-08-25T09:54:34.997 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-08-25T09:54:34.997 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-08-25T09:54:34.998 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-08-25T09:54:34.998 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-08-25T09:54:34.998 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-08-25T09:54:34.998 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-08-25T09:54:34.999 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-08-25T09:54:34.999 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-08-25T09:54:34.999 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-08-25T09:54:35.000 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-08-25T09:54:35.000 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-08-25T09:54:35.000 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-08-25T09:54:35.000 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-08-25T09:54:35.003 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-08-25T09:54:35.004 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-08-25T09:54:35.005 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-08-25T09:54:35.006 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-08-25T09:54:35.007 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-08-25T09:54:35.007 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-08-25T09:54:35.007 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-08-25T09:54:35.007 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-08-25T09:54:40.014 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-08-25T09:54:40.014 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-08-25T09:54:40.014 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-08-25T09:54:40.014 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-08-25T09:54:40.015 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-08-25T09:54:40.016 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-08-25T09:54:40.017 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-08-25T09:54:40.018 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-08-25T09:54:40.019 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-08-25T09:54:40.020 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-08-25T09:54:40.021 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-08-25T09:54:40.022 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-08-25T09:54:40.023 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-08-25T09:54:40.024 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-08-25T09:54:42.208 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-08-25T09:54:42.209 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-08-25T09:54:42.210 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-08-25T09:54:42.210 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-08-25T09:54:42.210 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-08-25T09:54:42.210 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-08-25T09:54:42.210 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-08-25T09:54:42.211 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-08-25T09:54:42.211 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-08-25T09:54:42.211 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-08-25T09:54:42.211 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-08-25T09:54:42.211 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-08-25T09:54:42.212 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-08-25T09:54:42.212 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-08-25T09:54:42.212 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-08-25T09:54:42.212 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-08-25T09:54:42.212 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-08-25T09:54:42.212 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-08-25T09:54:42.213 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-08-25T09:54:42.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-08-25T09:54:42.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-08-25T09:54:42.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-08-25T09:54:42.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-08-25T09:54:42.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-08-25T09:54:42.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-08-25T09:54:42.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-08-25T09:54:42.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-08-25T09:54:42.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-08-25T09:54:42.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-08-25T09:54:42.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-08-25T09:54:42.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-08-25T09:54:42.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-08-25T09:54:42.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-08-25T09:54:42.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-08-25T09:54:42.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-08-25T09:54:46.886 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-08-25T09:54:46.886 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-08-25T09:54:46.886 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-08-25T09:54:46.886 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-08-25T09:54:46.886 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-08-25T09:54:46.887 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-08-25T09:54:46.888 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-08-25T09:54:46.889 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-08-25T09:54:46.890 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-08-25T09:54:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-08-25T09:54:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-08-25T09:54:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-08-25T09:54:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-08-25T09:54:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-08-25T09:54:46.891 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-08-25T09:54:46.892 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-08-25T09:54:46.892 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-08-25T09:54:46.892 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-08-25T09:54:46.892 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-08-25T09:54:46.892 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-08-25T09:54:46.892 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-08-25T09:54:46.893 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-08-25T09:54:46.894 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-08-25T09:54:46.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-08-25T09:54:46.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-08-25T09:54:46.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-08-25T09:54:46.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-08-25T09:54:46.895 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-08-25T09:54:46.896 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-08-25T09:54:46.897 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-08-25T09:54:46.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-08-25T09:54:46.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-08-25T09:54:46.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-08-25T09:54:46.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-08-25T09:54:46.898 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-08-25T09:54:50.214 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-08-25T09:54:50.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-08-25T09:54:50.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-08-25T09:54:50.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-08-25T09:54:50.215 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-08-25T09:54:50.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-08-25T09:54:50.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-08-25T09:54:50.216 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-08-25T09:54:50.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-08-25T09:54:50.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-08-25T09:54:50.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-08-25T09:54:50.217 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-08-25T09:54:50.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-08-25T09:54:50.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-08-25T09:54:50.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-08-25T09:54:50.218 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-08-25T09:54:50.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-08-25T09:54:50.219 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-08-25T09:54:50.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-08-25T09:54:50.220 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-08-25T09:54:50.221 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-08-25T09:54:50.222 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-08-25T09:54:50.222 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-08-25T09:54:50.222 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-08-25T09:54:50.222 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-08-25T09:54:50.223 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-08-25T09:54:50.223 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-08-25T09:54:50.223 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-08-25T09:54:50.223 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-08-25T09:54:50.224 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-08-25T09:54:50.224 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-08-25T09:54:50.224 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-08-25T09:54:50.224 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-08-25T09:54:50.224 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-08-25T09:54:50.224 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-08-25T09:54:50.225 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-08-25T09:54:50.225 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-08-25T09:54:50.225 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-08-25T09:54:50.226 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-08-25T09:54:50.226 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-08-25T09:54:50.226 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-08-25T09:54:50.226 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-08-25T09:54:50.226 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-08-25T09:54:50.227 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-08-25T09:54:50.227 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-08-25T09:54:50.227 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-08-25T09:54:50.228 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-08-25T09:54:50.228 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-08-25T09:54:50.228 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-08-25T09:54:50.228 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-08-25T09:54:50.229 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-08-25T09:54:50.230 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-08-25T09:54:50.230 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-08-25T09:54:50.230 INFO:tasks.workunit.client.0.smithi071.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-08-25T09:54:50.230 INFO:tasks.workunit.client.0.smithi071.stdout:Take third snapshot .snap/test3 2023-08-25T09:54:50.383 INFO:tasks.workunit.client.0.smithi071.stdout:Delete the snapshots... 2023-08-25T09:54:51.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:54:52.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.201+0000 7f66b69cb640 1 Processor -- start 2023-08-25T09:54:52.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.201+0000 7f66b69cb640 1 -- start start 2023-08-25T09:54:52.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.202+0000 7f66b69cb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f66b00c60e0 0x7f66b0158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.202+0000 7f66b69cb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f66b0071be0 0x7f66b0159520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.202+0000 7f66b69cb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b00c5260 con 0x7f66b0152db0 2023-08-25T09:54:52.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.202+0000 7f66b69cb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b0158b90 con 0x7f66b0071be0 2023-08-25T09:54:52.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.202+0000 7f66b69cb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b00c6580 con 0x7f66b00c60e0 2023-08-25T09:54:52.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.202+0000 7f66b59c9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f66b0152db0 0x7f66b00c67c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60448/0 (socket says 172.21.15.71:60448) 2023-08-25T09:54:52.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b59c9640 1 -- 172.21.15.71:0/1627964690 learned_addr learned my addr 172.21.15.71:0/1627964690 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:54:52.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b61ca640 1 --2- 172.21.15.71:0/1627964690 >> v2:172.21.15.71:3300/0 conn(0x7f66b0071be0 0x7f66b0159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b51c8640 1 --2- 172.21.15.71:0/1627964690 >> v2:172.21.15.164:3300/0 conn(0x7f66b00c60e0 0x7f66b0158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b49c7640 1 -- 172.21.15.71:0/1627964690 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860918038 0 0) 0x7f66b00c5260 con 0x7f66b0152db0 2023-08-25T09:54:52.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b49c7640 1 -- 172.21.15.71:0/1627964690 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f669c003520 con 0x7f66b0152db0 2023-08-25T09:54:52.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b61ca640 1 -- 172.21.15.71:0/1627964690 >> v2:172.21.15.164:3300/0 conn(0x7f66b00c60e0 msgr2=0x7f66b0158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:52.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b61ca640 1 --2- 172.21.15.71:0/1627964690 >> v2:172.21.15.164:3300/0 conn(0x7f66b00c60e0 0x7f66b0158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b61ca640 1 -- 172.21.15.71:0/1627964690 >> v1:172.21.15.71:6789/0 conn(0x7f66b0152db0 legacy=0x7f66b00c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.203+0000 7f66b61ca640 1 -- 172.21.15.71:0/1627964690 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b015cc30 con 0x7f66b0071be0 2023-08-25T09:54:52.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.205+0000 7f66b61ca640 1 --2- 172.21.15.71:0/1627964690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0071be0 0x7f66b0159520 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f66b00c72c0 tx=0x7f66a802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b740291acdb8d25 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:52.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.205+0000 7f66b49c7640 1 -- 172.21.15.71:0/1627964690 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a802cc70 con 0x7f66b0071be0 2023-08-25T09:54:52.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.205+0000 7f66b49c7640 1 -- 172.21.15.71:0/1627964690 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a802cdd0 con 0x7f66b0071be0 2023-08-25T09:54:52.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.205+0000 7f66b49c7640 1 -- 172.21.15.71:0/1627964690 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8034660 con 0x7f66b0071be0 2023-08-25T09:54:52.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 -- 172.21.15.71:0/1627964690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0071be0 msgr2=0x7f66b0159520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 --2- 172.21.15.71:0/1627964690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0071be0 0x7f66b0159520 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f66b00c72c0 tx=0x7f66a802bec0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 -- 172.21.15.71:0/1627964690 shutdown_connections 2023-08-25T09:54:52.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 --2- 172.21.15.71:0/1627964690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0071be0 0x7f66b0159520 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 --2- 172.21.15.71:0/1627964690 >> v2:172.21.15.164:3300/0 conn(0x7f66b00c60e0 0x7f66b0158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 -- 172.21.15.71:0/1627964690 >> 172.21.15.71:0/1627964690 conn(0x7f66b006d0d0 msgr2=0x7f66b00ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:52.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 -- 172.21.15.71:0/1627964690 shutdown_connections 2023-08-25T09:54:52.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.206+0000 7f66b69cb640 1 -- 172.21.15.71:0/1627964690 wait complete. 2023-08-25T09:54:52.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.208+0000 7f66b69cb640 1 Processor -- start 2023-08-25T09:54:52.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.208+0000 7f66b69cb640 1 -- start start 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b69cb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66b0071be0 0x7f66b01ee770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b69cb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66b00c60e0 0x7f66b014f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b69cb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 0x7f66b015b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b69cb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b015b160 con 0x7f66b0152db0 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b69cb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0157520 con 0x7f66b00c60e0 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b69cb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b015ab30 con 0x7f66b0071be0 2023-08-25T09:54:52.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b61ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 0x7f66b015b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b61ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 0x7f66b015b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52220/0 (socket says 172.21.15.71:52220) 2023-08-25T09:54:52.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.209+0000 7f66b61ca640 1 -- 172.21.15.71:0/2227192113 learned_addr learned my addr 172.21.15.71:0/2227192113 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:54:52.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b59c9640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66b0071be0 0x7f66b01ee770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b51c8640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66b00c60e0 0x7f66b014f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b61ca640 1 -- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66b0071be0 msgr2=0x7f66b01ee770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b61ca640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66b0071be0 0x7f66b01ee770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b61ca640 1 -- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66b00c60e0 msgr2=0x7f66b014f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b61ca640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66b00c60e0 0x7f66b014f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b61ca640 1 -- 172.21.15.71:0/2227192113 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8009a70 con 0x7f66b0152db0 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.210+0000 7f66b61ca640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 0x7f66b015b5f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009e80 tx=0x7f66a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.211+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8040650 con 0x7f66b0152db0 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.211+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a80407b0 con 0x7f66b0152db0 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.211+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b015a250 con 0x7f66b0152db0 2023-08-25T09:54:52.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.211+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8040b60 con 0x7f66b0152db0 2023-08-25T09:54:52.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.211+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b015a810 con 0x7f66b0152db0 2023-08-25T09:54:52.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.212+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b015b310 con 0x7f66b0152db0 2023-08-25T09:54:52.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.213+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f66a8040cc0 con 0x7f66b0152db0 2023-08-25T09:54:52.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.214+0000 7f66a67fc640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f669406a080 0x7f669406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.214+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f66a80ab570 con 0x7f66b0152db0 2023-08-25T09:54:52.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.214+0000 7f66b59c9640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f669406a080 0x7f669406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.215+0000 7f66b59c9640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f669406a080 0x7f669406c540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0011e0 tx=0x7f66ac007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:52.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.220+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f66b015b310 con 0x7f66b0152db0 2023-08-25T09:54:52.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.492+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66b015b310 con 0x7f66b0152db0 2023-08-25T09:54:52.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.492+0000 7f66a67fc640 1 -- 172.21.15.71:0/2227192113 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7f66b015b310 con 0x7f66b0152db0 2023-08-25T09:54:52.494 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:54:52.494 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:54:52.495 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:54:52.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f669406a080 msgr2=0x7f669406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f669406a080 0x7f669406c540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f66ac0011e0 tx=0x7f66ac007d30 comp rx=0 tx=0).stop 2023-08-25T09:54:52.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 msgr2=0x7f66b015b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 0x7f66b015b5f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009e80 tx=0x7f66a8004a40 comp rx=0 tx=0).stop 2023-08-25T09:54:52.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 shutdown_connections 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f66b00c60e0 0x7f66b014f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f669406a080 0x7f669406c540 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f66b0152db0 0x7f66b015b5f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 --2- 172.21.15.71:0/2227192113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f66b0071be0 0x7f66b01ee770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 >> 172.21.15.71:0/2227192113 conn(0x7f66b006d0d0 msgr2=0x7f66b0151800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 shutdown_connections 2023-08-25T09:54:52.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.495+0000 7f66b69cb640 1 -- 172.21.15.71:0/2227192113 wait complete. 2023-08-25T09:54:52.516 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2023-08-25T09:54:52.517 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-08-25T09:54:52.517 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T09:54:52.542 INFO:tasks.ceph.mds.b:Stopped 2023-08-25T09:54:52.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:54:52.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.752+0000 7f797fbe8640 1 Processor -- start 2023-08-25T09:54:52.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.752+0000 7f797fbe8640 1 -- start start 2023-08-25T09:54:52.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.753+0000 7f797fbe8640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7978156cc0 0x7f79780c3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.753+0000 7f797fbe8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f79780c3d90 0x7f79780c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.753+0000 7f797fbe8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f797806f1c0 con 0x7f7978150fa0 2023-08-25T09:54:52.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.753+0000 7f797fbe8640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f797815b2b0 con 0x7f7978156cc0 2023-08-25T09:54:52.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.753+0000 7f797fbe8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79780c5f50 con 0x7f79780c3d90 2023-08-25T09:54:52.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.753+0000 7f797d183640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7978156cc0 0x7f79780c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797d183640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7978156cc0 0x7f79780c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52234/0 (socket says 172.21.15.71:52234) 2023-08-25T09:54:52.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797d183640 1 -- 172.21.15.71:0/1159996138 learned_addr learned my addr 172.21.15.71:0/1159996138 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:54:52.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797e185640 1 --2- 172.21.15.71:0/1159996138 >> v2:172.21.15.164:3300/0 conn(0x7f79780c3d90 0x7f79780c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797d183640 1 -- 172.21.15.71:0/1159996138 >> v2:172.21.15.164:3300/0 conn(0x7f79780c3d90 msgr2=0x7f79780c8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:52.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797d183640 1 --2- 172.21.15.71:0/1159996138 >> v2:172.21.15.164:3300/0 conn(0x7f79780c3d90 0x7f79780c8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797d183640 1 -- 172.21.15.71:0/1159996138 >> v1:172.21.15.164:6789/0 conn(0x7f7978150fa0 legacy=0x7f797815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:52.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.754+0000 7f797d183640 1 -- 172.21.15.71:0/1159996138 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797815a3e0 con 0x7f7978156cc0 2023-08-25T09:54:52.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.756+0000 7f797d183640 1 --2- 172.21.15.71:0/1159996138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978156cc0 0x7f79780c3850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7978071850 tx=0x7f796802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd280b0b2b3de9a3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:52.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.756+0000 7f797c982640 1 -- 172.21.15.71:0/1159996138 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968031870 con 0x7f7978156cc0 2023-08-25T09:54:52.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.756+0000 7f797c982640 1 -- 172.21.15.71:0/1159996138 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79680319d0 con 0x7f7978156cc0 2023-08-25T09:54:52.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.756+0000 7f797c982640 1 -- 172.21.15.71:0/1159996138 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968031d60 con 0x7f7978156cc0 2023-08-25T09:54:52.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.757+0000 7f797fbe8640 1 -- 172.21.15.71:0/1159996138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978156cc0 msgr2=0x7f79780c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.757+0000 7f797fbe8640 1 --2- 172.21.15.71:0/1159996138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978156cc0 0x7f79780c3850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7978071850 tx=0x7f796802b910 comp rx=0 tx=0).stop 2023-08-25T09:54:52.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.758+0000 7f797fbe8640 1 -- 172.21.15.71:0/1159996138 shutdown_connections 2023-08-25T09:54:52.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.758+0000 7f797fbe8640 1 --2- 172.21.15.71:0/1159996138 >> v2:172.21.15.164:3300/0 conn(0x7f79780c3d90 0x7f79780c8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.758+0000 7f797fbe8640 1 --2- 172.21.15.71:0/1159996138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978156cc0 0x7f79780c3850 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.758+0000 7f797fbe8640 1 -- 172.21.15.71:0/1159996138 >> 172.21.15.71:0/1159996138 conn(0x7f79780654f0 msgr2=0x7f7978065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:52.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.758+0000 7f797fbe8640 1 -- 172.21.15.71:0/1159996138 shutdown_connections 2023-08-25T09:54:52.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.758+0000 7f797fbe8640 1 -- 172.21.15.71:0/1159996138 wait complete. 2023-08-25T09:54:52.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.760+0000 7f797fbe8640 1 Processor -- start 2023-08-25T09:54:52.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.760+0000 7f797fbe8640 1 -- start start 2023-08-25T09:54:52.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797fbe8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79780c3d90 0x7f797814abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797fbe8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 0x7f7978158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797fbe8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7978156cc0 0x7f797806bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797fbe8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79780c8760 con 0x7f7978150fa0 2023-08-25T09:54:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797fbe8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79780c8500 con 0x7f7978156cc0 2023-08-25T09:54:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797fbe8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7978158020 con 0x7f79780c3d90 2023-08-25T09:54:52.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.761+0000 7f797d183640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 0x7f7978158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 0x7f7978158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52244/0 (socket says 172.21.15.71:52244) 2023-08-25T09:54:52.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 -- 172.21.15.71:0/567804125 learned_addr learned my addr 172.21.15.71:0/567804125 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797e185640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7978156cc0 0x7f797806bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d984640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79780c3d90 0x7f797814abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 -- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79780c3d90 msgr2=0x7f797814abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79780c3d90 0x7f797814abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 -- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7978156cc0 msgr2=0x7f797806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7978156cc0 0x7f797806bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:52.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.762+0000 7f797d183640 1 -- 172.21.15.71:0/567804125 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680094d0 con 0x7f7978150fa0 2023-08-25T09:54:52.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.763+0000 7f797d183640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 0x7f7978158de0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7968000c00 tx=0x7f79680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:52.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.763+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796803abe0 con 0x7f7978150fa0 2023-08-25T09:54:52.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.763+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796803ad40 con 0x7f7978150fa0 2023-08-25T09:54:52.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.763+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968044610 con 0x7f7978150fa0 2023-08-25T09:54:52.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.763+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978158300 con 0x7f7978150fa0 2023-08-25T09:54:52.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.763+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79781588c0 con 0x7f7978150fa0 2023-08-25T09:54:52.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.765+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7978069160 con 0x7f7978150fa0 2023-08-25T09:54:52.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.766+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f796804d450 con 0x7f7978150fa0 2023-08-25T09:54:52.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.767+0000 7f796e7fc640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f795c06a080 0x7f795c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:52.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.767+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f796802d040 con 0x7f7978150fa0 2023-08-25T09:54:52.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.768+0000 7f797d984640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f795c06a080 0x7f795c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:52.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.768+0000 7f797d984640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f795c06a080 0x7f795c06c540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7974004f60 tx=0x7f7974007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:52.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:52.774+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f79680761a0 con 0x7f7978150fa0 2023-08-25T09:54:53.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.008+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f797806ed90 con 0x7f7978150fa0 2023-08-25T09:54:53.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.009+0000 7f796e7fc640 1 -- 172.21.15.71:0/567804125 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7f7968035090 con 0x7f7978150fa0 2023-08-25T09:54:53.011 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:54:53.011 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:54:53.011 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:54:53.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f795c06a080 msgr2=0x7f795c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:53.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f795c06a080 0x7f795c06c540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7974004f60 tx=0x7f7974007e30 comp rx=0 tx=0).stop 2023-08-25T09:54:53.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 msgr2=0x7f7978158de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:53.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 0x7f7978158de0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7968000c00 tx=0x7f79680046c0 comp rx=0 tx=0).stop 2023-08-25T09:54:53.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 shutdown_connections 2023-08-25T09:54:53.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7978156cc0 0x7f797806bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:53.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f795c06a080 0x7f795c06c540 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:53.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7978150fa0 0x7f7978158de0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:53.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 --2- 172.21.15.71:0/567804125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79780c3d90 0x7f797814abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:53.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.011+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 >> 172.21.15.71:0/567804125 conn(0x7f79780654f0 msgr2=0x7f79780c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:53.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.012+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 shutdown_connections 2023-08-25T09:54:53.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:53.012+0000 7f797fbe8640 1 -- 172.21.15.71:0/567804125 wait complete. 2023-08-25T09:54:53.031 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-08-25T09:54:53.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:54:53.246 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:54:55.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:54:55.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.257+0000 7ff25a846640 1 Processor -- start 2023-08-25T09:54:55.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.257+0000 7ff25a846640 1 -- start start 2023-08-25T09:54:55.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff25a846640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff254070cf0 0x7ff25406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff25a846640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25406eb60 con 0x7ff254152d80 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff25a846640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25406ca30 con 0x7ff2540c4560 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff25a846640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25406cf90 con 0x7ff254070cf0 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff253fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff254152d80 0x7ff2540c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60450/0 (socket says 172.21.15.71:60450) 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff253fff640 1 -- 172.21.15.71:0/1617602238 learned_addr learned my addr 172.21.15.71:0/1617602238 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.259+0000 7ff258de3640 1 --2- 172.21.15.71:0/1617602238 >> v2:172.21.15.164:3300/0 conn(0x7ff254070cf0 0x7ff25406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595588595 0 0) 0x7ff25406eb60 con 0x7ff254152d80 2023-08-25T09:54:55.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff23c003520 con 0x7ff254152d80 2023-08-25T09:54:55.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3401774074 0 0) 0x7ff23c003520 con 0x7ff254152d80 2023-08-25T09:54:55.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 >> v2:172.21.15.164:3300/0 conn(0x7ff254070cf0 msgr2=0x7ff25406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:55.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 --2- 172.21.15.71:0/1617602238 >> v2:172.21.15.164:3300/0 conn(0x7ff254070cf0 0x7ff25406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 >> v1:172.21.15.164:6789/0 conn(0x7ff2540c4560 legacy=0x7ff2541617a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff254158fd0 con 0x7ff254152d80 2023-08-25T09:54:55.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff244001670 con 0x7ff254152d80 2023-08-25T09:54:55.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 >> v1:172.21.15.71:6789/0 conn(0x7ff254152d80 legacy=0x7ff2540c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff23c0039e0 0x7ff23c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff23c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff23c0051f0 0x7ff23c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25406cf90 con 0x7ff23c0047c0 2023-08-25T09:54:55.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c00c150 con 0x7ff23c0051f0 2023-08-25T09:54:55.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c00c310 con 0x7ff23c0039e0 2023-08-25T09:54:55.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.260+0000 7ff2537fe640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff23c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:55.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff253fff640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff23c0039e0 0x7ff23c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:55.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff258de3640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff23c0051f0 0x7ff23c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:55.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff2537fe640 1 -- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff23c0039e0 msgr2=0x7ff23c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:55.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff2537fe640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff23c0039e0 0x7ff23c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff2537fe640 1 -- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff23c0051f0 msgr2=0x7ff23c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:55.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff2537fe640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff23c0051f0 0x7ff23c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff258de3640 1 -- 172.21.15.71:0/1617602238 reap_dead start 2023-08-25T09:54:55.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.261+0000 7ff2537fe640 1 -- 172.21.15.71:0/1617602238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c003520 con 0x7ff23c0047c0 2023-08-25T09:54:55.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.262+0000 7ff2537fe640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff23c004c40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff248001750 tx=0x7ff23c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e819417203d363d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:55.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.263+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2480052d0 con 0x7ff23c0047c0 2023-08-25T09:54:55.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.263+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24802e050 con 0x7ff23c0047c0 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.263+0000 7ff252ffd640 1 -- 172.21.15.71:0/1617602238 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff248003830 con 0x7ff23c0047c0 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.263+0000 7ff25a846640 1 -- 172.21.15.71:0/1617602238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 msgr2=0x7ff23c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.263+0000 7ff25a846640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff23c004c40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff248001750 tx=0x7ff23c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.264+0000 7ff25a846640 1 -- 172.21.15.71:0/1617602238 shutdown_connections 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.264+0000 7ff25a846640 1 --2- 172.21.15.71:0/1617602238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff23c004c40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.264+0000 7ff25a846640 1 -- 172.21.15.71:0/1617602238 >> 172.21.15.71:0/1617602238 conn(0x7ff2540654f0 msgr2=0x7ff254065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:55.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.264+0000 7ff25a846640 1 -- 172.21.15.71:0/1617602238 shutdown_connections 2023-08-25T09:54:55.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.264+0000 7ff25a846640 1 -- 172.21.15.71:0/1617602238 wait complete. 2023-08-25T09:54:55.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.266+0000 7ff25a846640 1 Processor -- start 2023-08-25T09:54:55.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.266+0000 7ff25a846640 1 -- start start 2023-08-25T09:54:55.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff25a846640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff254146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff25a846640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff254156ca0 0x7ff25414d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff25a846640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff254157100 0x7ff25414dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff25a846640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540c8520 con 0x7ff23c0047c0 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff25a846640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2541eede0 con 0x7ff254157100 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff25a846640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff254158cb0 con 0x7ff254156ca0 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff254146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:55.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff254146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52256/0 (socket says 172.21.15.71:52256) 2023-08-25T09:54:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 -- 172.21.15.71:0/4095607969 learned_addr learned my addr 172.21.15.71:0/4095607969 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:54:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 -- 172.21.15.71:0/4095607969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff254156ca0 msgr2=0x7ff25414d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff254156ca0 0x7ff25414d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 -- 172.21.15.71:0/4095607969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff254157100 msgr2=0x7ff25414dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff254157100 0x7ff25414dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.267+0000 7ff253fff640 1 -- 172.21.15.71:0/4095607969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2541589d0 con 0x7ff23c0047c0 2023-08-25T09:54:55.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.268+0000 7ff253fff640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff254146770 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff24400e730 tx=0x7ff24400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:55.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.268+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400c640 con 0x7ff23c0047c0 2023-08-25T09:54:55.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.268+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24400c7c0 con 0x7ff23c0047c0 2023-08-25T09:54:55.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.268+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400cb70 con 0x7ff23c0047c0 2023-08-25T09:54:55.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.268+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2541ee020 con 0x7ff23c0047c0 2023-08-25T09:54:55.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.268+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2541ee5e0 con 0x7ff23c0047c0 2023-08-25T09:54:55.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.269+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff254069160 con 0x7ff23c0047c0 2023-08-25T09:54:55.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.270+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff24400ccd0 con 0x7ff23c0047c0 2023-08-25T09:54:55.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.271+0000 7ff250ff9640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff24006a080 0x7ff24006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:55.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.271+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff2440886c0 con 0x7ff23c0047c0 2023-08-25T09:54:55.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.271+0000 7ff2537fe640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff24006a080 0x7ff24006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:55.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.272+0000 7ff2537fe640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff24006a080 0x7ff24006c540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff248033000 tx=0x7ff24802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:55.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.276+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff244053a30 con 0x7ff23c0047c0 2023-08-25T09:54:55.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.506+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff254150eb0 con 0x7ff23c0047c0 2023-08-25T09:54:55.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.507+0000 7ff250ff9640 1 -- 172.21.15.71:0/4095607969 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7ff244057050 con 0x7ff23c0047c0 2023-08-25T09:54:55.510 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:54:55.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:54:55.510 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:54:55.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.510+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff24006a080 msgr2=0x7ff24006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.510+0000 7ff25a846640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff24006a080 0x7ff24006c540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff248033000 tx=0x7ff24802f000 comp rx=0 tx=0).stop 2023-08-25T09:54:55.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.510+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 msgr2=0x7ff254146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:55.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.510+0000 7ff25a846640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff254146770 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff24400e730 tx=0x7ff24400ebf0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.510+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 shutdown_connections 2023-08-25T09:54:55.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.510+0000 7ff25a846640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff254157100 0x7ff25414dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.511+0000 7ff25a846640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff24006a080 0x7ff24006c540 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.511+0000 7ff25a846640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff254156ca0 0x7ff25414d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.511+0000 7ff25a846640 1 --2- 172.21.15.71:0/4095607969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff23c0047c0 0x7ff254146770 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:55.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.511+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 >> 172.21.15.71:0/4095607969 conn(0x7ff2540654f0 msgr2=0x7ff2540c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:55.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.511+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 shutdown_connections 2023-08-25T09:54:55.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:55.511+0000 7ff25a846640 1 -- 172.21.15.71:0/4095607969 wait complete. 2023-08-25T09:54:55.532 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-08-25T09:54:57.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:54:57.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.768+0000 7f62dbfff640 1 Processor -- start 2023-08-25T09:54:57.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.769+0000 7f62dbfff640 1 -- start start 2023-08-25T09:54:57.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.770+0000 7f62dbfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f62dc0c72f0 0x7f62dc0acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.770+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 0x7f62dc152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.770+0000 7f62dbfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc0c1060 con 0x7f62dc0ad2f0 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.770+0000 7f62dbfff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc0adf70 con 0x7f62dc0c72f0 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.770+0000 7f62dbfff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0af730 con 0x7f62dc0c76d0 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.771+0000 7f62db7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f62dc0ad2f0 0x7f62dc155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60454/0 (socket says 172.21.15.71:60454) 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.771+0000 7f62db7fe640 1 -- 172.21.15.71:0/1959503965 learned_addr learned my addr 172.21.15.71:0/1959503965 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.771+0000 7f62daffd640 1 --2- 172.21.15.71:0/1959503965 >> v2:172.21.15.164:3300/0 conn(0x7f62dc0c72f0 0x7f62dc0acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.771+0000 7f62da7fc640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 0x7f62dc152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.771+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946452258 0 0) 0x7f62dc0c1060 con 0x7f62dc0ad2f0 2023-08-25T09:54:57.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.771+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c4005730 con 0x7f62dc0ad2f0 2023-08-25T09:54:57.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176557514 0 0) 0x7f62c4005730 con 0x7f62dc0ad2f0 2023-08-25T09:54:57.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 msgr2=0x7f62dc152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:57.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 0x7f62dc152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:57.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 >> v2:172.21.15.164:3300/0 conn(0x7f62dc0c72f0 msgr2=0x7f62dc0acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:57.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 --2- 172.21.15.71:0/1959503965 >> v2:172.21.15.164:3300/0 conn(0x7f62dc0c72f0 0x7f62dc0acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:57.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc06b4a0 con 0x7f62dc0ad2f0 2023-08-25T09:54:57.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f62d40019f0 con 0x7f62dc0ad2f0 2023-08-25T09:54:57.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 >> v1:172.21.15.71:6789/0 conn(0x7f62dc0ad2f0 legacy=0x7f62dc155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:57.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62c4005bf0 0x7f62c4006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 0x7f62c40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62c4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0adf70 con 0x7f62c4006f10 2023-08-25T09:54:57.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc0af730 con 0x7f62c4005bf0 2023-08-25T09:54:57.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c400dc10 con 0x7f62dc0c76d0 2023-08-25T09:54:57.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62db7fe640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62c4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62daffd640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62c4005bf0 0x7f62c4006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62da7fc640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 0x7f62c40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62db7fe640 1 -- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 msgr2=0x7f62c40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62db7fe640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc0c76d0 0x7f62c40069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62db7fe640 1 -- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62c4005bf0 msgr2=0x7f62c4006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62db7fe640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62c4005bf0 0x7f62c4006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62db7fe640 1 -- 172.21.15.71:0/1959503965 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c4005730 con 0x7f62c4006f10 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.772+0000 7f62daffd640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62c4005bf0 0x7f62c4006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.773+0000 7f62db7fe640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62c4007390 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f62d400a830 tx=0x7f62c4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef1f787839fbb50 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:57.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d4034630 con 0x7f62c4006f10 2023-08-25T09:54:57.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d402f070 con 0x7f62c4006f10 2023-08-25T09:54:57.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62d9ffb640 1 -- 172.21.15.71:0/1959503965 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40349f0 con 0x7f62c4006f10 2023-08-25T09:54:57.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62dbfff640 1 -- 172.21.15.71:0/1959503965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 msgr2=0x7f62c4007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:57.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62dbfff640 1 --2- 172.21.15.71:0/1959503965 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62c4007390 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f62d400a830 tx=0x7f62c4007a00 comp rx=0 tx=0).stop 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62db7fe640 1 -- 172.21.15.71:0/1959503965 reap_dead start 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62dbfff640 1 -- 172.21.15.71:0/1959503965 shutdown_connections 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62dbfff640 1 -- 172.21.15.71:0/1959503965 >> 172.21.15.71:0/1959503965 conn(0x7f62dc06c280 msgr2=0x7f62dc06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.774+0000 7f62dbfff640 1 -- 172.21.15.71:0/1959503965 shutdown_connections 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.775+0000 7f62dbfff640 1 -- 172.21.15.71:0/1959503965 wait complete. 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.776+0000 7f62dbfff640 1 Processor -- start 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.776+0000 7f62dbfff640 1 -- start start 2023-08-25T09:54:57.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.777+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62dc14d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.777+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc14d9f0 0x7f62dc0bf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.777+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc14e050 0x7f62dc14aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.777+0000 7f62dbfff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc14c850 con 0x7f62c4006f10 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.777+0000 7f62dbfff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1edc90 con 0x7f62dc14d9f0 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.777+0000 7f62dbfff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc14b740 con 0x7f62dc14e050 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62dc14d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62dc14d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52272/0 (socket says 172.21.15.71:52272) 2023-08-25T09:54:57.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 -- 172.21.15.71:0/606960880 learned_addr learned my addr 172.21.15.71:0/606960880 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:54:57.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 -- 172.21.15.71:0/606960880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc14e050 msgr2=0x7f62dc14aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:57.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc14e050 0x7f62dc14aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:57.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 -- 172.21.15.71:0/606960880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc14d9f0 msgr2=0x7f62dc0bf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc14d9f0 0x7f62dc0bf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 -- 172.21.15.71:0/606960880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc14ba20 con 0x7f62c4006f10 2023-08-25T09:54:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.778+0000 7f62daffd640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62dc14d3e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f62cc010a30 tx=0x7f62cc010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.779+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00ec60 con 0x7f62c4006f10 2023-08-25T09:54:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.779+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00edc0 con 0x7f62c4006f10 2023-08-25T09:54:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.779+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc01c620 con 0x7f62c4006f10 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.779+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc14de70 con 0x7f62c4006f10 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.779+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc1e97b0 con 0x7f62c4006f10 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.780+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc04e420 con 0x7f62c4006f10 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.781+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f62cc01c780 con 0x7f62c4006f10 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.782+0000 7f62c37fe640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f62c806a050 0x7f62c806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.782+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f62cc088ff0 con 0x7f62c4006f10 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.782+0000 7f62da7fc640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f62c806a050 0x7f62c806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:54:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.783+0000 7f62da7fc640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f62c806a050 0x7f62c806c510 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f62dc1562e0 tx=0x7f62d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:54:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:57.787+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f62cc054320 con 0x7f62c4006f10 2023-08-25T09:54:58.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.013+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62dc1ed9b0 con 0x7f62c4006f10 2023-08-25T09:54:58.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.015+0000 7f62c37fe640 1 -- 172.21.15.71:0/606960880 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7f62cc057940 con 0x7f62c4006f10 2023-08-25T09:54:58.017 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:54:58.017 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:54:58.017 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:54:58.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f62c806a050 msgr2=0x7f62c806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:58.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f62c806a050 0x7f62c806c510 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f62dc1562e0 tx=0x7f62d0008040 comp rx=0 tx=0).stop 2023-08-25T09:54:58.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 msgr2=0x7f62dc14d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:54:58.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62dc14d3e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f62cc010a30 tx=0x7f62cc010ef0 comp rx=0 tx=0).stop 2023-08-25T09:54:58.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 shutdown_connections 2023-08-25T09:54:58.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62dc14d9f0 0x7f62dc0bf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:58.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f62c806a050 0x7f62c806c510 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:58.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62dc14e050 0x7f62dc14aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:58.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 --2- 172.21.15.71:0/606960880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62c4006f10 0x7f62dc14d3e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:54:58.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 >> 172.21.15.71:0/606960880 conn(0x7f62dc06c280 msgr2=0x7f62dc06a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:54:58.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.017+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 shutdown_connections 2023-08-25T09:54:58.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:54:58.018+0000 7f62dbfff640 1 -- 172.21.15.71:0/606960880 wait complete. 2023-08-25T09:54:58.036 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-08-25T09:55:00.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:00.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.262+0000 7fde2e1e8640 1 Processor -- start 2023-08-25T09:55:00.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.262+0000 7fde2e1e8640 1 -- start start 2023-08-25T09:55:00.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.264+0000 7fde2e1e8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 0x7fde280c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.264+0000 7fde2e1e8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2815db40 con 0x7fde28066010 2023-08-25T09:55:00.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.264+0000 7fde2e1e8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2815d230 con 0x7fde28070cf0 2023-08-25T09:55:00.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.264+0000 7fde2e1e8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2806cb40 con 0x7fde28157450 2023-08-25T09:55:00.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.264+0000 7fde1ffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fde28066010 0x7fde2806e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39758/0 (socket says 172.21.15.71:39758) 2023-08-25T09:55:00.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.264+0000 7fde1ffff640 1 -- 172.21.15.71:0/1875178289 learned_addr learned my addr 172.21.15.71:0/1875178289 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde277fe640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 0x7fde280c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766899817 0 0) 0x7fde2815db40 con 0x7fde28066010 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde14003520 con 0x7fde28066010 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3641981302 0 0) 0x7fde14003520 con 0x7fde28066010 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 msgr2=0x7fde280c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 0x7fde280c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 >> v1:172.21.15.164:6789/0 conn(0x7fde28070cf0 legacy=0x7fde280c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2806ce20 con 0x7fde28066010 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fde10001670 con 0x7fde28066010 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 >> v1:172.21.15.71:6789/0 conn(0x7fde28066010 legacy=0x7fde2806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde14003a10 0x7fde14003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.265+0000 7fde26ffd640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde26ffd640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 0x7fde14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2806cb40 con 0x7fde140047c0 2023-08-25T09:55:00.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400bb10 con 0x7fde14003a10 2023-08-25T09:55:00.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1400bcd0 con 0x7fde28157450 2023-08-25T09:55:00.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde1ffff640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde27fff640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 0x7fde14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde1ffff640 1 -- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 msgr2=0x7fde14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:00.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde1ffff640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde28157450 0x7fde14005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde1ffff640 1 -- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde14003a10 msgr2=0x7fde14003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde1ffff640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde14003a10 0x7fde14003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.266+0000 7fde1ffff640 1 -- 172.21.15.71:0/1875178289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde14003520 con 0x7fde140047c0 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.267+0000 7fde1ffff640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde14004c40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c240 tx=0x7fde140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b57aa0023577e79 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.268+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10005ad0 con 0x7fde140047c0 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.268+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde10005c30 con 0x7fde140047c0 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.268+0000 7fde26ffd640 1 -- 172.21.15.71:0/1875178289 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1002c8b0 con 0x7fde140047c0 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.268+0000 7fde2e1e8640 1 -- 172.21.15.71:0/1875178289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 msgr2=0x7fde14004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.268+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/1875178289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde14004c40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c240 tx=0x7fde140057f0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.268+0000 7fde27fff640 1 -- 172.21.15.71:0/1875178289 reap_dead start 2023-08-25T09:55:00.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.269+0000 7fde2e1e8640 1 -- 172.21.15.71:0/1875178289 shutdown_connections 2023-08-25T09:55:00.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.269+0000 7fde2e1e8640 1 -- 172.21.15.71:0/1875178289 >> 172.21.15.71:0/1875178289 conn(0x7fde280654f0 msgr2=0x7fde28065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:00.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.269+0000 7fde2e1e8640 1 -- 172.21.15.71:0/1875178289 shutdown_connections 2023-08-25T09:55:00.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.269+0000 7fde2e1e8640 1 -- 172.21.15.71:0/1875178289 wait complete. 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.270+0000 7fde2e1e8640 1 Processor -- start 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.271+0000 7fde2e1e8640 1 -- start start 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde2e1e8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde28157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde2e1e8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde281f2c00 0x7fde280c35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde2e1e8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde280c50a0 0x7fde281522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde2e1e8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde280c3130 con 0x7fde140047c0 2023-08-25T09:55:00.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde2e1e8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde28158220 con 0x7fde281f2c00 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde2e1e8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde280c2800 con 0x7fde280c50a0 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde28157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde28157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59592/0 (socket says 172.21.15.71:59592) 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 -- 172.21.15.71:0/3649313880 learned_addr learned my addr 172.21.15.71:0/3649313880 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde1ffff640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde281f2c00 0x7fde280c35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde27fff640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde280c50a0 0x7fde281522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 -- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde280c50a0 msgr2=0x7fde281522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde280c50a0 0x7fde281522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 -- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde281f2c00 msgr2=0x7fde280c35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde281f2c00 0x7fde280c35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.272+0000 7fde277fe640 1 -- 172.21.15.71:0/3649313880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde280c2ae0 con 0x7fde140047c0 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.273+0000 7fde277fe640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde28157b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e7a0 tx=0x7fde1800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.273+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1800eeb0 con 0x7fde140047c0 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.273+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1800c940 con 0x7fde140047c0 2023-08-25T09:55:00.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.273+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1800cd10 con 0x7fde140047c0 2023-08-25T09:55:00.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.273+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde280c2dc0 con 0x7fde140047c0 2023-08-25T09:55:00.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.273+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde281f3040 con 0x7fde140047c0 2023-08-25T09:55:00.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.274+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde280c2f50 con 0x7fde140047c0 2023-08-25T09:55:00.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.276+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fde18013470 con 0x7fde140047c0 2023-08-25T09:55:00.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.277+0000 7fde24ff9640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde0806a080 0x7fde0806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:00.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.277+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fde18088b40 con 0x7fde140047c0 2023-08-25T09:55:00.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.277+0000 7fde1ffff640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde0806a080 0x7fde0806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:00.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.278+0000 7fde1ffff640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde0806a080 0x7fde0806c540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde10034000 tx=0x7fde10009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:00.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.281+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fde18053d80 con 0x7fde140047c0 2023-08-25T09:55:00.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.507+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde28069160 con 0x7fde140047c0 2023-08-25T09:55:00.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.508+0000 7fde24ff9640 1 -- 172.21.15.71:0/3649313880 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7fde18021090 con 0x7fde140047c0 2023-08-25T09:55:00.510 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:00.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:00.511 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:55:00.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde0806a080 msgr2=0x7fde0806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde0806a080 0x7fde0806c540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde10034000 tx=0x7fde10009030 comp rx=0 tx=0).stop 2023-08-25T09:55:00.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 msgr2=0x7fde28157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:00.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde28157b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e7a0 tx=0x7fde1800ec60 comp rx=0 tx=0).stop 2023-08-25T09:55:00.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 shutdown_connections 2023-08-25T09:55:00.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde281f2c00 0x7fde280c35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde0806a080 0x7fde0806c540 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde280c50a0 0x7fde281522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 --2- 172.21.15.71:0/3649313880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde140047c0 0x7fde28157b20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:00.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 >> 172.21.15.71:0/3649313880 conn(0x7fde280654f0 msgr2=0x7fde2815ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:00.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 shutdown_connections 2023-08-25T09:55:00.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:00.511+0000 7fde2e1e8640 1 -- 172.21.15.71:0/3649313880 wait complete. 2023-08-25T09:55:00.530 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-08-25T09:55:02.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:02.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.755+0000 7f7e57df3640 1 Processor -- start 2023-08-25T09:55:02.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.755+0000 7f7e57df3640 1 -- start start 2023-08-25T09:55:02.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.756+0000 7f7e57df3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e500c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e57df3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e500c1dc0 con 0x7f7e500c3a40 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e57df3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e50156e00 con 0x7f7e501509d0 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e57df3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50157a80 con 0x7f7e500c3660 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e5538e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7e500c3a40 0x7f7e5006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39774/0 (socket says 172.21.15.71:39774) 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e5538e640 1 -- 172.21.15.71:0/1001937877 learned_addr learned my addr 172.21.15.71:0/1001937877 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e500c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781032865 0 0) 0x7f7e500c1dc0 con 0x7f7e500c3a40 2023-08-25T09:55:02.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.757+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c003520 con 0x7f7e500c3a40 2023-08-25T09:55:02.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766385164 0 0) 0x7f7e3c003520 con 0x7f7e500c3a40 2023-08-25T09:55:02.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 msgr2=0x7f7e500c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:02.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e500c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:02.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 >> v1:172.21.15.164:6789/0 conn(0x7f7e501509d0 legacy=0x7f7e500c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:02.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e501570e0 con 0x7f7e500c3a40 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f7e40001670 con 0x7f7e500c3a40 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 >> v1:172.21.15.71:6789/0 conn(0x7f7e500c3a40 legacy=0x7f7e5006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c003a10 0x7f7e3c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50157a80 con 0x7f7e3c0047c0 2023-08-25T09:55:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c00bb10 con 0x7f7e3c003a10 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c00bcd0 con 0x7f7e500c3660 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.758+0000 7f7e5538e640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c003a10 0x7f7e3c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e56390640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e5538e640 1 -- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 msgr2=0x7f7e3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e5538e640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:02.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e5538e640 1 -- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c003a10 msgr2=0x7f7e3c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:02.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e5538e640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c003a10 0x7f7e3c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:02.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e5538e640 1 -- 172.21.15.71:0/1001937877 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c003520 con 0x7f7e3c0047c0 2023-08-25T09:55:02.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c003a10 0x7f7e3c003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:02.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.759+0000 7f7e56390640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c3660 0x7f7e3c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:02.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.760+0000 7f7e5538e640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000c240 tx=0x7f7e3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9b463025ce7f0f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:02.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.760+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40005ad0 con 0x7f7e3c0047c0 2023-08-25T09:55:02.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.760+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e40005c30 con 0x7f7e3c0047c0 2023-08-25T09:55:02.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.760+0000 7f7e54b8d640 1 -- 172.21.15.71:0/1001937877 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4002c8b0 con 0x7f7e3c0047c0 2023-08-25T09:55:02.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e57df3640 1 -- 172.21.15.71:0/1001937877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c0047c0 msgr2=0x7f7e3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:02.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e57df3640 1 --2- 172.21.15.71:0/1001937877 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e3c0047c0 0x7f7e3c004c40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000c240 tx=0x7f7e3c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:55:02.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e56390640 1 -- 172.21.15.71:0/1001937877 reap_dead start 2023-08-25T09:55:02.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e57df3640 1 -- 172.21.15.71:0/1001937877 shutdown_connections 2023-08-25T09:55:02.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e57df3640 1 -- 172.21.15.71:0/1001937877 >> 172.21.15.71:0/1001937877 conn(0x7f7e500654f0 msgr2=0x7f7e50065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:02.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e57df3640 1 -- 172.21.15.71:0/1001937877 shutdown_connections 2023-08-25T09:55:02.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.761+0000 7f7e57df3640 1 -- 172.21.15.71:0/1001937877 wait complete. 2023-08-25T09:55:02.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.763+0000 7f7e57df3640 1 Processor -- start 2023-08-25T09:55:02.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.763+0000 7f7e57df3640 1 -- start start 2023-08-25T09:55:02.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e57df3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c0047c0 0x7f7e5014d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e57df3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 0x7f7e5006fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e57df3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c88b0 0x7f7e501e5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e57df3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e501e27b0 con 0x7f7e501e2da0 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e57df3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50153c90 con 0x7f7e3c0047c0 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e57df3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e501e1c20 con 0x7f7e500c88b0 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 0x7f7e5006fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 0x7f7e5006fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59610/0 (socket says 172.21.15.71:59610) 2023-08-25T09:55:02.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 -- 172.21.15.71:0/4139197838 learned_addr learned my addr 172.21.15.71:0/4139197838 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:02.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c0047c0 0x7f7e5014d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e56390640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c88b0 0x7f7e501e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 -- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c88b0 msgr2=0x7f7e501e5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c88b0 0x7f7e501e5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 -- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c0047c0 msgr2=0x7f7e5014d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c0047c0 0x7f7e5014d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e5538e640 1 -- 172.21.15.71:0/4139197838 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e501e1f00 con 0x7f7e501e2da0 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.764+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c0047c0 0x7f7e5014d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:02.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.765+0000 7f7e5538e640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 0x7f7e5006fc30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7e4002c450 tx=0x7f7e40001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:02.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.765+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40039070 con 0x7f7e501e2da0 2023-08-25T09:55:02.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.765+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4002ccc0 con 0x7f7e501e2da0 2023-08-25T09:55:02.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.765+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4003c040 con 0x7f7e501e2da0 2023-08-25T09:55:02.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.765+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e501e21e0 con 0x7f7e501e2da0 2023-08-25T09:55:02.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.765+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e50154210 con 0x7f7e501e2da0 2023-08-25T09:55:02.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.766+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e501543a0 con 0x7f7e501e2da0 2023-08-25T09:55:02.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.767+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7e40032070 con 0x7f7e501e2da0 2023-08-25T09:55:02.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.768+0000 7f7e467fc640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7e3406a050 0x7f7e3406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:02.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.768+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7e400a9d20 con 0x7f7e501e2da0 2023-08-25T09:55:02.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.769+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7e3406a050 0x7f7e3406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:02.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.769+0000 7f7e55b8f640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7e3406a050 0x7f7e3406c510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009aa0 tx=0x7f7e4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:02.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:02.773+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7e40074ff0 con 0x7f7e501e2da0 2023-08-25T09:55:03.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.001+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e50069160 con 0x7f7e501e2da0 2023-08-25T09:55:03.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.002+0000 7f7e467fc640 1 -- 172.21.15.71:0/4139197838 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7f7e40078610 con 0x7f7e501e2da0 2023-08-25T09:55:03.005 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:03.005 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:03.006 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:55:03.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7e3406a050 msgr2=0x7f7e3406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:03.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7e3406a050 0x7f7e3406c510 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009aa0 tx=0x7f7e4c008040 comp rx=0 tx=0).stop 2023-08-25T09:55:03.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 msgr2=0x7f7e5006fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:03.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 0x7f7e5006fc30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7e4002c450 tx=0x7f7e40001a00 comp rx=0 tx=0).stop 2023-08-25T09:55:03.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 shutdown_connections 2023-08-25T09:55:03.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7e3406a050 0x7f7e3406c510 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:03.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e500c88b0 0x7f7e501e5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:03.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7e501e2da0 0x7f7e5006fc30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:03.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 --2- 172.21.15.71:0/4139197838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e3c0047c0 0x7f7e5014d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:03.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.005+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 >> 172.21.15.71:0/4139197838 conn(0x7f7e500654f0 msgr2=0x7f7e500c9660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:03.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.006+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 shutdown_connections 2023-08-25T09:55:03.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:03.006+0000 7f7e57df3640 1 -- 172.21.15.71:0/4139197838 wait complete. 2023-08-25T09:55:03.025 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-08-25T09:55:05.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:05.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.249+0000 7fcf6796b640 1 Processor -- start 2023-08-25T09:55:05.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.249+0000 7fcf6796b640 1 -- start start 2023-08-25T09:55:05.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf6796b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 0x7fcf600c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf6796b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcf601509d0 0x7fcf60150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf6796b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf600c1dc0 con 0x7fcf600c3a40 2023-08-25T09:55:05.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf6796b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf60157a80 con 0x7fcf601509d0 2023-08-25T09:55:05.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf6796b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf601513a0 con 0x7fcf600c3660 2023-08-25T09:55:05.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf64f06640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcf600c3a40 0x7fcf6006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39778/0 (socket says 172.21.15.71:39778) 2023-08-25T09:55:05.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.251+0000 7fcf64f06640 1 -- 172.21.15.71:0/454907781 learned_addr learned my addr 172.21.15.71:0/454907781 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:55:05.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf65707640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 0x7fcf600c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf65f08640 1 --2- 172.21.15.71:0/454907781 >> v2:172.21.15.164:3300/0 conn(0x7fcf601509d0 0x7fcf60150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081721375 0 0) 0x7fcf600c1dc0 con 0x7fcf600c3a40 2023-08-25T09:55:05.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf4c003520 con 0x7fcf600c3a40 2023-08-25T09:55:05.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1514177315 0 0) 0x7fcf4c003520 con 0x7fcf600c3a40 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 msgr2=0x7fcf600c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 0x7fcf600c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 >> v2:172.21.15.164:3300/0 conn(0x7fcf601509d0 msgr2=0x7fcf60150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 --2- 172.21.15.71:0/454907781 >> v2:172.21.15.164:3300/0 conn(0x7fcf601509d0 0x7fcf60150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf60156e00 con 0x7fcf600c3a40 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fcf50001670 con 0x7fcf600c3a40 2023-08-25T09:55:05.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 >> v1:172.21.15.71:6789/0 conn(0x7fcf600c3a40 legacy=0x7fcf6006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf4c0039e0 0x7fcf4c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf4c0047c0 0x7fcf4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 0x7fcf4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf60157a80 con 0x7fcf4c0047c0 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf601513a0 con 0x7fcf4c0039e0 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.252+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c00ba30 con 0x7fcf600c3660 2023-08-25T09:55:05.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf64f06640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf4c0047c0 0x7fcf4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf65f08640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 0x7fcf4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf65707640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf4c0039e0 0x7fcf4c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf64f06640 1 -- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 msgr2=0x7fcf4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf64f06640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf600c3660 0x7fcf4c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf64f06640 1 -- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf4c0039e0 msgr2=0x7fcf4c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf64f06640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf4c0039e0 0x7fcf4c003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.253+0000 7fcf64f06640 1 -- 172.21.15.71:0/454907781 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c003520 con 0x7fcf4c0047c0 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.254+0000 7fcf64f06640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf4c0047c0 0x7fcf4c004c40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c110 tx=0x7fcf4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30907b74f8f143ef server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.254+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50005ad0 con 0x7fcf4c0047c0 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.254+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf50005c50 con 0x7fcf4c0047c0 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.254+0000 7fcf57fff640 1 -- 172.21.15.71:0/454907781 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5002c8b0 con 0x7fcf4c0047c0 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf6796b640 1 -- 172.21.15.71:0/454907781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf4c0047c0 msgr2=0x7fcf4c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf6796b640 1 --2- 172.21.15.71:0/454907781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf4c0047c0 0x7fcf4c004c40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c110 tx=0x7fcf4c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf65f08640 1 -- 172.21.15.71:0/454907781 reap_dead start 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf6796b640 1 -- 172.21.15.71:0/454907781 shutdown_connections 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf6796b640 1 -- 172.21.15.71:0/454907781 >> 172.21.15.71:0/454907781 conn(0x7fcf600654f0 msgr2=0x7fcf60065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf6796b640 1 -- 172.21.15.71:0/454907781 shutdown_connections 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.255+0000 7fcf6796b640 1 -- 172.21.15.71:0/454907781 wait complete. 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.257+0000 7fcf6796b640 1 Processor -- start 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.257+0000 7fcf6796b640 1 -- start start 2023-08-25T09:55:05.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf6796b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf4c0047c0 0x7fcf60151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf6796b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6015a9d0 0x7fcf60157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf6796b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 0x7fcf60158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf6796b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6006a8a0 con 0x7fcf60158360 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf6796b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf601538a0 con 0x7fcf6015a9d0 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf6796b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf60154400 con 0x7fcf4c0047c0 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf65f08640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 0x7fcf60158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf65f08640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 0x7fcf60158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59632/0 (socket says 172.21.15.71:59632) 2023-08-25T09:55:05.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.258+0000 7fcf65f08640 1 -- 172.21.15.71:0/1211323938 learned_addr learned my addr 172.21.15.71:0/1211323938 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:05.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf64f06640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6015a9d0 0x7fcf60157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65707640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf4c0047c0 0x7fcf60151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65f08640 1 -- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf4c0047c0 msgr2=0x7fcf60151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:05.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65f08640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf4c0047c0 0x7fcf60151d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65f08640 1 -- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6015a9d0 msgr2=0x7fcf60157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:05.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65f08640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6015a9d0 0x7fcf60157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65f08640 1 -- 172.21.15.71:0/1211323938 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf60152fb0 con 0x7fcf60158360 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf64f06640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6015a9d0 0x7fcf60157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf65f08640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 0x7fcf60158af0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcf60150db0 tx=0x7fcf5800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.259+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58016620 con 0x7fcf60158360 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.260+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf580167a0 con 0x7fcf60158360 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.260+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58016b50 con 0x7fcf60158360 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.260+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6006aab0 con 0x7fcf60158360 2023-08-25T09:55:05.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.260+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf60153250 con 0x7fcf60158360 2023-08-25T09:55:05.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.261+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf600c8cd0 con 0x7fcf60158360 2023-08-25T09:55:05.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.262+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcf58016cb0 con 0x7fcf60158360 2023-08-25T09:55:05.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.263+0000 7fcf55ffb640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf4406a050 0x7fcf4406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:05.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.263+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fcf58089870 con 0x7fcf60158360 2023-08-25T09:55:05.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.263+0000 7fcf65707640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf4406a050 0x7fcf4406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:05.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.264+0000 7fcf65707640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf4406a050 0x7fcf4406c510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009990 tx=0x7fcf5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:05.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.267+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fcf58054c70 con 0x7fcf60158360 2023-08-25T09:55:05.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.492+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf60069160 con 0x7fcf60158360 2023-08-25T09:55:05.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.494+0000 7fcf55ffb640 1 -- 172.21.15.71:0/1211323938 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+6542 (secure 0 0 0) 0x7fcf58058290 con 0x7fcf60158360 2023-08-25T09:55:05.497 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:05.497 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4810,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:54:47.837562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"b","rank":4,"incarnation":22,"state":"up:active","state_seq":54,"addr":"172.21.15.164:6833/2402500458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":2402500458},{"type":"v1","addr":"172.21.15.164:6833","nonce":2402500458}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:05.498 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 33 2023-08-25T09:55:05.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf4406a050 msgr2=0x7fcf4406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:05.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf4406a050 0x7fcf4406c510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009990 tx=0x7fcf5c008040 comp rx=0 tx=0).stop 2023-08-25T09:55:05.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 msgr2=0x7fcf60158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:05.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 0x7fcf60158af0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcf60150db0 tx=0x7fcf5800f520 comp rx=0 tx=0).stop 2023-08-25T09:55:05.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 shutdown_connections 2023-08-25T09:55:05.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf6015a9d0 0x7fcf60157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf4406a050 0x7fcf4406c510 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf60158360 0x7fcf60158af0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 --2- 172.21.15.71:0/1211323938 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf4c0047c0 0x7fcf60151d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:05.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 >> 172.21.15.71:0/1211323938 conn(0x7fcf600654f0 msgr2=0x7fcf600c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:05.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 shutdown_connections 2023-08-25T09:55:05.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:05.496+0000 7fcf6796b640 1 -- 172.21.15.71:0/1211323938 wait complete. 2023-08-25T09:55:05.518 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-08-25T09:55:07.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:07.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.744+0000 7f3064a1f640 1 Processor -- start 2023-08-25T09:55:07.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.745+0000 7f3064a1f640 1 -- start start 2023-08-25T09:55:07.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.746+0000 7f3064a1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f30600c4560 0x7f306006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:07.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.746+0000 7f3064a1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306006eb60 con 0x7f3060152d80 2023-08-25T09:55:07.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.746+0000 7f3064a1f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306006ca30 con 0x7f3060070cf0 2023-08-25T09:55:07.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.746+0000 7f3064a1f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306006cf90 con 0x7f30600c4560 2023-08-25T09:55:07.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305e59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3060152d80 0x7f30600c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39788/0 (socket says 172.21.15.71:39788) 2023-08-25T09:55:07.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305e59c640 1 -- 172.21.15.71:0/3804182307 learned_addr learned my addr 172.21.15.71:0/3804182307 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:55:07.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305dd9b640 1 --2- 172.21.15.71:0/3804182307 >> v2:172.21.15.71:3300/0 conn(0x7f30600c4560 0x7f306006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:07.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305dd9b640 1 -- 172.21.15.71:0/3804182307 >> v1:172.21.15.164:6789/0 conn(0x7f3060070cf0 legacy=0x7f30601637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:07.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305dd9b640 1 -- 172.21.15.71:0/3804182307 >> v1:172.21.15.71:6789/0 conn(0x7f3060152d80 legacy=0x7f30600c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:07.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305dd9b640 1 -- 172.21.15.71:0/3804182307 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3060158fd0 con 0x7f30600c4560 2023-08-25T09:55:07.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.747+0000 7f305d59a640 1 -- 172.21.15.71:0/3804182307 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718207182 0 0) 0x7f306006eb60 con 0x7f3060152d80 2023-08-25T09:55:07.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.748+0000 7f305dd9b640 1 --2- 172.21.15.71:0/3804182307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306006e620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f30540097a0 tx=0x7f305402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10541ccbc346390 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:07.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f305d59a640 1 -- 172.21.15.71:0/3804182307 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305402e070 con 0x7f30600c4560 2023-08-25T09:55:07.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f305d59a640 1 -- 172.21.15.71:0/3804182307 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30540315a0 con 0x7f30600c4560 2023-08-25T09:55:07.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f305d59a640 1 -- 172.21.15.71:0/3804182307 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054031930 con 0x7f30600c4560 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 -- 172.21.15.71:0/3804182307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 msgr2=0x7f306006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 --2- 172.21.15.71:0/3804182307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306006e620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f30540097a0 tx=0x7f305402b840 comp rx=0 tx=0).stop 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 -- 172.21.15.71:0/3804182307 shutdown_connections 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 --2- 172.21.15.71:0/3804182307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306006e620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 -- 172.21.15.71:0/3804182307 >> 172.21.15.71:0/3804182307 conn(0x7f30600654f0 msgr2=0x7f3060065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 -- 172.21.15.71:0/3804182307 shutdown_connections 2023-08-25T09:55:07.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.749+0000 7f3064a1f640 1 -- 172.21.15.71:0/3804182307 wait complete. 2023-08-25T09:55:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.751+0000 7f3064a1f640 1 Processor -- start 2023-08-25T09:55:07.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.751+0000 7f3064a1f640 1 -- start start 2023-08-25T09:55:07.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f3064a1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3060070cf0 0x7f3060147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:07.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f3064a1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306014caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:07.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f3064a1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3060152d80 0x7f306014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f3064a1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3060156430 con 0x7f30600c4560 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f3064a1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306014bd30 con 0x7f3060152d80 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f3064a1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306014c660 con 0x7f3060070cf0 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.752+0000 7f305dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306014caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306014caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59644/0 (socket says 172.21.15.71:59644) 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 -- 172.21.15.71:0/2701897710 learned_addr learned my addr 172.21.15.71:0/2701897710 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:07.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305e59c640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3060070cf0 0x7f3060147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:07.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305ed9d640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3060152d80 0x7f306014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:07.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 -- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3060070cf0 msgr2=0x7f3060147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:07.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3060070cf0 0x7f3060147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 -- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3060152d80 msgr2=0x7f306014d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:07.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3060152d80 0x7f306014d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 -- 172.21.15.71:0/2701897710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3054009400 con 0x7f30600c4560 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305e59c640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3060070cf0 0x7f3060147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.753+0000 7f305dd9b640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306014caf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f30540097a0 tx=0x7f30540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.754+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054032840 con 0x7f30600c4560 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.754+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30540329a0 con 0x7f30600c4560 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.754+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054032d50 con 0x7f30600c4560 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.754+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30601475c0 con 0x7f30600c4560 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.754+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3060154700 con 0x7f30600c4560 2023-08-25T09:55:07.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.755+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306014c810 con 0x7f30600c4560 2023-08-25T09:55:07.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.756+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f305404d3f0 con 0x7f30600c4560 2023-08-25T09:55:07.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.757+0000 7f304effd640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f304006a050 0x7f304006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:07.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.757+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f30540aa740 con 0x7f30600c4560 2023-08-25T09:55:07.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.757+0000 7f305e59c640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f304006a050 0x7f304006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:07.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.758+0000 7f305e59c640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f304006a050 0x7f304006c510 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3050000f60 tx=0x7f3050009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:07.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.761+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f306014c810 con 0x7f30600c4560 2023-08-25T09:55:07.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.986+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f306014c7d0 con 0x7f30600c4560 2023-08-25T09:55:07.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.987+0000 7f304effd640 1 -- 172.21.15.71:0/2701897710 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f30540759b0 con 0x7f30600c4560 2023-08-25T09:55:07.990 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:07.990 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:05.614506+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:07.991 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 34 2023-08-25T09:55:07.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.989+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f304006a050 msgr2=0x7f304006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f304006a050 0x7f304006c510 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3050000f60 tx=0x7f3050009040 comp rx=0 tx=0).stop 2023-08-25T09:55:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 msgr2=0x7f306014caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306014caf0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f30540097a0 tx=0x7f30540046c0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 shutdown_connections 2023-08-25T09:55:07.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3060152d80 0x7f306014d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f304006a050 0x7f304006c510 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f30600c4560 0x7f306014caf0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 --2- 172.21.15.71:0/2701897710 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3060070cf0 0x7f3060147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 >> 172.21.15.71:0/2701897710 conn(0x7f30600654f0 msgr2=0x7f306006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 shutdown_connections 2023-08-25T09:55:07.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:07.990+0000 7f3064a1f640 1 -- 172.21.15.71:0/2701897710 wait complete. 2023-08-25T09:55:08.010 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-08-25T09:55:08.010 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-08-25T09:55:08.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:08.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.217+0000 7fb6b24fa640 1 Processor -- start 2023-08-25T09:55:08.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.217+0000 7fb6b24fa640 1 -- start start 2023-08-25T09:55:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.218+0000 7fb6b24fa640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb6ac0c3660 0x7fb6ac0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.218+0000 7fb6b24fa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb6ac1509d0 0x7fb6ac150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.218+0000 7fb6b24fa640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6ac0c1dc0 con 0x7fb6ac0c3a40 2023-08-25T09:55:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.218+0000 7fb6b24fa640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac157a80 con 0x7fb6ac1509d0 2023-08-25T09:55:08.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.218+0000 7fb6b24fa640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6ac151320 con 0x7fb6ac0c3660 2023-08-25T09:55:08.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb6ac1509d0 0x7fb6ac150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:08.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb6ac1509d0 0x7fb6ac150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59656/0 (socket says 172.21.15.71:59656) 2023-08-25T09:55:08.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 -- 172.21.15.71:0/737834775 learned_addr learned my addr 172.21.15.71:0/737834775 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:08.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6abfff640 1 --2- 172.21.15.71:0/737834775 >> v2:172.21.15.164:3300/0 conn(0x7fb6ac0c3660 0x7fb6ac0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:08.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 -- 172.21.15.71:0/737834775 >> v2:172.21.15.164:3300/0 conn(0x7fb6ac0c3660 msgr2=0x7fb6ac0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:08.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 --2- 172.21.15.71:0/737834775 >> v2:172.21.15.164:3300/0 conn(0x7fb6ac0c3660 0x7fb6ac0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 -- 172.21.15.71:0/737834775 >> v1:172.21.15.164:6789/0 conn(0x7fb6ac0c3a40 legacy=0x7fb6ac06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:08.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.219+0000 7fb6b0a97640 1 -- 172.21.15.71:0/737834775 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac156e00 con 0x7fb6ac1509d0 2023-08-25T09:55:08.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.220+0000 7fb6b0a97640 1 --2- 172.21.15.71:0/737834775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac1509d0 0x7fb6ac150de0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac066440 tx=0x7fb6a402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd92db16f2bd7af server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:08.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6ab7fe640 1 -- 172.21.15.71:0/737834775 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a402cb20 con 0x7fb6ac1509d0 2023-08-25T09:55:08.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6ab7fe640 1 -- 172.21.15.71:0/737834775 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a402cc80 con 0x7fb6ac1509d0 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6ab7fe640 1 -- 172.21.15.71:0/737834775 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a40343f0 con 0x7fb6ac1509d0 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 -- 172.21.15.71:0/737834775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac1509d0 msgr2=0x7fb6ac150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/737834775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac1509d0 0x7fb6ac150de0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac066440 tx=0x7fb6a402bf80 comp rx=0 tx=0).stop 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 -- 172.21.15.71:0/737834775 shutdown_connections 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/737834775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac1509d0 0x7fb6ac150de0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/737834775 >> v2:172.21.15.164:3300/0 conn(0x7fb6ac0c3660 0x7fb6ac0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 -- 172.21.15.71:0/737834775 >> 172.21.15.71:0/737834775 conn(0x7fb6ac0654f0 msgr2=0x7fb6ac065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:08.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.221+0000 7fb6b24fa640 1 -- 172.21.15.71:0/737834775 shutdown_connections 2023-08-25T09:55:08.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.222+0000 7fb6b24fa640 1 -- 172.21.15.71:0/737834775 wait complete. 2023-08-25T09:55:08.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.223+0000 7fb6b24fa640 1 Processor -- start 2023-08-25T09:55:08.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.223+0000 7fb6b24fa640 1 -- start start 2023-08-25T09:55:08.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.224+0000 7fb6b24fa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 0x7fb6ac1ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:08.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.224+0000 7fb6b24fa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac0c3a40 0x7fb6ac1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:08.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.224+0000 7fb6b24fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac1509d0 0x7fb6ac142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:08.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.224+0000 7fb6b24fa640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1eec60 con 0x7fb6ac0c3660 2023-08-25T09:55:08.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.224+0000 7fb6b24fa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1ef5c0 con 0x7fb6ac0c3a40 2023-08-25T09:55:08.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.224+0000 7fb6b24fa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1f2a50 con 0x7fb6ac1509d0 2023-08-25T09:55:08.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 0x7fb6ac1ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 0x7fb6ac1ef080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59662/0 (socket says 172.21.15.71:59662) 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 -- 172.21.15.71:0/2177567313 learned_addr learned my addr 172.21.15.71:0/2177567313 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6a3fff640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac0c3a40 0x7fb6ac1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 -- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac1509d0 msgr2=0x7fb6ac142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac1509d0 0x7fb6ac142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 -- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac0c3a40 msgr2=0x7fb6ac1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac0c3a40 0x7fb6ac1423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 -- 172.21.15.71:0/2177567313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4009b30 con 0x7fb6ac0c3660 2023-08-25T09:55:08.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6abfff640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 0x7fb6ac1ef080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00d620 tx=0x7fb69c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:08.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.225+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0125f0 con 0x7fb6ac0c3660 2023-08-25T09:55:08.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.226+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69c012750 con 0x7fb6ac0c3660 2023-08-25T09:55:08.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.226+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c012b00 con 0x7fb6ac0c3660 2023-08-25T09:55:08.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.226+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac1ee600 con 0x7fb6ac0c3660 2023-08-25T09:55:08.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.226+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac065ad0 con 0x7fb6ac0c3660 2023-08-25T09:55:08.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.227+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac0c29c0 con 0x7fb6ac0c3660 2023-08-25T09:55:08.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.228+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb69c012c60 con 0x7fb6ac0c3660 2023-08-25T09:55:08.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.229+0000 7fb6a97fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb68c06a080 0x7fb68c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:08.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.229+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fb69c088e30 con 0x7fb6ac0c3660 2023-08-25T09:55:08.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.229+0000 7fb6a3fff640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb68c06a080 0x7fb68c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:08.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.230+0000 7fb6a3fff640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb68c06a080 0x7fb68c06c540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac1562e0 tx=0x7fb694009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:08.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.233+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb69c0540a0 con 0x7fb6ac0c3660 2023-08-25T09:55:08.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.460+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6ac0c93a0 con 0x7fb6ac0c3660 2023-08-25T09:55:08.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.461+0000 7fb6a97fa640 1 -- 172.21.15.71:0/2177567313 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fb69c0576c0 con 0x7fb6ac0c3660 2023-08-25T09:55:08.463 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:08.464 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:05.614506+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:08.464 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 34 2023-08-25T09:55:08.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb68c06a080 msgr2=0x7fb68c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:08.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb68c06a080 0x7fb68c06c540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac1562e0 tx=0x7fb694009040 comp rx=0 tx=0).stop 2023-08-25T09:55:08.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 msgr2=0x7fb6ac1ef080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 0x7fb6ac1ef080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00d620 tx=0x7fb69c00dae0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 shutdown_connections 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac0c3a40 0x7fb6ac1423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb68c06a080 0x7fb68c06c540 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac1509d0 0x7fb6ac142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 --2- 172.21.15.71:0/2177567313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ac0c3660 0x7fb6ac1ef080 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 >> 172.21.15.71:0/2177567313 conn(0x7fb6ac0654f0 msgr2=0x7fb6ac0c9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:08.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 shutdown_connections 2023-08-25T09:55:08.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:08.463+0000 7fb6b24fa640 1 -- 172.21.15.71:0/2177567313 wait complete. 2023-08-25T09:55:08.485 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4287": { "addr": "172.21.15.71:6836/1556118065", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 1556118065, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 1556118065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4287, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_4290": { "addr": "172.21.15.71:6837/3404832976", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3404832976, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3404832976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:55:05.614506+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 4290, "mds_3": 4287, "mds_4": 4810 } } } ], "standbys": [] } 2023-08-25T09:55:10.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:10.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.709+0000 7f29dfacc640 1 Processor -- start 2023-08-25T09:55:10.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.710+0000 7f29dfacc640 1 -- start start 2023-08-25T09:55:10.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dfacc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f29d81530c0 0x7f29d80c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:10.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dfacc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:10.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dfacc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d8158e40 con 0x7f29d806aea0 2023-08-25T09:55:10.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dfacc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d806bc80 con 0x7f29d81530c0 2023-08-25T09:55:10.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dfacc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d81534a0 con 0x7f29d80c7740 2023-08-25T09:55:10.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dd868640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f29d81530c0 0x7f29d80c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:10.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dd868640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f29d81530c0 0x7f29d80c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50010/0 (socket says 172.21.15.71:50010) 2023-08-25T09:55:10.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.711+0000 7f29dd868640 1 -- 172.21.15.71:0/3882213836 learned_addr learned my addr 172.21.15.71:0/3882213836 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:10.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.712+0000 7f29dd067640 1 --2- 172.21.15.71:0/3882213836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:10.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.712+0000 7f29dd868640 1 -- 172.21.15.71:0/3882213836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 msgr2=0x7f29d806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:10.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.712+0000 7f29dd868640 1 --2- 172.21.15.71:0/3882213836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.712+0000 7f29dd868640 1 -- 172.21.15.71:0/3882213836 >> v1:172.21.15.164:6789/0 conn(0x7f29d806aea0 legacy=0x7f29d81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:10.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.712+0000 7f29dd868640 1 -- 172.21.15.71:0/3882213836 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80c7380 con 0x7f29d81530c0 2023-08-25T09:55:10.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.713+0000 7f29dd868640 1 --2- 172.21.15.71:0/3882213836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d80c6e40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009870 tx=0x7f29d402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ee4001559f9f22 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:10.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.713+0000 7f29dc866640 1 -- 172.21.15.71:0/3882213836 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d402f840 con 0x7f29d81530c0 2023-08-25T09:55:10.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.713+0000 7f29dc866640 1 -- 172.21.15.71:0/3882213836 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d402f9a0 con 0x7f29d81530c0 2023-08-25T09:55:10.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.713+0000 7f29dc866640 1 -- 172.21.15.71:0/3882213836 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d402fd30 con 0x7f29d81530c0 2023-08-25T09:55:10.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 -- 172.21.15.71:0/3882213836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 msgr2=0x7f29d80c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:10.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 --2- 172.21.15.71:0/3882213836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d80c6e40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009870 tx=0x7f29d402ba20 comp rx=0 tx=0).stop 2023-08-25T09:55:10.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 -- 172.21.15.71:0/3882213836 shutdown_connections 2023-08-25T09:55:10.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 --2- 172.21.15.71:0/3882213836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 --2- 172.21.15.71:0/3882213836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d80c6e40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 -- 172.21.15.71:0/3882213836 >> 172.21.15.71:0/3882213836 conn(0x7f29d80654f0 msgr2=0x7f29d8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:10.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 -- 172.21.15.71:0/3882213836 shutdown_connections 2023-08-25T09:55:10.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.714+0000 7f29dfacc640 1 -- 172.21.15.71:0/3882213836 wait complete. 2023-08-25T09:55:10.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.716+0000 7f29dfacc640 1 Processor -- start 2023-08-25T09:55:10.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.716+0000 7f29dfacc640 1 -- start start 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dfacc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29d806aea0 0x7f29d81f18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dfacc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d814a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dfacc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d814af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dfacc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d814a530 con 0x7f29d81530c0 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dfacc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d81f23b0 con 0x7f29d806aea0 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dfacc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d8149c00 con 0x7f29d80c7740 2023-08-25T09:55:10.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29de069640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d814af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:10.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29de069640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d814af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50022/0 (socket says 172.21.15.71:50022) 2023-08-25T09:55:10.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29de069640 1 -- 172.21.15.71:0/148471314 learned_addr learned my addr 172.21.15.71:0/148471314 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:10.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.718+0000 7f29dd868640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29d806aea0 0x7f29d81f18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:10.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29dd067640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d814a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:10.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29de069640 1 -- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 msgr2=0x7f29d814a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:10.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29de069640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d814a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29de069640 1 -- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29d806aea0 msgr2=0x7f29d81f18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29de069640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29d806aea0 0x7f29d81f18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29de069640 1 -- 172.21.15.71:0/148471314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d40094d0 con 0x7f29d81530c0 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29de069640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d814af00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f29d000c6b0 tx=0x7f29d000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0003b70 con 0x7f29d81530c0 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d0003cf0 con 0x7f29d81530c0 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d8149ee0 con 0x7f29d81530c0 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.719+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0003ea0 con 0x7f29d81530c0 2023-08-25T09:55:10.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.720+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d80cd0a0 con 0x7f29d81530c0 2023-08-25T09:55:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.721+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d814a6e0 con 0x7f29d81530c0 2023-08-25T09:55:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.722+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f29d0005310 con 0x7f29d81530c0 2023-08-25T09:55:10.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.723+0000 7f29ce7fc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f29bc06a050 0x7f29bc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.723+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f29d00886d0 con 0x7f29d81530c0 2023-08-25T09:55:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.723+0000 7f29dd868640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f29bc06a050 0x7f29bc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:10.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.724+0000 7f29dd868640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f29bc06a050 0x7f29bc06c510 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036000 tx=0x7f29d4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:10.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.728+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f29d814a6e0 con 0x7f29d81530c0 2023-08-25T09:55:10.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.963+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d80cd2f0 con 0x7f29d81530c0 2023-08-25T09:55:10.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.964+0000 7f29ce7fc640 1 -- 172.21.15.71:0/148471314 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f29d00539d0 con 0x7f29d81530c0 2023-08-25T09:55:10.967 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:10.967 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:05.614506+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:10.968 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 34 2023-08-25T09:55:10.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f29bc06a050 msgr2=0x7f29bc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f29bc06a050 0x7f29bc06c510 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f29d4036000 tx=0x7f29d4008030 comp rx=0 tx=0).stop 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 msgr2=0x7f29d814af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d814af00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f29d000c6b0 tx=0x7f29d000cb70 comp rx=0 tx=0).stop 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 shutdown_connections 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f29bc06a050 0x7f29bc06c510 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29d81530c0 0x7f29d814af00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f29d80c7740 0x7f29d814a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 --2- 172.21.15.71:0/148471314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29d806aea0 0x7f29d81f18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 >> 172.21.15.71:0/148471314 conn(0x7f29d80654f0 msgr2=0x7f29d8151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 shutdown_connections 2023-08-25T09:55:10.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:10.967+0000 7f29dfacc640 1 -- 172.21.15.71:0/148471314 wait complete. 2023-08-25T09:55:10.987 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:55:12.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:13.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.208+0000 7f068c004640 1 Processor -- start 2023-08-25T09:55:13.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.208+0000 7f068c004640 1 -- start start 2023-08-25T09:55:13.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.210+0000 7f068c004640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0684152be0 0x7f068406aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:13.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.210+0000 7f068c004640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f06840c4c10 0x7f0684069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:13.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.210+0000 7f068c004640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684159730 con 0x7f0684069d90 2023-08-25T09:55:13.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.210+0000 7f068c004640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0684159c40 con 0x7f06840c4c10 2023-08-25T09:55:13.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.210+0000 7f068c004640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068415d190 con 0x7f0684152be0 2023-08-25T09:55:13.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f06840c4c10 0x7f0684069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:13.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f0689da0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0684152be0 0x7f068406aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:13.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f06840c4c10 0x7f0684069850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:40166/0 (socket says 172.21.15.71:40166) 2023-08-25T09:55:13.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 -- 172.21.15.71:0/2979337649 learned_addr learned my addr 172.21.15.71:0/2979337649 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:13.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 -- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0684152be0 msgr2=0x7f068406aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:13.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 --2- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0684152be0 0x7f068406aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 -- 172.21.15.71:0/2979337649 >> v1:172.21.15.164:6789/0 conn(0x7f0684069d90 legacy=0x7f06840c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:13.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.211+0000 7f068959f640 1 -- 172.21.15.71:0/2979337649 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684159960 con 0x7f06840c4c10 2023-08-25T09:55:13.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.213+0000 7f068959f640 1 --2- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06840c4c10 0x7f0684069850 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291f6cb1f7b52eef server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:13.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.213+0000 7f0688d9e640 1 -- 172.21.15.71:0/2979337649 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674030840 con 0x7f06840c4c10 2023-08-25T09:55:13.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.214+0000 7f0688d9e640 1 -- 172.21.15.71:0/2979337649 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06740309a0 con 0x7f06840c4c10 2023-08-25T09:55:13.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.214+0000 7f0688d9e640 1 -- 172.21.15.71:0/2979337649 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674030d30 con 0x7f06840c4c10 2023-08-25T09:55:13.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.214+0000 7f068c004640 1 -- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06840c4c10 msgr2=0x7f0684069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:13.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.214+0000 7f068c004640 1 --2- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06840c4c10 0x7f0684069850 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067402b980 comp rx=0 tx=0).stop 2023-08-25T09:55:13.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.214+0000 7f068c004640 1 -- 172.21.15.71:0/2979337649 shutdown_connections 2023-08-25T09:55:13.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.214+0000 7f068c004640 1 --2- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06840c4c10 0x7f0684069850 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.215+0000 7f068c004640 1 --2- 172.21.15.71:0/2979337649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0684152be0 0x7f068406aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.215+0000 7f068c004640 1 -- 172.21.15.71:0/2979337649 >> 172.21.15.71:0/2979337649 conn(0x7f068406bff0 msgr2=0x7f068406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:13.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.215+0000 7f068c004640 1 -- 172.21.15.71:0/2979337649 shutdown_connections 2023-08-25T09:55:13.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.215+0000 7f068c004640 1 -- 172.21.15.71:0/2979337649 wait complete. 2023-08-25T09:55:13.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.217+0000 7f068c004640 1 Processor -- start 2023-08-25T09:55:13.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.217+0000 7f068c004640 1 -- start start 2023-08-25T09:55:13.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068c004640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0684069d90 0x7f068414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:13.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068c004640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06840c4c10 0x7f06840c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:13.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068c004640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 0x7f06841ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:13.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068c004640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068414f9b0 con 0x7f0684152be0 2023-08-25T09:55:13.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068c004640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068414e750 con 0x7f0684069d90 2023-08-25T09:55:13.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068c004640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f068414d1c0 con 0x7f06840c4c10 2023-08-25T09:55:13.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068a5a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 0x7f06841ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:13.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068a5a1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 0x7f06841ef770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50032/0 (socket says 172.21.15.71:50032) 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.218+0000 7f068a5a1640 1 -- 172.21.15.71:0/1956788303 learned_addr learned my addr 172.21.15.71:0/1956788303 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f0689da0640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0684069d90 0x7f068414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068959f640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06840c4c10 0x7f06840c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068a5a1640 1 -- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06840c4c10 msgr2=0x7f06840c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068a5a1640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06840c4c10 0x7f06840c3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068a5a1640 1 -- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0684069d90 msgr2=0x7f068414f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:13.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068a5a1640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0684069d90 0x7f068414f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068a5a1640 1 -- 172.21.15.71:0/1956788303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740094d0 con 0x7f0684152be0 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.219+0000 7f068a5a1640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 0x7f06841ef770 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f067c00c760 tx=0x7f067c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.220+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c003b70 con 0x7f0684152be0 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.220+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c003cd0 con 0x7f0684152be0 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.220+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c00aa90 con 0x7f0684152be0 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.220+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068414cf20 con 0x7f0684152be0 2023-08-25T09:55:13.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.220+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068414e100 con 0x7f0684152be0 2023-08-25T09:55:13.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.221+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068414e900 con 0x7f0684152be0 2023-08-25T09:55:13.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.222+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f067c00abf0 con 0x7f0684152be0 2023-08-25T09:55:13.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.223+0000 7f067a7fc640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f066806a050 0x7f066806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:13.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.223+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f067c088710 con 0x7f0684152be0 2023-08-25T09:55:13.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.223+0000 7f0689da0640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f066806a050 0x7f066806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:13.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.224+0000 7f0689da0640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f066806a050 0x7f066806c510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0680009b30 tx=0x7f0680009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:13.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.227+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f068414e900 con 0x7f0684152be0 2023-08-25T09:55:13.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.457+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f068414e900 con 0x7f0684152be0 2023-08-25T09:55:13.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.458+0000 7f067a7fc640 1 -- 172.21.15.71:0/1956788303 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f068414e900 con 0x7f0684152be0 2023-08-25T09:55:13.461 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:13.461 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:11.620677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:resolve","state_seq":34,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:13.462 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 35 2023-08-25T09:55:13.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f066806a050 msgr2=0x7f066806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:13.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f066806a050 0x7f066806c510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0680009b30 tx=0x7f0680009110 comp rx=0 tx=0).stop 2023-08-25T09:55:13.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 msgr2=0x7f06841ef770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:13.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 0x7f06841ef770 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f067c00c760 tx=0x7f067c00cc20 comp rx=0 tx=0).stop 2023-08-25T09:55:13.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 shutdown_connections 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f066806a050 0x7f066806c510 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0684152be0 0x7f06841ef770 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06840c4c10 0x7f06840c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 --2- 172.21.15.71:0/1956788303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0684069d90 0x7f068414f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 >> 172.21.15.71:0/1956788303 conn(0x7f068406bff0 msgr2=0x7f068406e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 shutdown_connections 2023-08-25T09:55:13.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:13.461+0000 7f068c004640 1 -- 172.21.15.71:0/1956788303 wait complete. 2023-08-25T09:55:13.483 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:55:15.484 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:15.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.709+0000 7f3bcd285640 1 Processor -- start 2023-08-25T09:55:15.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.709+0000 7f3bcd285640 1 -- start start 2023-08-25T09:55:15.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bcd285640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3bc80c72f0 0x7f3bc80acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:15.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bcd285640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80ad2f0 0x7f3bc8155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:15.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bcd285640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc80c1060 con 0x7f3bc80c76d0 2023-08-25T09:55:15.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bcd285640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc80adf70 con 0x7f3bc80c72f0 2023-08-25T09:55:15.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bcd285640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80af730 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bc7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3bc80c72f0 0x7f3bc80acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:15.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bc7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3bc80c72f0 0x7f3bc80acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50034/0 (socket says 172.21.15.71:50034) 2023-08-25T09:55:15.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.711+0000 7f3bc7fff640 1 -- 172.21.15.71:0/3855618232 learned_addr learned my addr 172.21.15.71:0/3855618232 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:15.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.712+0000 7f3bcca84640 1 --2- 172.21.15.71:0/3855618232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80ad2f0 0x7f3bc8155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:15.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.712+0000 7f3bc7fff640 1 -- 172.21.15.71:0/3855618232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80ad2f0 msgr2=0x7f3bc8155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:15.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.712+0000 7f3bc7fff640 1 --2- 172.21.15.71:0/3855618232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80ad2f0 0x7f3bc8155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.712+0000 7f3bc7fff640 1 -- 172.21.15.71:0/3855618232 >> v1:172.21.15.71:6789/0 conn(0x7f3bc80c76d0 legacy=0x7f3bc8152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:15.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.712+0000 7f3bc7fff640 1 -- 172.21.15.71:0/3855618232 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc806b4a0 con 0x7f3bc80c72f0 2023-08-25T09:55:15.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.713+0000 7f3bc7fff640 1 --2- 172.21.15.71:0/3855618232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80c72f0 0x7f3bc80acdb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800bb10 tx=0x7f3bb802e1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b6acb9df34ed05 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:15.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.713+0000 7f3bc6ffd640 1 -- 172.21.15.71:0/3855618232 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8036440 con 0x7f3bc80c72f0 2023-08-25T09:55:15.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.713+0000 7f3bc6ffd640 1 -- 172.21.15.71:0/3855618232 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb80365a0 con 0x7f3bc80c72f0 2023-08-25T09:55:15.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.713+0000 7f3bc6ffd640 1 -- 172.21.15.71:0/3855618232 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8036930 con 0x7f3bc80c72f0 2023-08-25T09:55:15.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 -- 172.21.15.71:0/3855618232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80c72f0 msgr2=0x7f3bc80acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:15.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 --2- 172.21.15.71:0/3855618232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80c72f0 0x7f3bc80acdb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800bb10 tx=0x7f3bb802e1e0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 -- 172.21.15.71:0/3855618232 shutdown_connections 2023-08-25T09:55:15.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 --2- 172.21.15.71:0/3855618232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80ad2f0 0x7f3bc8155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 --2- 172.21.15.71:0/3855618232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80c72f0 0x7f3bc80acdb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 -- 172.21.15.71:0/3855618232 >> 172.21.15.71:0/3855618232 conn(0x7f3bc806c280 msgr2=0x7f3bc806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:15.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 -- 172.21.15.71:0/3855618232 shutdown_connections 2023-08-25T09:55:15.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.714+0000 7f3bcd285640 1 -- 172.21.15.71:0/3855618232 wait complete. 2023-08-25T09:55:15.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.716+0000 7f3bcd285640 1 Processor -- start 2023-08-25T09:55:15.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.716+0000 7f3bcd285640 1 -- start start 2023-08-25T09:55:15.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.717+0000 7f3bcd285640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 0x7f3bc806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:15.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bcd285640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bc80c72f0 0x7f3bc8070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:15.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bcd285640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80c76d0 0x7f3bc80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bcd285640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80c6320 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bcd285640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80712e0 con 0x7f3bc80c72f0 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bcd285640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc806f1e0 con 0x7f3bc80c76d0 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 0x7f3bc806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 0x7f3bc806eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50048/0 (socket says 172.21.15.71:50048) 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 -- 172.21.15.71:0/2495734890 learned_addr learned my addr 172.21.15.71:0/2495734890 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bcca84640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80c76d0 0x7f3bc80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 -- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80c76d0 msgr2=0x7f3bc80aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc77fe640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bc80c72f0 0x7f3bc8070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80c76d0 0x7f3bc80aa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 -- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bc80c72f0 msgr2=0x7f3bc8070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bc80c72f0 0x7f3bc8070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.718+0000 7f3bc7fff640 1 -- 172.21.15.71:0/2495734890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb800b770 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.719+0000 7f3bc7fff640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 0x7f3bc806eca0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8000c00 tx=0x7f3bb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.719+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8036440 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.719+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb8034ba0 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.719+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8041440 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.719+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc81e9ba0 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.719+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc81ea160 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.721+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc8150b60 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.721+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3bb803d040 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.723+0000 7f3bc4ff9640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bb406a080 0x7f3bb406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:15.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.723+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f3bb80ad7e0 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.723+0000 7f3bc77fe640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bb406a080 0x7f3bb406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:15.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.724+0000 7f3bc77fe640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bb406a080 0x7f3bb406c540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8072ad0 tx=0x7f3bbc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:15.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.728+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3bb80789c0 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.958+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bc804e420 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.959+0000 7f3bc4ff9640 1 -- 172.21.15.71:0/2495734890 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f3bb807bfe0 con 0x7f3bc80ad2f0 2023-08-25T09:55:15.963 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:15.963 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:11.620677+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:resolve","state_seq":34,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:15.963 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 35 2023-08-25T09:55:15.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.961+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bb406a080 msgr2=0x7f3bb406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.961+0000 7f3bcd285640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bb406a080 0x7f3bb406c540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8072ad0 tx=0x7f3bbc009040 comp rx=0 tx=0).stop 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.961+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 msgr2=0x7f3bc806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.961+0000 7f3bcd285640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 0x7f3bc806eca0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8000c00 tx=0x7f3bb80046c0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 shutdown_connections 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bc80c72f0 0x7f3bc8070a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3bb406a080 0x7f3bb406c540 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bc80c76d0 0x7f3bc80aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 --2- 172.21.15.71:0/2495734890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3bc80ad2f0 0x7f3bc806eca0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 >> 172.21.15.71:0/2495734890 conn(0x7f3bc806c280 msgr2=0x7f3bc80ac610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 shutdown_connections 2023-08-25T09:55:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:15.962+0000 7f3bcd285640 1 -- 172.21.15.71:0/2495734890 wait complete. 2023-08-25T09:55:15.982 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:55:17.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:18.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.207+0000 7f44b72d3640 1 Processor -- start 2023-08-25T09:55:18.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.207+0000 7f44b72d3640 1 -- start start 2023-08-25T09:55:18.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.208+0000 7f44b72d3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f44b00c3660 0x7f44b00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.208+0000 7f44b72d3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44b00c3a40 0x7f44b006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.208+0000 7f44b72d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b01509d0 0x7f44b0150e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.208+0000 7f44b72d3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b0157a80 con 0x7f44b00c3a40 2023-08-25T09:55:18.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.208+0000 7f44b72d3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b00c1dc0 con 0x7f44b00c3660 2023-08-25T09:55:18.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.208+0000 7f44b72d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b006cf70 con 0x7f44b01509d0 2023-08-25T09:55:18.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44b00c3a40 0x7f44b006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44b00c3a40 0x7f44b006ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50062/0 (socket says 172.21.15.71:50062) 2023-08-25T09:55:18.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 -- 172.21.15.71:0/930076231 learned_addr learned my addr 172.21.15.71:0/930076231 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:18.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b506f640 1 --2- 172.21.15.71:0/930076231 >> v2:172.21.15.164:3300/0 conn(0x7f44b00c3660 0x7f44b00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b5870640 1 --2- 172.21.15.71:0/930076231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b01509d0 0x7f44b0150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 -- 172.21.15.71:0/930076231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b01509d0 msgr2=0x7f44b0150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:18.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 --2- 172.21.15.71:0/930076231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b01509d0 0x7f44b0150e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 -- 172.21.15.71:0/930076231 >> v2:172.21.15.164:3300/0 conn(0x7f44b00c3660 msgr2=0x7f44b00c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:18.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 --2- 172.21.15.71:0/930076231 >> v2:172.21.15.164:3300/0 conn(0x7f44b00c3660 0x7f44b00c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.209+0000 7f44b486e640 1 -- 172.21.15.71:0/930076231 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0151340 con 0x7f44b00c3a40 2023-08-25T09:55:18.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.210+0000 7f44b486e640 1 --2- 172.21.15.71:0/930076231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b006ca30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f44b0066440 tx=0x7f44a002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1e43519d35ddfb server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:18.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44a7fff640 1 -- 172.21.15.71:0/930076231 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0030850 con 0x7f44b00c3a40 2023-08-25T09:55:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44a7fff640 1 -- 172.21.15.71:0/930076231 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a00309b0 con 0x7f44b00c3a40 2023-08-25T09:55:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44a7fff640 1 -- 172.21.15.71:0/930076231 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0030d40 con 0x7f44b00c3a40 2023-08-25T09:55:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 -- 172.21.15.71:0/930076231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 msgr2=0x7f44b006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 --2- 172.21.15.71:0/930076231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b006ca30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f44b0066440 tx=0x7f44a002b990 comp rx=0 tx=0).stop 2023-08-25T09:55:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 -- 172.21.15.71:0/930076231 shutdown_connections 2023-08-25T09:55:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 --2- 172.21.15.71:0/930076231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b01509d0 0x7f44b0150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 --2- 172.21.15.71:0/930076231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b006ca30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 --2- 172.21.15.71:0/930076231 >> v2:172.21.15.164:3300/0 conn(0x7f44b00c3660 0x7f44b00c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 -- 172.21.15.71:0/930076231 >> 172.21.15.71:0/930076231 conn(0x7f44b00654f0 msgr2=0x7f44b0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 -- 172.21.15.71:0/930076231 shutdown_connections 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.211+0000 7f44b72d3640 1 -- 172.21.15.71:0/930076231 wait complete. 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.213+0000 7f44b72d3640 1 Processor -- start 2023-08-25T09:55:18.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.213+0000 7f44b72d3640 1 -- start start 2023-08-25T09:55:18.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b72d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b00c3660 0x7f44b015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b72d3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b0141f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b72d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44b01509d0 0x7f44b01424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b72d3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b015a500 con 0x7f44b00c3a40 2023-08-25T09:55:18.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b72d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b014a5e0 con 0x7f44b01509d0 2023-08-25T09:55:18.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b72d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0159d20 con 0x7f44b00c3660 2023-08-25T09:55:18.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b486e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b0141f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b486e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b0141f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50068/0 (socket says 172.21.15.71:50068) 2023-08-25T09:55:18.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.214+0000 7f44b486e640 1 -- 172.21.15.71:0/1459186172 learned_addr learned my addr 172.21.15.71:0/1459186172 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:18.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b506f640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b00c3660 0x7f44b015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b5870640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44b01509d0 0x7f44b01424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b486e640 1 -- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b00c3660 msgr2=0x7f44b015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:18.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b486e640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b00c3660 0x7f44b015a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b486e640 1 -- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44b01509d0 msgr2=0x7f44b01424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:18.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b486e640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44b01509d0 0x7f44b01424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b486e640 1 -- 172.21.15.71:0/1459186172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a00094d0 con 0x7f44b00c3a40 2023-08-25T09:55:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b506f640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b00c3660 0x7f44b015a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b486e640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b0141f70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000c00 tx=0x7f44a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0030840 con 0x7f44b00c3a40 2023-08-25T09:55:18.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0039d90 con 0x7f44b00c3a40 2023-08-25T09:55:18.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0043650 con 0x7f44b00c3a40 2023-08-25T09:55:18.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.215+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b0149a20 con 0x7f44b00c3a40 2023-08-25T09:55:18.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.216+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b014ab10 con 0x7f44b00c3a40 2023-08-25T09:55:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.217+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b00c29c0 con 0x7f44b00c3a40 2023-08-25T09:55:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.218+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f44a002e070 con 0x7f44b00c3a40 2023-08-25T09:55:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.218+0000 7f44a5ffb640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449406a050 0x7f449406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.218+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f44a00aaf30 con 0x7f44b00c3a40 2023-08-25T09:55:18.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.219+0000 7f44b506f640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449406a050 0x7f449406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.219+0000 7f44b506f640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449406a050 0x7f449406c510 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00b820 tx=0x7f44ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.223+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f44a00762c0 con 0x7f44b00c3a40 2023-08-25T09:55:18.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.451+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44b015a640 con 0x7f44b00c3a40 2023-08-25T09:55:18.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.452+0000 7f44a5ffb640 1 -- 172.21.15.71:0/1459186172 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f44b015a640 con 0x7f44b00c3a40 2023-08-25T09:55:18.455 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:18.455 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:16.841367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:18.456 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 36 2023-08-25T09:55:18.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449406a050 msgr2=0x7f449406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:18.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449406a050 0x7f449406c510 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00b820 tx=0x7f44ac008040 comp rx=0 tx=0).stop 2023-08-25T09:55:18.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 msgr2=0x7f44b0141f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:18.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b0141f70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000c00 tx=0x7f44a0004730 comp rx=0 tx=0).stop 2023-08-25T09:55:18.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 shutdown_connections 2023-08-25T09:55:18.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44b01509d0 0x7f44b01424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449406a050 0x7f449406c510 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44b00c3a40 0x7f44b0141f70 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 --2- 172.21.15.71:0/1459186172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44b00c3660 0x7f44b015a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:18.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 >> 172.21.15.71:0/1459186172 conn(0x7f44b00654f0 msgr2=0x7f44b00c40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:18.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 shutdown_connections 2023-08-25T09:55:18.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:18.455+0000 7f44b72d3640 1 -- 172.21.15.71:0/1459186172 wait complete. 2023-08-25T09:55:18.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:55:20.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:20.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.703+0000 7f6b0289b640 1 Processor -- start 2023-08-25T09:55:20.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.703+0000 7f6b0289b640 1 -- start start 2023-08-25T09:55:20.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6b0289b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6afc0c3a40 0x7f6afc0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:20.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6b0289b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:20.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6b0289b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc0c1dc0 con 0x7f6afc0c3660 2023-08-25T09:55:20.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6b0289b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6afc157a80 con 0x7f6afc0c3a40 2023-08-25T09:55:20.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6b0289b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc151360 con 0x7f6afc1509d0 2023-08-25T09:55:20.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6afc0c3a40 0x7f6afc0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:20.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6b00e38640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:20.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6afc0c3a40 0x7f6afc0c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:42064/0 (socket says 172.21.15.71:42064) 2023-08-25T09:55:20.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 -- 172.21.15.71:0/3778883707 learned_addr learned my addr 172.21.15.71:0/3778883707 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:20.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 -- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 msgr2=0x7f6afc150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:20.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 --2- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 -- 172.21.15.71:0/3778883707 >> v1:172.21.15.164:6789/0 conn(0x7f6afc0c3660 legacy=0x7f6afc06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:20.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.706+0000 7f6afb7fe640 1 -- 172.21.15.71:0/3778883707 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc156e00 con 0x7f6afc0c3a40 2023-08-25T09:55:20.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.708+0000 7f6afb7fe640 1 --2- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc0c1710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009870 tx=0x7f6af002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dfdb4975ec110e4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:20.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.708+0000 7f6afaffd640 1 -- 172.21.15.71:0/3778883707 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af002f830 con 0x7f6afc0c3a40 2023-08-25T09:55:20.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.708+0000 7f6afaffd640 1 -- 172.21.15.71:0/3778883707 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af002f990 con 0x7f6afc0c3a40 2023-08-25T09:55:20.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.708+0000 7f6afaffd640 1 -- 172.21.15.71:0/3778883707 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af002fd20 con 0x7f6afc0c3a40 2023-08-25T09:55:20.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.708+0000 7f6b0289b640 1 -- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 msgr2=0x7f6afc0c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:20.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.708+0000 7f6b0289b640 1 --2- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc0c1710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009870 tx=0x7f6af002ba10 comp rx=0 tx=0).stop 2023-08-25T09:55:20.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.709+0000 7f6b0289b640 1 -- 172.21.15.71:0/3778883707 shutdown_connections 2023-08-25T09:55:20.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.709+0000 7f6b0289b640 1 --2- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.709+0000 7f6b0289b640 1 --2- 172.21.15.71:0/3778883707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc0c1710 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.709+0000 7f6b0289b640 1 -- 172.21.15.71:0/3778883707 >> 172.21.15.71:0/3778883707 conn(0x7f6afc0654f0 msgr2=0x7f6afc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:20.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.709+0000 7f6b0289b640 1 -- 172.21.15.71:0/3778883707 shutdown_connections 2023-08-25T09:55:20.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.709+0000 7f6b0289b640 1 -- 172.21.15.71:0/3778883707 wait complete. 2023-08-25T09:55:20.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.710+0000 7f6b0289b640 1 Processor -- start 2023-08-25T09:55:20.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.711+0000 7f6b0289b640 1 -- start start 2023-08-25T09:55:20.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b0289b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 0x7f6afc1e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:20.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b0289b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc1e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:20.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b0289b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc1e2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:20.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b0289b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc15aa40 con 0x7f6afc0c3660 2023-08-25T09:55:20.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b0289b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc15af70 con 0x7f6afc0c3a40 2023-08-25T09:55:20.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b0289b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc15ac90 con 0x7f6afc1509d0 2023-08-25T09:55:20.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 0x7f6afc1e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:20.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 0x7f6afc1e00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35072/0 (socket says 172.21.15.71:35072) 2023-08-25T09:55:20.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 -- 172.21.15.71:0/1976659844 learned_addr learned my addr 172.21.15.71:0/1976659844 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afb7fe640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc1e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6b00e38640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc1e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 -- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 msgr2=0x7f6afc1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc1e2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 -- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 msgr2=0x7f6afc1e1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:20.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc1e1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.712+0000 7f6afbfff640 1 -- 172.21.15.71:0/1976659844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00094d0 con 0x7f6afc0c3660 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.713+0000 7f6afbfff640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 0x7f6afc1e00b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00c430 tx=0x7f6aec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.713+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0052d0 con 0x7f6afc0c3660 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.713+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aec005430 con 0x7f6afc0c3660 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.713+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec003830 con 0x7f6afc0c3660 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.713+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc159480 con 0x7f6afc0c3660 2023-08-25T09:55:20.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.713+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc159a40 con 0x7f6afc0c3660 2023-08-25T09:55:20.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.714+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc0c29c0 con 0x7f6afc0c3660 2023-08-25T09:55:20.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.715+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6aec003990 con 0x7f6afc0c3660 2023-08-25T09:55:20.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.716+0000 7f6af8ff9640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ae806a050 0x7f6ae806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:20.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.716+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f6aec087ff0 con 0x7f6afc0c3660 2023-08-25T09:55:20.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.717+0000 7f6afb7fe640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ae806a050 0x7f6ae806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:20.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.717+0000 7f6afb7fe640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ae806a050 0x7f6ae806c510 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6af0036000 tx=0x7f6af0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:20.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.722+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6aec053230 con 0x7f6afc0c3660 2023-08-25T09:55:20.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.947+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6afc069160 con 0x7f6afc0c3660 2023-08-25T09:55:20.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.948+0000 7f6af8ff9640 1 -- 172.21.15.71:0/1976659844 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f6aec017090 con 0x7f6afc0c3660 2023-08-25T09:55:20.951 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:20.951 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:16.841367+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:20.951 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 36 2023-08-25T09:55:20.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ae806a050 msgr2=0x7f6ae806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:20.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ae806a050 0x7f6ae806c510 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6af0036000 tx=0x7f6af0008030 comp rx=0 tx=0).stop 2023-08-25T09:55:20.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 msgr2=0x7f6afc1e00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:20.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 0x7f6afc1e00b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00c430 tx=0x7f6aec00c8f0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 shutdown_connections 2023-08-25T09:55:20.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6afc0c3a40 0x7f6afc1e1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ae806a050 0x7f6ae806c510 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6afc1509d0 0x7f6afc1e2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 --2- 172.21.15.71:0/1976659844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6afc0c3660 0x7f6afc1e00b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.950+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 >> 172.21.15.71:0/1976659844 conn(0x7f6afc0654f0 msgr2=0x7f6afc0c4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.951+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 shutdown_connections 2023-08-25T09:55:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:20.951+0000 7f6b0289b640 1 -- 172.21.15.71:0/1976659844 wait complete. 2023-08-25T09:55:20.973 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4287": { "addr": "172.21.15.71:6836/1556118065", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 1556118065, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 1556118065, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4287, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 3, "state": "up:active", "state_seq": 14 }, "gid_4290": { "addr": "172.21.15.71:6837/3404832976", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3404832976, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3404832976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:55:16.841367+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 4290, "mds_3": 4287, "mds_4": 4810 } } } ], "standbys": [] } 2023-08-25T09:55:22.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:23.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.197+0000 7f466efb5640 1 Processor -- start 2023-08-25T09:55:23.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.197+0000 7f466efb5640 1 -- start start 2023-08-25T09:55:23.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.199+0000 7f466efb5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4668159fc0 0x7f46680c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.199+0000 7f466efb5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466806a960 con 0x7f46681594d0 2023-08-25T09:55:23.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.199+0000 7f466efb5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466806ac20 con 0x7f466806cc70 2023-08-25T09:55:23.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.199+0000 7f466efb5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466806af00 con 0x7f4668159fc0 2023-08-25T09:55:23.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.199+0000 7f466cd51640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f46681594d0 0x7f466814f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50714/0 (socket says 172.21.15.71:50714) 2023-08-25T09:55:23.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.199+0000 7f466cd51640 1 -- 172.21.15.71:0/1670763190 learned_addr learned my addr 172.21.15.71:0/1670763190 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f466d552640 1 --2- 172.21.15.71:0/1670763190 >> v2:172.21.15.164:3300/0 conn(0x7f4668159fc0 0x7f46680c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471064816 0 0) 0x7f466806a960 con 0x7f46681594d0 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4658003520 con 0x7f46681594d0 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448195268 0 0) 0x7f4658003520 con 0x7f46681594d0 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 >> v2:172.21.15.164:3300/0 conn(0x7f4668159fc0 msgr2=0x7f46680c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 --2- 172.21.15.71:0/1670763190 >> v2:172.21.15.164:3300/0 conn(0x7f4668159fc0 0x7f46680c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 >> v1:172.21.15.164:6789/0 conn(0x7f466806cc70 legacy=0x7f46681598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46680c2e60 con 0x7f46681594d0 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f465c001670 con 0x7f46681594d0 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.200+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 >> v1:172.21.15.71:6789/0 conn(0x7f46681594d0 legacy=0x7f466814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f46677fe640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46580039e0 0x7f4658003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f46677fe640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580047c0 0x7f4658004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f46677fe640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46580051f0 0x7f465800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466806af00 con 0x7f46580051f0 2023-08-25T09:55:23.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465800c140 con 0x7f46580039e0 2023-08-25T09:55:23.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465800c300 con 0x7f46580047c0 2023-08-25T09:55:23.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466d552640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46580051f0 0x7f465800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466cd51640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46580039e0 0x7f4658003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f4667fff640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580047c0 0x7f4658004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466d552640 1 -- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580047c0 msgr2=0x7f4658004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:23.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466d552640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580047c0 0x7f4658004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466d552640 1 -- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46580039e0 msgr2=0x7f4658003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466d552640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46580039e0 0x7f4658003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466d552640 1 -- 172.21.15.71:0/1670763190 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658003520 con 0x7f46580051f0 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.201+0000 7f466cd51640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f46580039e0 0x7f4658003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.202+0000 7f466d552640 1 -- 172.21.15.71:0/1670763190 reap_dead start 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f466d552640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46580051f0 0x7f465800b980 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4668071ce0 tx=0x7f46580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986d0d275d436bd0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:23.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466002cca0 con 0x7f46580051f0 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466002ce00 con 0x7f46580051f0 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f46677fe640 1 -- 172.21.15.71:0/1670763190 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4660034660 con 0x7f46580051f0 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f466efb5640 1 -- 172.21.15.71:0/1670763190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46580051f0 msgr2=0x7f465800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f466efb5640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46580051f0 0x7f465800b980 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4668071ce0 tx=0x7f46580057f0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f466efb5640 1 -- 172.21.15.71:0/1670763190 shutdown_connections 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f466efb5640 1 --2- 172.21.15.71:0/1670763190 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46580051f0 0x7f465800b980 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.203+0000 7f466efb5640 1 -- 172.21.15.71:0/1670763190 >> 172.21.15.71:0/1670763190 conn(0x7f46680654f0 msgr2=0x7f4668065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:23.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.204+0000 7f466efb5640 1 -- 172.21.15.71:0/1670763190 shutdown_connections 2023-08-25T09:55:23.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.204+0000 7f466efb5640 1 -- 172.21.15.71:0/1670763190 wait complete. 2023-08-25T09:55:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.205+0000 7f466efb5640 1 Processor -- start 2023-08-25T09:55:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.205+0000 7f466efb5640 1 -- start start 2023-08-25T09:55:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.206+0000 7f466efb5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580051f0 0x7f466813b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.206+0000 7f466efb5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 0x7f466813c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.206+0000 7f466efb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f466814f0b0 0x7f4668069850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.206+0000 7f466efb5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814ed20 con 0x7f46681577c0 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.206+0000 7f466efb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466814dfe0 con 0x7f466814f0b0 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.206+0000 7f466efb5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466813ba20 con 0x7f46580051f0 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 0x7f466813c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 0x7f466813c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35088/0 (socket says 172.21.15.71:35088) 2023-08-25T09:55:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 -- 172.21.15.71:0/2485651523 learned_addr learned my addr 172.21.15.71:0/2485651523 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f466cd51640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580051f0 0x7f466813b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f466d552640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f466814f0b0 0x7f4668069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 -- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580051f0 msgr2=0x7f466813b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580051f0 0x7f466813b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 -- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f466814f0b0 msgr2=0x7f4668069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f466814f0b0 0x7f4668069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 -- 172.21.15.71:0/2485651523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466814e480 con 0x7f46681577c0 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f466d552640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f466814f0b0 0x7f4668069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f4667fff640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 0x7f466813c1e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f465400e480 tx=0x7f465400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.207+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46540052e0 con 0x7f46681577c0 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.208+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4654005440 con 0x7f46681577c0 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.208+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654003830 con 0x7f46681577c0 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.208+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466814e760 con 0x7f46681577c0 2023-08-25T09:55:23.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.208+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466813bcc0 con 0x7f46681577c0 2023-08-25T09:55:23.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.209+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466813b880 con 0x7f46681577c0 2023-08-25T09:55:23.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.210+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4654003990 con 0x7f46681577c0 2023-08-25T09:55:23.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.211+0000 7f46657fa640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f464c06a080 0x7f464c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.211+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f46540886e0 con 0x7f46681577c0 2023-08-25T09:55:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.211+0000 7f466cd51640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f464c06a080 0x7f464c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:23.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.212+0000 7f466cd51640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f464c06a080 0x7f464c06c540 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f465c001eb0 tx=0x7f465c001ab0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:23.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.215+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f466813b880 con 0x7f46681577c0 2023-08-25T09:55:23.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:55:23.397 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:55:23.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.443+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f466813b880 con 0x7f46681577c0 2023-08-25T09:55:23.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.444+0000 7f46657fa640 1 -- 172.21.15.71:0/2485651523 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f466813b880 con 0x7f46681577c0 2023-08-25T09:55:23.446 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:23.446 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:22.636545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:23.447 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 37 2023-08-25T09:55:23.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f464c06a080 msgr2=0x7f464c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f464c06a080 0x7f464c06c540 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f465c001eb0 tx=0x7f465c001ab0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 msgr2=0x7f466813c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 0x7f466813c1e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f465400e480 tx=0x7f465400e940 comp rx=0 tx=0).stop 2023-08-25T09:55:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 shutdown_connections 2023-08-25T09:55:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f466814f0b0 0x7f4668069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f464c06a080 0x7f464c06c540 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f46681577c0 0x7f466813c1e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 --2- 172.21.15.71:0/2485651523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46580051f0 0x7f466813b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:23.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 >> 172.21.15.71:0/2485651523 conn(0x7f46680654f0 msgr2=0x7f46680c4960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:23.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 shutdown_connections 2023-08-25T09:55:23.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:23.447+0000 7f466efb5640 1 -- 172.21.15.71:0/2485651523 wait complete. 2023-08-25T09:55:23.468 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:55:25.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:25.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.697+0000 7fe93e96b640 1 Processor -- start 2023-08-25T09:55:25.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.697+0000 7fe93e96b640 1 -- start start 2023-08-25T09:55:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.698+0000 7fe93e96b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe9380c3660 0x7fe9380c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.698+0000 7fe93e96b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9380c1dc0 con 0x7fe9381509d0 2023-08-25T09:55:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.698+0000 7fe93e96b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe938156e00 con 0x7fe9380c3a40 2023-08-25T09:55:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.698+0000 7fe93e96b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe938157a80 con 0x7fe9380c3660 2023-08-25T09:55:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe937fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe9380c3660 0x7fe9380c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe937fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe9380c3660 0x7fe9380c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35102/0 (socket says 172.21.15.71:35102) 2023-08-25T09:55:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe937fff640 1 -- 172.21.15.71:0/4085098588 learned_addr learned my addr 172.21.15.71:0/4085098588 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe93cf08640 1 --1- 172.21.15.71:0/4085098588 >> v1:172.21.15.71:6789/0 conn(0x7fe9381509d0 0x7fe938150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50716/0 (socket says 172.21.15.71:50716) 2023-08-25T09:55:25.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe937fff640 1 -- 172.21.15.71:0/4085098588 >> v1:172.21.15.164:6789/0 conn(0x7fe9380c3a40 legacy=0x7fe93806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:25.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe937fff640 1 -- 172.21.15.71:0/4085098588 >> v1:172.21.15.71:6789/0 conn(0x7fe9381509d0 legacy=0x7fe938150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:25.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.699+0000 7fe937fff640 1 -- 172.21.15.71:0/4085098588 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9381570e0 con 0x7fe9380c3660 2023-08-25T09:55:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.700+0000 7fe937fff640 1 --2- 172.21.15.71:0/4085098588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe9380c1710 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe920009870 tx=0x7fe92002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f3aad174d9f1c5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.701+0000 7fe936ffd640 1 -- 172.21.15.71:0/4085098588 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92002f850 con 0x7fe9380c3660 2023-08-25T09:55:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.701+0000 7fe936ffd640 1 -- 172.21.15.71:0/4085098588 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92002f9b0 con 0x7fe9380c3660 2023-08-25T09:55:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.701+0000 7fe936ffd640 1 -- 172.21.15.71:0/4085098588 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92002fd40 con 0x7fe9380c3660 2023-08-25T09:55:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.701+0000 7fe93e96b640 1 -- 172.21.15.71:0/4085098588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 msgr2=0x7fe9380c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.701+0000 7fe93e96b640 1 --2- 172.21.15.71:0/4085098588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe9380c1710 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe920009870 tx=0x7fe92002ba30 comp rx=0 tx=0).stop 2023-08-25T09:55:25.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.701+0000 7fe93e96b640 1 -- 172.21.15.71:0/4085098588 shutdown_connections 2023-08-25T09:55:25.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.702+0000 7fe93e96b640 1 --2- 172.21.15.71:0/4085098588 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe9380c1710 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.702+0000 7fe93e96b640 1 -- 172.21.15.71:0/4085098588 >> 172.21.15.71:0/4085098588 conn(0x7fe9380654f0 msgr2=0x7fe938065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.702+0000 7fe93e96b640 1 -- 172.21.15.71:0/4085098588 shutdown_connections 2023-08-25T09:55:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.702+0000 7fe93e96b640 1 -- 172.21.15.71:0/4085098588 wait complete. 2023-08-25T09:55:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.704+0000 7fe93e96b640 1 Processor -- start 2023-08-25T09:55:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.704+0000 7fe93e96b640 1 -- start start 2023-08-25T09:55:25.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe93e96b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe93815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe93e96b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9380c3a40 0x7fe9381ef960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe93e96b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9381509d0 0x7fe9381efea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe93e96b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe938152090 con 0x7fe9380c3660 2023-08-25T09:55:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe93e96b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93806a8a0 con 0x7fe9381509d0 2023-08-25T09:55:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe93e96b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9381f04c0 con 0x7fe9380c3a40 2023-08-25T09:55:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe937fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe93815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:25.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe937fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe93815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35116/0 (socket says 172.21.15.71:35116) 2023-08-25T09:55:25.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.705+0000 7fe937fff640 1 -- 172.21.15.71:0/1587790827 learned_addr learned my addr 172.21.15.71:0/1587790827 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:25.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe9377fe640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9380c3a40 0x7fe9381ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:25.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe93cf08640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9381509d0 0x7fe9381efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:25.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe937fff640 1 -- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9380c3a40 msgr2=0x7fe9381ef960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:25.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe937fff640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9380c3a40 0x7fe9381ef960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe937fff640 1 -- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9381509d0 msgr2=0x7fe9381efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe937fff640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9381509d0 0x7fe9381efea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe937fff640 1 -- 172.21.15.71:0/1587790827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200094d0 con 0x7fe9380c3660 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe9377fe640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9380c3a40 0x7fe9381ef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe937fff640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe93815ad10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe920009870 tx=0x7fe920004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92002f800 con 0x7fe9380c3660 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920041400 con 0x7fe9380c3660 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200417b0 con 0x7fe9380c3660 2023-08-25T09:55:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.706+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93806ab40 con 0x7fe9380c3660 2023-08-25T09:55:25.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.707+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9381542e0 con 0x7fe9380c3660 2023-08-25T09:55:25.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.708+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93806b060 con 0x7fe9380c3660 2023-08-25T09:55:25.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.709+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe920041910 con 0x7fe9380c3660 2023-08-25T09:55:25.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.709+0000 7fe934ff9640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91806a050 0x7fe91806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:25.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.710+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fe92003c070 con 0x7fe9380c3660 2023-08-25T09:55:25.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.710+0000 7fe9377fe640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91806a050 0x7fe91806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:25.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.710+0000 7fe9377fe640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91806a050 0x7fe91806c510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe9380c6140 tx=0x7fe928009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:25.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.714+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe920076cf0 con 0x7fe9380c3660 2023-08-25T09:55:25.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.942+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe938154470 con 0x7fe9380c3660 2023-08-25T09:55:25.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.943+0000 7fe934ff9640 1 -- 172.21.15.71:0/1587790827 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fe92007a310 con 0x7fe9380c3660 2023-08-25T09:55:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:25.946 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:22.636545+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:25.946 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 37 2023-08-25T09:55:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91806a050 msgr2=0x7fe91806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91806a050 0x7fe91806c510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe9380c6140 tx=0x7fe928009040 comp rx=0 tx=0).stop 2023-08-25T09:55:25.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 msgr2=0x7fe93815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe93815ad10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe920009870 tx=0x7fe920004730 comp rx=0 tx=0).stop 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 shutdown_connections 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9381509d0 0x7fe9381efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91806a050 0x7fe91806c510 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9380c3a40 0x7fe9381ef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 --2- 172.21.15.71:0/1587790827 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9380c3660 0x7fe93815ad10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:25.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 >> 172.21.15.71:0/1587790827 conn(0x7fe9380654f0 msgr2=0x7fe938150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:25.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 shutdown_connections 2023-08-25T09:55:25.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:25.946+0000 7fe93e96b640 1 -- 172.21.15.71:0/1587790827 wait complete. 2023-08-25T09:55:25.968 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:55:27.969 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:28.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.194+0000 7fda7359e640 1 Processor -- start 2023-08-25T09:55:28.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.194+0000 7fda7359e640 1 -- start start 2023-08-25T09:55:28.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda7359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fda74069850 0x7fda741576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:28.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda7359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda740c4fb0 0x7fda740c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:28.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda7359e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda74157bf0 con 0x7fda741542d0 2023-08-25T09:55:28.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda7359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda740c4b60 con 0x7fda74069850 2023-08-25T09:55:28.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda7359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7406a130 con 0x7fda740c4fb0 2023-08-25T09:55:28.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda71d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fda74069850 0x7fda741576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda71d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fda74069850 0x7fda741576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35130/0 (socket says 172.21.15.71:35130) 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda71d9b640 1 -- 172.21.15.71:0/854112680 learned_addr learned my addr 172.21.15.71:0/854112680 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.196+0000 7fda72d9d640 1 --2- 172.21.15.71:0/854112680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda740c4fb0 0x7fda740c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.197+0000 7fda71d9b640 1 -- 172.21.15.71:0/854112680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda740c4fb0 msgr2=0x7fda740c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.197+0000 7fda71d9b640 1 --2- 172.21.15.71:0/854112680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda740c4fb0 0x7fda740c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.197+0000 7fda71d9b640 1 -- 172.21.15.71:0/854112680 >> v1:172.21.15.164:6789/0 conn(0x7fda741542d0 legacy=0x7fda740c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:28.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.197+0000 7fda71d9b640 1 -- 172.21.15.71:0/854112680 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda740c5980 con 0x7fda74069850 2023-08-25T09:55:28.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.198+0000 7fda71d9b640 1 --2- 172.21.15.71:0/854112680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda741576b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fda68009870 tx=0x7fda6802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fc6d4944e54390 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:28.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.198+0000 7fda7159a640 1 -- 172.21.15.71:0/854112680 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68031640 con 0x7fda74069850 2023-08-25T09:55:28.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.198+0000 7fda7159a640 1 -- 172.21.15.71:0/854112680 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda680317a0 con 0x7fda74069850 2023-08-25T09:55:28.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.198+0000 7fda7159a640 1 -- 172.21.15.71:0/854112680 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68031b30 con 0x7fda74069850 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.198+0000 7fda7359e640 1 -- 172.21.15.71:0/854112680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 msgr2=0x7fda741576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.198+0000 7fda7359e640 1 --2- 172.21.15.71:0/854112680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda741576b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fda68009870 tx=0x7fda6802b8f0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.199+0000 7fda7359e640 1 -- 172.21.15.71:0/854112680 shutdown_connections 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.199+0000 7fda7359e640 1 --2- 172.21.15.71:0/854112680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda740c4fb0 0x7fda740c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.199+0000 7fda7359e640 1 --2- 172.21.15.71:0/854112680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda741576b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.199+0000 7fda7359e640 1 -- 172.21.15.71:0/854112680 >> 172.21.15.71:0/854112680 conn(0x7fda7406c280 msgr2=0x7fda7406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.199+0000 7fda7359e640 1 -- 172.21.15.71:0/854112680 shutdown_connections 2023-08-25T09:55:28.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.199+0000 7fda7359e640 1 -- 172.21.15.71:0/854112680 wait complete. 2023-08-25T09:55:28.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.200+0000 7fda7359e640 1 Processor -- start 2023-08-25T09:55:28.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 -- start start 2023-08-25T09:55:28.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda74155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:28.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda740c4fb0 0x7fda740c3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:28.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda741542d0 0x7fda74156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:28.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda74153570 con 0x7fda74069850 2023-08-25T09:55:28.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda74155b70 con 0x7fda740c4fb0 2023-08-25T09:55:28.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.201+0000 7fda7359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda741f2240 con 0x7fda741542d0 2023-08-25T09:55:28.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda74155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:28.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda74155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35142/0 (socket says 172.21.15.71:35142) 2023-08-25T09:55:28.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 -- 172.21.15.71:0/442832807 learned_addr learned my addr 172.21.15.71:0/442832807 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:28.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda72d9d640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda741542d0 0x7fda74156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:28.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda71d9b640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda740c4fb0 0x7fda740c3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:28.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 -- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda741542d0 msgr2=0x7fda74156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:28.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda741542d0 0x7fda74156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 -- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda740c4fb0 msgr2=0x7fda740c3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:28.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda740c4fb0 0x7fda740c3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda7259c640 1 -- 172.21.15.71:0/442832807 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda680094d0 con 0x7fda74069850 2023-08-25T09:55:28.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.202+0000 7fda71d9b640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda740c4fb0 0x7fda740c3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:55:28.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.203+0000 7fda7259c640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda74155630 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fda6400b3f0 tx=0x7fda6400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:28.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.203+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64011830 con 0x7fda74069850 2023-08-25T09:55:28.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.203+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda64011990 con 0x7fda74069850 2023-08-25T09:55:28.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.203+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64011d40 con 0x7fda74069850 2023-08-25T09:55:28.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.203+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda741f2520 con 0x7fda74069850 2023-08-25T09:55:28.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.203+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda74153aa0 con 0x7fda74069850 2023-08-25T09:55:28.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.205+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7404e420 con 0x7fda74069850 2023-08-25T09:55:28.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.205+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fda6401a400 con 0x7fda74069850 2023-08-25T09:55:28.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.206+0000 7fda62ffd640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fda5406a050 0x7fda5406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:28.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.206+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fda640887e0 con 0x7fda74069850 2023-08-25T09:55:28.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.207+0000 7fda71d9b640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fda5406a050 0x7fda5406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:28.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.207+0000 7fda71d9b640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fda5406a050 0x7fda5406c510 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fda68036000 tx=0x7fda6803c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:28.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.211+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fda64053af0 con 0x7fda74069850 2023-08-25T09:55:28.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.434+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda74069f10 con 0x7fda74069850 2023-08-25T09:55:28.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.436+0000 7fda62ffd640 1 -- 172.21.15.71:0/442832807 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fda64057110 con 0x7fda74069850 2023-08-25T09:55:28.438 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:28.439 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:27.858875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:28.439 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 38 2023-08-25T09:55:28.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.438+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fda5406a050 msgr2=0x7fda5406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:28.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.438+0000 7fda7359e640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fda5406a050 0x7fda5406c510 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fda68036000 tx=0x7fda6803c040 comp rx=0 tx=0).stop 2023-08-25T09:55:28.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.438+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 msgr2=0x7fda74155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:28.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.438+0000 7fda7359e640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda74155630 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fda6400b3f0 tx=0x7fda6400b8b0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 shutdown_connections 2023-08-25T09:55:28.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda740c4fb0 0x7fda740c3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fda5406a050 0x7fda5406c510 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda741542d0 0x7fda74156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 --2- 172.21.15.71:0/442832807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda74069850 0x7fda74155630 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:28.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 >> 172.21.15.71:0/442832807 conn(0x7fda7406c280 msgr2=0x7fda7406c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:28.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 shutdown_connections 2023-08-25T09:55:28.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:28.439+0000 7fda7359e640 1 -- 172.21.15.71:0/442832807 wait complete. 2023-08-25T09:55:28.458 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=4, replacing gid=4235 2023-08-25T09:55:28.459 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2023-08-25T09:55:28.785 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T09:55:28.783+0000 7f5fc1b18640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-08-25T09:55:34.460 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-08-25T09:55:34.477 INFO:tasks.ceph.mds.b:Restarting daemon 2023-08-25T09:55:34.478 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-08-25T09:55:34.486 INFO:tasks.ceph.mds.b:Started 2023-08-25T09:55:34.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:55:34.673 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2023-08-25T09:55:34.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.722+0000 7fa1791a4640 1 Processor -- start 2023-08-25T09:55:34.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.722+0000 7fa1791a4640 1 -- start start 2023-08-25T09:55:34.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa1791a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa174159fc0 0x7fa1740c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa1791a4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17406a960 con 0x7fa1741594d0 2023-08-25T09:55:34.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa1791a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17406ac20 con 0x7fa17406cc70 2023-08-25T09:55:34.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa1791a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17406af00 con 0x7fa174159fc0 2023-08-25T09:55:34.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa172d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa1741594d0 0x7fa17414f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54298/0 (socket says 172.21.15.71:54298) 2023-08-25T09:55:34.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa172d9d640 1 -- 172.21.15.71:0/1599978583 learned_addr learned my addr 172.21.15.71:0/1599978583 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:55:34.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa17359e640 1 --2- 172.21.15.71:0/1599978583 >> v2:172.21.15.164:3300/0 conn(0x7fa174159fc0 0x7fa1740c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.724+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081893897 0 0) 0x7fa17406a960 con 0x7fa1741594d0 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c003520 con 0x7fa1741594d0 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4199381955 0 0) 0x7fa15c003520 con 0x7fa1741594d0 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 >> v2:172.21.15.164:3300/0 conn(0x7fa174159fc0 msgr2=0x7fa1740c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 --2- 172.21.15.71:0/1599978583 >> v2:172.21.15.164:3300/0 conn(0x7fa174159fc0 0x7fa1740c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 >> v1:172.21.15.164:6789/0 conn(0x7fa17406cc70 legacy=0x7fa1741598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740c2e60 con 0x7fa1741594d0 2023-08-25T09:55:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fa164001670 con 0x7fa1741594d0 2023-08-25T09:55:34.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 >> v1:172.21.15.71:6789/0 conn(0x7fa1741594d0 legacy=0x7fa17414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa15c0039e0 0x7fa15c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c0047c0 0x7fa15c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa15c005180 0x7fa15c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17406af00 con 0x7fa15c005180 2023-08-25T09:55:34.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c00bf70 con 0x7fa15c0047c0 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c00e340 con 0x7fa15c0039e0 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.725+0000 7fa17359e640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa15c005180 0x7fa15c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa172d9d640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa15c0039e0 0x7fa15c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17259c640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c0047c0 0x7fa15c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17359e640 1 -- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa15c0039e0 msgr2=0x7fa15c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17359e640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa15c0039e0 0x7fa15c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17359e640 1 -- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c0047c0 msgr2=0x7fa15c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:55:34.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17359e640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c0047c0 0x7fa15c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17359e640 1 -- 172.21.15.71:0/1599978583 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c003520 con 0x7fa15c005180 2023-08-25T09:55:34.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.726+0000 7fa17259c640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c0047c0 0x7fa15c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:34.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.727+0000 7fa17359e640 1 -- 172.21.15.71:0/1599978583 reap_dead start 2023-08-25T09:55:34.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.727+0000 7fa17359e640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa15c005180 0x7fa15c00b980 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa174074160 tx=0x7fa15c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deba6633c4319bfc server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:34.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.727+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02cb90 con 0x7fa15c005180 2023-08-25T09:55:34.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.727+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c00f040 con 0x7fa15c005180 2023-08-25T09:55:34.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.727+0000 7fa171d9b640 1 -- 172.21.15.71:0/1599978583 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c034440 con 0x7fa15c005180 2023-08-25T09:55:34.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 -- 172.21.15.71:0/1599978583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa15c005180 msgr2=0x7fa15c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa15c005180 0x7fa15c00b980 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa174074160 tx=0x7fa15c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 -- 172.21.15.71:0/1599978583 shutdown_connections 2023-08-25T09:55:34.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 --2- 172.21.15.71:0/1599978583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa15c005180 0x7fa15c00b980 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 -- 172.21.15.71:0/1599978583 >> 172.21.15.71:0/1599978583 conn(0x7fa1740654f0 msgr2=0x7fa174065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:34.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 -- 172.21.15.71:0/1599978583 shutdown_connections 2023-08-25T09:55:34.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.728+0000 7fa1791a4640 1 -- 172.21.15.71:0/1599978583 wait complete. 2023-08-25T09:55:34.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.730+0000 7fa1791a4640 1 Processor -- start 2023-08-25T09:55:34.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.730+0000 7fa1791a4640 1 -- start start 2023-08-25T09:55:34.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa1791a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c005180 0x7fa17414d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa1791a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 0x7fa174141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa1791a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1741491c0 0x7fa174149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa1791a4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1741ecf20 con 0x7fa1741416d0 2023-08-25T09:55:34.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa1791a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1741edaa0 con 0x7fa15c005180 2023-08-25T09:55:34.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa1791a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17414daf0 con 0x7fa1741491c0 2023-08-25T09:55:34.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa17259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 0x7fa174141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa17259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 0x7fa174141b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49020/0 (socket says 172.21.15.71:49020) 2023-08-25T09:55:34.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.731+0000 7fa17259c640 1 -- 172.21.15.71:0/336756567 learned_addr learned my addr 172.21.15.71:0/336756567 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa172d9d640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c005180 0x7fa17414d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17359e640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1741491c0 0x7fa174149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17259c640 1 -- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1741491c0 msgr2=0x7fa174149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17259c640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1741491c0 0x7fa174149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17259c640 1 -- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c005180 msgr2=0x7fa17414d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17259c640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c005180 0x7fa17414d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17259c640 1 -- 172.21.15.71:0/336756567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17414dd90 con 0x7fa1741416d0 2023-08-25T09:55:34.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17359e640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1741491c0 0x7fa174149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:55:34.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.732+0000 7fa17259c640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 0x7fa174141b30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa16800d7f0 tx=0x7fa16800dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:34.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.734+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168017070 con 0x7fa1741416d0 2023-08-25T09:55:34.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.734+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa168005070 con 0x7fa1741416d0 2023-08-25T09:55:34.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.734+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168005440 con 0x7fa1741416d0 2023-08-25T09:55:34.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.734+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1741ed170 con 0x7fa1741416d0 2023-08-25T09:55:34.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.734+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1741ed730 con 0x7fa1741416d0 2023-08-25T09:55:34.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.735+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1741ed8f0 con 0x7fa1741416d0 2023-08-25T09:55:34.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.736+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa168007240 con 0x7fa1741416d0 2023-08-25T09:55:34.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.737+0000 7fa15b7fe640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa16006a080 0x7fa16006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:55:34.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.737+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fa168088480 con 0x7fa1741416d0 2023-08-25T09:55:34.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.738+0000 7fa172d9d640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa16006a080 0x7fa16006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:55:34.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.738+0000 7fa172d9d640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa16006a080 0x7fa16006c540 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa164001eb0 tx=0x7fa164009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:55:34.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.742+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa168053790 con 0x7fa1741416d0 2023-08-25T09:55:34.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.971+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa174069160 con 0x7fa1741416d0 2023-08-25T09:55:34.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.972+0000 7fa15b7fe640 1 -- 172.21.15.71:0/336756567 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fa1680074f0 con 0x7fa1741416d0 2023-08-25T09:55:34.974 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:55:34.975 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:32.638657+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:55:34.975 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 40 2023-08-25T09:55:34.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa16006a080 msgr2=0x7fa16006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa16006a080 0x7fa16006c540 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa164001eb0 tx=0x7fa164009040 comp rx=0 tx=0).stop 2023-08-25T09:55:34.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 msgr2=0x7fa174141b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:55:34.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 0x7fa174141b30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa16800d7f0 tx=0x7fa16800dcb0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 shutdown_connections 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa16006a080 0x7fa16006c540 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa1741491c0 0x7fa174149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1741416d0 0x7fa174141b30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 --2- 172.21.15.71:0/336756567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa15c005180 0x7fa17414d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 >> 172.21.15.71:0/336756567 conn(0x7fa1740654f0 msgr2=0x7fa174143fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 shutdown_connections 2023-08-25T09:55:34.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:55:34.975+0000 7fa1791a4640 1 -- 172.21.15.71:0/336756567 wait complete. 2023-08-25T09:55:34.996 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-08-25T09:55:34.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-08-25T09:55:40.640 INFO:tasks.workunit.client.0.smithi071.stdout:.//.snap/test1 2023-08-25T09:55:40.640 INFO:tasks.workunit.client.0.smithi071.stdout:.//.snap/test2 2023-08-25T09:55:40.640 INFO:tasks.workunit.client.0.smithi071.stdout:.//.snap/test3 2023-08-25T09:55:40.674 INFO:tasks.workunit.client.0.smithi071.stdout:./274/.snap/snap-subdir-test 2023-08-25T09:55:40.708 INFO:tasks.workunit.client.0.smithi071.stdout:./166/.snap/snap-subdir-test 2023-08-25T09:55:40.737 INFO:tasks.workunit.client.0.smithi071.stdout:./256/.snap/snap-subdir-test 2023-08-25T09:55:40.769 INFO:tasks.workunit.client.0.smithi071.stdout:./247/.snap/snap-subdir-test 2023-08-25T09:55:40.801 INFO:tasks.workunit.client.0.smithi071.stdout:./248/.snap/snap-subdir-test 2023-08-25T09:55:40.832 INFO:tasks.workunit.client.0.smithi071.stdout:./147/.snap/snap-subdir-test 2023-08-25T09:55:40.859 INFO:tasks.workunit.client.0.smithi071.stdout:./151/.snap/snap-subdir-test 2023-08-25T09:55:40.888 INFO:tasks.workunit.client.0.smithi071.stdout:./170/.snap/snap-subdir-test 2023-08-25T09:55:40.926 INFO:tasks.workunit.client.0.smithi071.stdout:./167/.snap/snap-subdir-test 2023-08-25T09:55:40.963 INFO:tasks.workunit.client.0.smithi071.stdout:./140/.snap/snap-subdir-test 2023-08-25T09:55:41.004 INFO:tasks.workunit.client.0.smithi071.stdout:./142/.snap/snap-subdir-test 2023-08-25T09:55:41.030 INFO:tasks.workunit.client.0.smithi071.stdout:./150/.snap/snap-subdir-test 2023-08-25T09:55:41.056 INFO:tasks.workunit.client.0.smithi071.stdout:./273/.snap/snap-subdir-test 2023-08-25T09:55:41.084 INFO:tasks.workunit.client.0.smithi071.stdout:./269/.snap/snap-subdir-test 2023-08-25T09:55:41.118 INFO:tasks.workunit.client.0.smithi071.stdout:./105/.snap/snap-subdir-test 2023-08-25T09:55:41.146 INFO:tasks.workunit.client.0.smithi071.stdout:./210/.snap/snap-subdir-test 2023-08-25T09:55:41.174 INFO:tasks.workunit.client.0.smithi071.stdout:./265/.snap/snap-subdir-test 2023-08-25T09:55:41.196 INFO:tasks.workunit.client.0.smithi071.stdout:./218/.snap/snap-subdir-test 2023-08-25T09:55:41.218 INFO:tasks.workunit.client.0.smithi071.stdout:./290/.snap/snap-subdir-test 2023-08-25T09:55:41.248 INFO:tasks.workunit.client.0.smithi071.stdout:./149/.snap/snap-subdir-test 2023-08-25T09:55:41.282 INFO:tasks.workunit.client.0.smithi071.stdout:./109/.snap/snap-subdir-test 2023-08-25T09:55:41.315 INFO:tasks.workunit.client.0.smithi071.stdout:./292/.snap/snap-subdir-test 2023-08-25T09:55:41.352 INFO:tasks.workunit.client.0.smithi071.stdout:./242/.snap/snap-subdir-test 2023-08-25T09:55:41.375 INFO:tasks.workunit.client.0.smithi071.stdout:./158/.snap/snap-subdir-test 2023-08-25T09:55:41.396 INFO:tasks.workunit.client.0.smithi071.stdout:./243/.snap/snap-subdir-test 2023-08-25T09:55:41.419 INFO:tasks.workunit.client.0.smithi071.stdout:./106/.snap/snap-subdir-test 2023-08-25T09:55:41.446 INFO:tasks.workunit.client.0.smithi071.stdout:./250/.snap/snap-subdir-test 2023-08-25T09:55:41.827 INFO:tasks.workunit.client.0.smithi071.stdout:./221/.snap/snap-subdir-test 2023-08-25T09:55:41.865 INFO:tasks.workunit.client.0.smithi071.stdout:./244/.snap/snap-subdir-test 2023-08-25T09:55:41.900 INFO:tasks.workunit.client.0.smithi071.stdout:./258/.snap/snap-subdir-test 2023-08-25T09:55:41.943 INFO:tasks.workunit.client.0.smithi071.stdout:./268/.snap/snap-subdir-test 2023-08-25T09:55:41.989 INFO:tasks.workunit.client.0.smithi071.stdout:./189/.snap/snap-subdir-test 2023-08-25T09:55:42.015 INFO:tasks.workunit.client.0.smithi071.stdout:./285/.snap/snap-subdir-test 2023-08-25T09:55:42.039 INFO:tasks.workunit.client.0.smithi071.stdout:./208/.snap/snap-subdir-test 2023-08-25T09:55:42.065 INFO:tasks.workunit.client.0.smithi071.stdout:./288/.snap/snap-subdir-test 2023-08-25T09:55:42.089 INFO:tasks.workunit.client.0.smithi071.stdout:./209/.snap/snap-subdir-test 2023-08-25T09:55:42.115 INFO:tasks.workunit.client.0.smithi071.stdout:./253/.snap/snap-subdir-test 2023-08-25T09:55:42.146 INFO:tasks.workunit.client.0.smithi071.stdout:./249/.snap/snap-subdir-test 2023-08-25T09:55:42.177 INFO:tasks.workunit.client.0.smithi071.stdout:./133/.snap/snap-subdir-test 2023-08-25T09:55:42.210 INFO:tasks.workunit.client.0.smithi071.stdout:./291/.snap/snap-subdir-test 2023-08-25T09:55:42.240 INFO:tasks.workunit.client.0.smithi071.stdout:./226/.snap/snap-subdir-test 2023-08-25T09:55:42.268 INFO:tasks.workunit.client.0.smithi071.stdout:./173/.snap/snap-subdir-test 2023-08-25T09:55:42.295 INFO:tasks.workunit.client.0.smithi071.stdout:./174/.snap/snap-subdir-test 2023-08-25T09:55:42.319 INFO:tasks.workunit.client.0.smithi071.stdout:./299/.snap/snap-subdir-test 2023-08-25T09:55:42.342 INFO:tasks.workunit.client.0.smithi071.stdout:./262/.snap/snap-subdir-test 2023-08-25T09:55:42.366 INFO:tasks.workunit.client.0.smithi071.stdout:./231/.snap/snap-subdir-test 2023-08-25T09:55:42.389 INFO:tasks.workunit.client.0.smithi071.stdout:./214/.snap/snap-subdir-test 2023-08-25T09:55:42.413 INFO:tasks.workunit.client.0.smithi071.stdout:./281/.snap/snap-subdir-test 2023-08-25T09:55:42.436 INFO:tasks.workunit.client.0.smithi071.stdout:./289/.snap/snap-subdir-test 2023-08-25T09:55:42.472 INFO:tasks.workunit.client.0.smithi071.stdout:./283/.snap/snap-subdir-test 2023-08-25T09:55:42.510 INFO:tasks.workunit.client.0.smithi071.stdout:./232/.snap/snap-subdir-test 2023-08-25T09:55:42.544 INFO:tasks.workunit.client.0.smithi071.stdout:./298/.snap/snap-subdir-test 2023-08-25T09:55:42.577 INFO:tasks.workunit.client.0.smithi071.stdout:./233/.snap/snap-subdir-test 2023-08-25T09:55:42.609 INFO:tasks.workunit.client.0.smithi071.stdout:./155/.snap/snap-subdir-test 2023-08-25T09:55:42.640 INFO:tasks.workunit.client.0.smithi071.stdout:./114/.snap/snap-subdir-test 2023-08-25T09:55:42.665 INFO:tasks.workunit.client.0.smithi071.stdout:./159/.snap/snap-subdir-test 2023-08-25T09:55:42.698 INFO:tasks.workunit.client.0.smithi071.stdout:./187/.snap/snap-subdir-test 2023-08-25T09:55:42.742 INFO:tasks.workunit.client.0.smithi071.stdout:./185/.snap/snap-subdir-test 2023-08-25T09:55:42.767 INFO:tasks.workunit.client.0.smithi071.stdout:./293/.snap/snap-subdir-test 2023-08-25T09:55:42.797 INFO:tasks.workunit.client.0.smithi071.stdout:./228/.snap/snap-subdir-test 2023-08-25T09:55:42.829 INFO:tasks.workunit.client.0.smithi071.stdout:./120/.snap/snap-subdir-test 2023-08-25T09:55:42.850 INFO:tasks.workunit.client.0.smithi071.stdout:./110/.snap/snap-subdir-test 2023-08-25T09:55:43.411 INFO:tasks.workunit.client.0.smithi071.stdout:./267/.snap/snap-subdir-test 2023-08-25T09:55:43.757 INFO:tasks.workunit.client.0.smithi071.stdout:./202/.snap/snap-subdir-test 2023-08-25T09:55:43.787 INFO:tasks.workunit.client.0.smithi071.stdout:./137/.snap/snap-subdir-test 2023-08-25T09:55:43.811 INFO:tasks.workunit.client.0.smithi071.stdout:./246/.snap/snap-subdir-test 2023-08-25T09:55:43.832 INFO:tasks.workunit.client.0.smithi071.stdout:./219/.snap/snap-subdir-test 2023-08-25T09:55:44.126 INFO:tasks.workunit.client.0.smithi071.stdout:./198/.snap/snap-subdir-test 2023-08-25T09:55:44.579 INFO:tasks.workunit.client.0.smithi071.stdout:./136/.snap/snap-subdir-test 2023-08-25T09:55:44.615 INFO:tasks.workunit.client.0.smithi071.stdout:./270/.snap/snap-subdir-test 2023-08-25T09:55:45.014 INFO:tasks.workunit.client.0.smithi071.stdout:./125/.snap/snap-subdir-test 2023-08-25T09:55:45.049 INFO:tasks.workunit.client.0.smithi071.stdout:./194/.snap/snap-subdir-test 2023-08-25T09:55:45.083 INFO:tasks.workunit.client.0.smithi071.stdout:./119/.snap/snap-subdir-test 2023-08-25T09:55:45.119 INFO:tasks.workunit.client.0.smithi071.stdout:./297/.snap/snap-subdir-test 2023-08-25T09:55:45.148 INFO:tasks.workunit.client.0.smithi071.stdout:./230/.snap/snap-subdir-test 2023-08-25T09:55:45.180 INFO:tasks.workunit.client.0.smithi071.stdout:./260/.snap/snap-subdir-test 2023-08-25T09:55:45.205 INFO:tasks.workunit.client.0.smithi071.stdout:./157/.snap/snap-subdir-test 2023-08-25T09:55:45.240 INFO:tasks.workunit.client.0.smithi071.stdout:./172/.snap/snap-subdir-test 2023-08-25T09:55:45.272 INFO:tasks.workunit.client.0.smithi071.stdout:./277/.snap/snap-subdir-test 2023-08-25T09:55:45.316 INFO:tasks.workunit.client.0.smithi071.stdout:./186/.snap/snap-subdir-test 2023-08-25T09:55:45.360 INFO:tasks.workunit.client.0.smithi071.stdout:./237/.snap/snap-subdir-test 2023-08-25T09:55:45.409 INFO:tasks.workunit.client.0.smithi071.stdout:./190/.snap/snap-subdir-test 2023-08-25T09:55:45.456 INFO:tasks.workunit.client.0.smithi071.stdout:./213/.snap/snap-subdir-test 2023-08-25T09:55:45.499 INFO:tasks.workunit.client.0.smithi071.stdout:./104/.snap/snap-subdir-test 2023-08-25T09:55:45.550 INFO:tasks.workunit.client.0.smithi071.stdout:./164/.snap/snap-subdir-test 2023-08-25T09:55:45.595 INFO:tasks.workunit.client.0.smithi071.stdout:./112/.snap/snap-subdir-test 2023-08-25T09:55:45.638 INFO:tasks.workunit.client.0.smithi071.stdout:./295/.snap/snap-subdir-test 2023-08-25T09:55:45.676 INFO:tasks.workunit.client.0.smithi071.stdout:./272/.snap/snap-subdir-test 2023-08-25T09:55:45.712 INFO:tasks.workunit.client.0.smithi071.stdout:./100/.snap/snap-subdir-test 2023-08-25T09:55:45.743 INFO:tasks.workunit.client.0.smithi071.stdout:./134/.snap/snap-subdir-test 2023-08-25T09:55:45.779 INFO:tasks.workunit.client.0.smithi071.stdout:./200/.snap/snap-subdir-test 2023-08-25T09:55:45.814 INFO:tasks.workunit.client.0.smithi071.stdout:./115/.snap/snap-subdir-test 2023-08-25T09:55:45.852 INFO:tasks.workunit.client.0.smithi071.stdout:./284/.snap/snap-subdir-test 2023-08-25T09:55:45.892 INFO:tasks.workunit.client.0.smithi071.stdout:./280/.snap/snap-subdir-test 2023-08-25T09:55:45.924 INFO:tasks.workunit.client.0.smithi071.stdout:./116/.snap/snap-subdir-test 2023-08-25T09:55:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:./245/.snap/snap-subdir-test 2023-08-25T09:55:46.008 INFO:tasks.workunit.client.0.smithi071.stdout:./195/.snap/snap-subdir-test 2023-08-25T09:55:46.050 INFO:tasks.workunit.client.0.smithi071.stdout:./138/.snap/snap-subdir-test 2023-08-25T09:55:46.091 INFO:tasks.workunit.client.0.smithi071.stdout:./135/.snap/snap-subdir-test 2023-08-25T09:55:46.130 INFO:tasks.workunit.client.0.smithi071.stdout:./117/.snap/snap-subdir-test 2023-08-25T09:55:46.237 INFO:tasks.workunit.client.0.smithi071.stdout:./241/.snap/snap-subdir-test 2023-08-25T09:55:46.280 INFO:tasks.workunit.client.0.smithi071.stdout:./143/.snap/snap-subdir-test 2023-08-25T09:55:46.320 INFO:tasks.workunit.client.0.smithi071.stdout:./180/.snap/snap-subdir-test 2023-08-25T09:55:46.357 INFO:tasks.workunit.client.0.smithi071.stdout:./126/.snap/snap-subdir-test 2023-08-25T09:55:46.391 INFO:tasks.workunit.client.0.smithi071.stdout:./161/.snap/snap-subdir-test 2023-08-25T09:55:46.423 INFO:tasks.workunit.client.0.smithi071.stdout:./204/.snap/snap-subdir-test 2023-08-25T09:55:46.462 INFO:tasks.workunit.client.0.smithi071.stdout:./176/.snap/snap-subdir-test 2023-08-25T09:55:46.500 INFO:tasks.workunit.client.0.smithi071.stdout:./255/.snap/snap-subdir-test 2023-08-25T09:55:46.533 INFO:tasks.workunit.client.0.smithi071.stdout:./215/.snap/snap-subdir-test 2023-08-25T09:55:46.567 INFO:tasks.workunit.client.0.smithi071.stdout:./111/.snap/snap-subdir-test 2023-08-25T09:55:47.352 INFO:tasks.workunit.client.0.smithi071.stdout:./238/.snap/snap-subdir-test 2023-08-25T09:55:47.395 INFO:tasks.workunit.client.0.smithi071.stdout:./191/.snap/snap-subdir-test 2023-08-25T09:55:47.434 INFO:tasks.workunit.client.0.smithi071.stdout:./184/.snap/snap-subdir-test 2023-08-25T09:55:47.472 INFO:tasks.workunit.client.0.smithi071.stdout:./192/.snap/snap-subdir-test 2023-08-25T09:55:47.853 INFO:tasks.workunit.client.0.smithi071.stdout:./103/.snap/snap-subdir-test 2023-08-25T09:55:47.897 INFO:tasks.workunit.client.0.smithi071.stdout:./128/.snap/snap-subdir-test 2023-08-25T09:55:47.935 INFO:tasks.workunit.client.0.smithi071.stdout:./229/.snap/snap-subdir-test 2023-08-25T09:55:47.977 INFO:tasks.workunit.client.0.smithi071.stdout:./144/.snap/snap-subdir-test 2023-08-25T09:55:48.018 INFO:tasks.workunit.client.0.smithi071.stdout:./146/.snap/snap-subdir-test 2023-08-25T09:55:48.055 INFO:tasks.workunit.client.0.smithi071.stdout:./227/.snap/snap-subdir-test 2023-08-25T09:55:48.098 INFO:tasks.workunit.client.0.smithi071.stdout:./278/.snap/snap-subdir-test 2023-08-25T09:55:48.137 INFO:tasks.workunit.client.0.smithi071.stdout:./141/.snap/snap-subdir-test 2023-08-25T09:55:48.173 INFO:tasks.workunit.client.0.smithi071.stdout:./234/.snap/snap-subdir-test 2023-08-25T09:55:48.210 INFO:tasks.workunit.client.0.smithi071.stdout:./225/.snap/snap-subdir-test 2023-08-25T09:55:48.248 INFO:tasks.workunit.client.0.smithi071.stdout:./182/.snap/snap-subdir-test 2023-08-25T09:55:48.274 INFO:tasks.workunit.client.0.smithi071.stdout:./107/.snap/snap-subdir-test 2023-08-25T09:55:48.305 INFO:tasks.workunit.client.0.smithi071.stdout:./178/.snap/snap-subdir-test 2023-08-25T09:55:48.336 INFO:tasks.workunit.client.0.smithi071.stdout:./266/.snap/snap-subdir-test 2023-08-25T09:55:48.370 INFO:tasks.workunit.client.0.smithi071.stdout:./240/.snap/snap-subdir-test 2023-08-25T09:55:48.402 INFO:tasks.workunit.client.0.smithi071.stdout:./156/.snap/snap-subdir-test 2023-08-25T09:55:48.439 INFO:tasks.workunit.client.0.smithi071.stdout:./131/.snap/snap-subdir-test 2023-08-25T09:55:48.480 INFO:tasks.workunit.client.0.smithi071.stdout:./123/.snap/snap-subdir-test 2023-08-25T09:55:48.514 INFO:tasks.workunit.client.0.smithi071.stdout:./160/.snap/snap-subdir-test 2023-08-25T09:55:48.551 INFO:tasks.workunit.client.0.smithi071.stdout:./165/.snap/snap-subdir-test 2023-08-25T09:55:48.583 INFO:tasks.workunit.client.0.smithi071.stdout:./101/.snap/snap-subdir-test 2023-08-25T09:55:48.626 INFO:tasks.workunit.client.0.smithi071.stdout:./148/.snap/snap-subdir-test 2023-08-25T09:55:48.662 INFO:tasks.workunit.client.0.smithi071.stdout:./168/.snap/snap-subdir-test 2023-08-25T09:55:48.686 INFO:tasks.workunit.client.0.smithi071.stdout:./179/.snap/snap-subdir-test 2023-08-25T09:55:48.717 INFO:tasks.workunit.client.0.smithi071.stdout:./257/.snap/snap-subdir-test 2023-08-25T09:55:48.743 INFO:tasks.workunit.client.0.smithi071.stdout:./252/.snap/snap-subdir-test 2023-08-25T09:55:48.771 INFO:tasks.workunit.client.0.smithi071.stdout:./132/.snap/snap-subdir-test 2023-08-25T09:55:48.795 INFO:tasks.workunit.client.0.smithi071.stdout:./236/.snap/snap-subdir-test 2023-08-25T09:55:48.817 INFO:tasks.workunit.client.0.smithi071.stdout:./211/.snap/snap-subdir-test 2023-08-25T09:55:48.844 INFO:tasks.workunit.client.0.smithi071.stdout:./199/.snap/snap-subdir-test 2023-08-25T09:55:48.870 INFO:tasks.workunit.client.0.smithi071.stdout:./205/.snap/snap-subdir-test 2023-08-25T09:55:48.893 INFO:tasks.workunit.client.0.smithi071.stdout:./163/.snap/snap-subdir-test 2023-08-25T09:55:48.911 INFO:tasks.workunit.client.0.smithi071.stdout:./261/.snap/snap-subdir-test 2023-08-25T09:55:48.933 INFO:tasks.workunit.client.0.smithi071.stdout:./201/.snap/snap-subdir-test 2023-08-25T09:55:48.956 INFO:tasks.workunit.client.0.smithi071.stdout:./282/.snap/snap-subdir-test 2023-08-25T09:55:48.977 INFO:tasks.workunit.client.0.smithi071.stdout:./296/.snap/snap-subdir-test 2023-08-25T09:55:49.004 INFO:tasks.workunit.client.0.smithi071.stdout:./217/.snap/snap-subdir-test 2023-08-25T09:55:49.028 INFO:tasks.workunit.client.0.smithi071.stdout:./153/.snap/snap-subdir-test 2023-08-25T09:55:49.883 INFO:tasks.workunit.client.0.smithi071.stdout:./206/.snap/snap-subdir-test 2023-08-25T09:55:49.921 INFO:tasks.workunit.client.0.smithi071.stdout:./122/.snap/snap-subdir-test 2023-08-25T09:55:49.956 INFO:tasks.workunit.client.0.smithi071.stdout:./162/.snap/snap-subdir-test 2023-08-25T09:55:49.993 INFO:tasks.workunit.client.0.smithi071.stdout:./197/.snap/snap-subdir-test 2023-08-25T09:55:50.030 INFO:tasks.workunit.client.0.smithi071.stdout:./216/.snap/snap-subdir-test 2023-08-25T09:55:53.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:55:53.862 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:55:54.235 INFO:tasks.workunit.client.0.smithi071.stdout:./254/.snap/snap-subdir-test 2023-08-25T09:55:54.280 INFO:tasks.workunit.client.0.smithi071.stdout:./113/.snap/snap-subdir-test 2023-08-25T09:55:54.325 INFO:tasks.workunit.client.0.smithi071.stdout:./152/.snap/snap-subdir-test 2023-08-25T09:55:54.358 INFO:tasks.workunit.client.0.smithi071.stdout:./121/.snap/snap-subdir-test 2023-08-25T09:55:54.391 INFO:tasks.workunit.client.0.smithi071.stdout:./118/.snap/snap-subdir-test 2023-08-25T09:55:54.425 INFO:tasks.workunit.client.0.smithi071.stdout:./196/.snap/snap-subdir-test 2023-08-25T09:55:54.460 INFO:tasks.workunit.client.0.smithi071.stdout:./124/.snap/snap-subdir-test 2023-08-25T09:55:54.492 INFO:tasks.workunit.client.0.smithi071.stdout:./239/.snap/snap-subdir-test 2023-08-25T09:55:54.532 INFO:tasks.workunit.client.0.smithi071.stdout:./188/.snap/snap-subdir-test 2023-08-25T09:55:54.556 INFO:tasks.workunit.client.0.smithi071.stdout:./287/.snap/snap-subdir-test 2023-08-25T09:55:54.595 INFO:tasks.workunit.client.0.smithi071.stdout:./275/.snap/snap-subdir-test 2023-08-25T09:55:54.623 INFO:tasks.workunit.client.0.smithi071.stdout:./259/.snap/snap-subdir-test 2023-08-25T09:55:54.654 INFO:tasks.workunit.client.0.smithi071.stdout:./224/.snap/snap-subdir-test 2023-08-25T09:55:54.683 INFO:tasks.workunit.client.0.smithi071.stdout:./264/.snap/snap-subdir-test 2023-08-25T09:55:54.710 INFO:tasks.workunit.client.0.smithi071.stdout:./145/.snap/snap-subdir-test 2023-08-25T09:55:54.727 INFO:tasks.workunit.client.0.smithi071.stdout:./279/.snap/snap-subdir-test 2023-08-25T09:55:54.743 INFO:tasks.workunit.client.0.smithi071.stdout:./263/.snap/snap-subdir-test 2023-08-25T09:55:54.762 INFO:tasks.workunit.client.0.smithi071.stdout:./130/.snap/snap-subdir-test 2023-08-25T09:55:54.785 INFO:tasks.workunit.client.0.smithi071.stdout:./193/.snap/snap-subdir-test 2023-08-25T09:55:54.812 INFO:tasks.workunit.client.0.smithi071.stdout:./223/.snap/snap-subdir-test 2023-08-25T09:55:54.840 INFO:tasks.workunit.client.0.smithi071.stdout:./207/.snap/snap-subdir-test 2023-08-25T09:55:54.866 INFO:tasks.workunit.client.0.smithi071.stdout:./276/.snap/snap-subdir-test 2023-08-25T09:55:54.891 INFO:tasks.workunit.client.0.smithi071.stdout:./271/.snap/snap-subdir-test 2023-08-25T09:55:54.926 INFO:tasks.workunit.client.0.smithi071.stdout:./251/.snap/snap-subdir-test 2023-08-25T09:55:54.958 INFO:tasks.workunit.client.0.smithi071.stdout:./175/.snap/snap-subdir-test 2023-08-25T09:55:55.921 INFO:tasks.workunit.client.0.smithi071.stdout:./222/.snap/snap-subdir-test 2023-08-25T09:55:55.959 INFO:tasks.workunit.client.0.smithi071.stdout:./220/.snap/snap-subdir-test 2023-08-25T09:55:55.996 INFO:tasks.workunit.client.0.smithi071.stdout:./203/.snap/snap-subdir-test 2023-08-25T09:55:56.029 INFO:tasks.workunit.client.0.smithi071.stdout:./181/.snap/snap-subdir-test 2023-08-25T09:55:56.064 INFO:tasks.workunit.client.0.smithi071.stdout:./286/.snap/snap-subdir-test 2023-08-25T09:55:56.107 INFO:tasks.workunit.client.0.smithi071.stdout:./127/.snap/snap-subdir-test 2023-08-25T09:55:56.148 INFO:tasks.workunit.client.0.smithi071.stdout:./108/.snap/snap-subdir-test 2023-08-25T09:55:56.188 INFO:tasks.workunit.client.0.smithi071.stdout:./183/.snap/snap-subdir-test 2023-08-25T09:55:56.220 INFO:tasks.workunit.client.0.smithi071.stdout:./294/.snap/snap-subdir-test 2023-08-25T09:55:57.196 INFO:tasks.workunit.client.0.smithi071.stdout:./235/.snap/snap-subdir-test 2023-08-25T09:55:57.813 INFO:tasks.workunit.client.0.smithi071.stdout:./177/.snap/snap-subdir-test 2023-08-25T09:55:57.849 INFO:tasks.workunit.client.0.smithi071.stdout:./169/.snap/snap-subdir-test 2023-08-25T09:55:58.090 INFO:tasks.workunit.client.0.smithi071.stdout:./212/.snap/snap-subdir-test 2023-08-25T09:55:58.132 INFO:tasks.workunit.client.0.smithi071.stdout:./154/.snap/snap-subdir-test 2023-08-25T09:55:58.167 INFO:tasks.workunit.client.0.smithi071.stdout:./171/.snap/snap-subdir-test 2023-08-25T09:55:58.203 INFO:tasks.workunit.client.0.smithi071.stdout:./139/.snap/snap-subdir-test 2023-08-25T09:55:58.237 INFO:tasks.workunit.client.0.smithi071.stdout:./102/.snap/snap-subdir-test 2023-08-25T09:55:58.277 INFO:tasks.workunit.client.0.smithi071.stdout:./129/.snap/snap-subdir-test 2023-08-25T09:56:06.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:07.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.193+0000 7fd015a14640 1 Processor -- start 2023-08-25T09:56:07.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.193+0000 7fd015a14640 1 -- start start 2023-08-25T09:56:07.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd015a14640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd0100c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd015a14640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd0100c3a40 0x7fd01006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd015a14640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0100c1dc0 con 0x7fd0101509d0 2023-08-25T09:56:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd015a14640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd010157a80 con 0x7fd0100c3a40 2023-08-25T09:56:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd015a14640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01006cf70 con 0x7fd0100c3660 2023-08-25T09:56:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd0067fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd0100c3a40 0x7fd01006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd0067fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd0100c3a40 0x7fd01006ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43326/0 (socket says 172.21.15.71:43326) 2023-08-25T09:56:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd0067fc640 1 -- 172.21.15.71:0/1144942997 learned_addr learned my addr 172.21.15.71:0/1144942997 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.195+0000 7fd00f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd0101509d0 0x7fd010150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49476/0 (socket says 172.21.15.71:49476) 2023-08-25T09:56:07.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd00effd640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd0100c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd0067fc640 1 -- 172.21.15.71:0/1144942997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 msgr2=0x7fd0100c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd0067fc640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd0100c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd0067fc640 1 -- 172.21.15.71:0/1144942997 >> v1:172.21.15.71:6789/0 conn(0x7fd0101509d0 legacy=0x7fd010150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd0067fc640 1 -- 172.21.15.71:0/1144942997 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd010156e00 con 0x7fd0100c3a40 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd00e7fc640 1 -- 172.21.15.71:0/1144942997 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147936369 0 0) 0x7fd0100c1dc0 con 0x7fd0101509d0 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.196+0000 7fd00effd640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd0100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:07.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.197+0000 7fd0067fc640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd01006ca30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009870 tx=0x7fcff802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788355c60adce118 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:07.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.197+0000 7fd00e7fc640 1 -- 172.21.15.71:0/1144942997 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff802f830 con 0x7fd0100c3a40 2023-08-25T09:56:07.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.197+0000 7fd00e7fc640 1 -- 172.21.15.71:0/1144942997 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff802f990 con 0x7fd0100c3a40 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.197+0000 7fd00e7fc640 1 -- 172.21.15.71:0/1144942997 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff802fd20 con 0x7fd0100c3a40 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 -- 172.21.15.71:0/1144942997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 msgr2=0x7fd01006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd01006ca30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009870 tx=0x7fcff802ba10 comp rx=0 tx=0).stop 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 -- 172.21.15.71:0/1144942997 shutdown_connections 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd01006ca30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 --2- 172.21.15.71:0/1144942997 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd0100c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 -- 172.21.15.71:0/1144942997 >> 172.21.15.71:0/1144942997 conn(0x7fd0100654f0 msgr2=0x7fd010065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 -- 172.21.15.71:0/1144942997 shutdown_connections 2023-08-25T09:56:07.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.198+0000 7fd015a14640 1 -- 172.21.15.71:0/1144942997 wait complete. 2023-08-25T09:56:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.200+0000 7fd015a14640 1 Processor -- start 2023-08-25T09:56:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.200+0000 7fd015a14640 1 -- start start 2023-08-25T09:56:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd015a14640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd01006fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd015a14640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd010152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd015a14640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0101509d0 0x7fd010151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd015a14640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01015abd0 con 0x7fd0100c3a40 2023-08-25T09:56:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd015a14640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd010153f90 con 0x7fd0101509d0 2023-08-25T09:56:07.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd015a14640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01015b080 con 0x7fd0100c3660 2023-08-25T09:56:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd010152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd010152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43328/0 (socket says 172.21.15.71:43328) 2023-08-25T09:56:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 -- 172.21.15.71:0/1785575569 learned_addr learned my addr 172.21.15.71:0/1785575569 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 -- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 msgr2=0x7fd01006fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd01006fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 -- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0101509d0 msgr2=0x7fd010151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd00f7fe640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0101509d0 0x7fd010151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0101509d0 0x7fd010151170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.201+0000 7fd0067fc640 1 -- 172.21.15.71:0/1785575569 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80094d0 con 0x7fd0100c3a40 2023-08-25T09:56:07.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.202+0000 7fd0067fc640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd010152670 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcff8000c00 tx=0x7fcff8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.202+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff802f800 con 0x7fd0100c3a40 2023-08-25T09:56:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.202+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff8041400 con 0x7fd0100c3a40 2023-08-25T09:56:07.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.202+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff80417d0 con 0x7fd0100c3a40 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.202+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0100c1930 con 0x7fd0100c3a40 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.202+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01015a8f0 con 0x7fd0100c3a40 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.203+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0100c29c0 con 0x7fd0100c3a40 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.204+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcff8041930 con 0x7fd0100c3a40 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.205+0000 7fd00cff9640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcff006a050 0x7fcff006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.205+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 8147+0+0 (secure 0 0 0) 0x7fcff803c070 con 0x7fd0100c3a40 2023-08-25T09:56:07.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.205+0000 7fd00effd640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcff006a050 0x7fcff006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.206+0000 7fd00effd640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcff006a050 0x7fcff006c510 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd000008e00 tx=0x7fd000008330 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.209+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fcff8076ae0 con 0x7fd0100c3a40 2023-08-25T09:56:07.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.426+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd010069160 con 0x7fd0100c3a40 2023-08-25T09:56:07.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.427+0000 7fd00cff9640 1 -- 172.21.15.71:0/1785575569 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6548 (secure 0 0 0) 0x7fcff807a100 con 0x7fd0100c3a40 2023-08-25T09:56:07.430 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:07.430 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:47.853332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:07.431 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 45 2023-08-25T09:56:07.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcff006a050 msgr2=0x7fcff006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcff006a050 0x7fcff006c510 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd000008e00 tx=0x7fd000008330 comp rx=0 tx=0).stop 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 msgr2=0x7fd010152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd010152670 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcff8000c00 tx=0x7fcff8004730 comp rx=0 tx=0).stop 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 shutdown_connections 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0101509d0 0x7fd010151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcff006a050 0x7fcff006c510 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0100c3a40 0x7fd010152670 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 --2- 172.21.15.71:0/1785575569 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0100c3660 0x7fd01006fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 >> 172.21.15.71:0/1785575569 conn(0x7fd0100654f0 msgr2=0x7fd0100658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:07.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 shutdown_connections 2023-08-25T09:56:07.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.430+0000 7fd015a14640 1 -- 172.21.15.71:0/1785575569 wait complete. 2023-08-25T09:56:07.450 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=3) 2023-08-25T09:56:07.450 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-08-25T09:56:07.451 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T09:56:07.470 INFO:tasks.ceph.mds.a:Stopped 2023-08-25T09:56:07.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:07.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.708+0000 7f68adc85640 1 Processor -- start 2023-08-25T09:56:07.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.708+0000 7f68adc85640 1 -- start start 2023-08-25T09:56:07.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.710+0000 7f68adc85640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 0x7f68a81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.710+0000 7f68adc85640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f68a8069850 0x7f68a80c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.710+0000 7f68adc85640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a8157bf0 con 0x7f68a80c75c0 2023-08-25T09:56:07.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.710+0000 7f68adc85640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68a80c4b60 con 0x7f68a8069850 2023-08-25T09:56:07.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.710+0000 7f68adc85640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a806a130 con 0x7f68a81542d0 2023-08-25T09:56:07.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.711+0000 7f68acc83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 0x7f68a81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.711+0000 7f68acc83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 0x7f68a81576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:32878/0 (socket says 172.21.15.71:32878) 2023-08-25T09:56:07.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.711+0000 7f68acc83640 1 -- 172.21.15.71:0/430470583 learned_addr learned my addr 172.21.15.71:0/430470583 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.711+0000 7f68ad484640 1 --1- 172.21.15.71:0/430470583 >> v1:172.21.15.164:6789/0 conn(0x7f68a80c75c0 0x7f68a80c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:49398/0 (socket says 172.21.15.71:49398) 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.711+0000 7f68a7fff640 1 --2- 172.21.15.71:0/430470583 >> v2:172.21.15.164:3300/0 conn(0x7f68a8069850 0x7f68a80c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.712+0000 7f68acc83640 1 -- 172.21.15.71:0/430470583 >> v2:172.21.15.164:3300/0 conn(0x7f68a8069850 msgr2=0x7f68a80c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.712+0000 7f68acc83640 1 --2- 172.21.15.71:0/430470583 >> v2:172.21.15.164:3300/0 conn(0x7f68a8069850 0x7f68a80c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.712+0000 7f68acc83640 1 -- 172.21.15.71:0/430470583 >> v1:172.21.15.164:6789/0 conn(0x7f68a80c75c0 legacy=0x7f68a80c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.712+0000 7f68acc83640 1 -- 172.21.15.71:0/430470583 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a81546b0 con 0x7f68a81542d0 2023-08-25T09:56:07.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.712+0000 7f68a7fff640 1 --2- 172.21.15.71:0/430470583 >> v2:172.21.15.164:3300/0 conn(0x7f68a8069850 0x7f68a80c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:56:07.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.713+0000 7f68acc83640 1 --2- 172.21.15.71:0/430470583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 0x7f68a81576b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f689c009870 tx=0x7f689c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd6e390e29355e6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:07.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.713+0000 7f68a77fe640 1 -- 172.21.15.71:0/430470583 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c02f830 con 0x7f68a81542d0 2023-08-25T09:56:07.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68a77fe640 1 -- 172.21.15.71:0/430470583 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c02f990 con 0x7f68a81542d0 2023-08-25T09:56:07.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68a77fe640 1 -- 172.21.15.71:0/430470583 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c02fd20 con 0x7f68a81542d0 2023-08-25T09:56:07.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 -- 172.21.15.71:0/430470583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 msgr2=0x7f68a81576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 --2- 172.21.15.71:0/430470583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 0x7f68a81576b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f689c009870 tx=0x7f689c02ba10 comp rx=0 tx=0).stop 2023-08-25T09:56:07.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 -- 172.21.15.71:0/430470583 shutdown_connections 2023-08-25T09:56:07.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 --2- 172.21.15.71:0/430470583 >> v2:172.21.15.164:3300/0 conn(0x7f68a8069850 0x7f68a80c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 --2- 172.21.15.71:0/430470583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a81542d0 0x7f68a81576b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 -- 172.21.15.71:0/430470583 >> 172.21.15.71:0/430470583 conn(0x7f68a806c280 msgr2=0x7f68a806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:07.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 -- 172.21.15.71:0/430470583 shutdown_connections 2023-08-25T09:56:07.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.714+0000 7f68adc85640 1 -- 172.21.15.71:0/430470583 wait complete. 2023-08-25T09:56:07.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.716+0000 7f68adc85640 1 Processor -- start 2023-08-25T09:56:07.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.717+0000 7f68adc85640 1 -- start start 2023-08-25T09:56:07.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68adc85640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68a8069850 0x7f68a806e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68adc85640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a80c75c0 0x7f68a81f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68adc85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 0x7f68a81f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68adc85640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8154de0 con 0x7f68a81542d0 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68adc85640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a81f24c0 con 0x7f68a8069850 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68adc85640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a8157200 con 0x7f68a80c75c0 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 0x7f68a81f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 0x7f68a81f01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:43334/0 (socket says 172.21.15.71:43334) 2023-08-25T09:56:07.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 -- 172.21.15.71:0/3881777434 learned_addr learned my addr 172.21.15.71:0/3881777434 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:07.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68acc83640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68a8069850 0x7f68a806e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68a7fff640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a80c75c0 0x7f68a81f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 -- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a80c75c0 msgr2=0x7f68a81f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a80c75c0 0x7f68a81f1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 -- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68a8069850 msgr2=0x7f68a806e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68a8069850 0x7f68a806e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.718+0000 7f68ad484640 1 -- 172.21.15.71:0/3881777434 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894009400 con 0x7f68a81542d0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.719+0000 7f68ad484640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 0x7f68a81f01a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c7b0 tx=0x7f68a000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.719+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a000a640 con 0x7f68a81542d0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.719+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a000a7a0 con 0x7f68a81542d0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.719+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689c0094d0 con 0x7f68a81542d0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.719+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a000ab10 con 0x7f68a81542d0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.719+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a8156bb0 con 0x7f68a81542d0 2023-08-25T09:56:07.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.720+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a81efea0 con 0x7f68a81542d0 2023-08-25T09:56:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.721+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f68a000ac70 con 0x7f68a81542d0 2023-08-25T09:56:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.722+0000 7f68a57fa640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f688c06a080 0x7f688c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.723+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 8147+0+0 (secure 0 0 0) 0x7f68a0088410 con 0x7f68a81542d0 2023-08-25T09:56:07.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.723+0000 7f68acc83640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f688c06a080 0x7f688c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:07.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.723+0000 7f68acc83640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f688c06a080 0x7f688c06c540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f689c036000 tx=0x7f689c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:07.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.727+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f68a00532e0 con 0x7f68a81542d0 2023-08-25T09:56:07.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.966+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68a8157bf0 con 0x7f68a81542d0 2023-08-25T09:56:07.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.967+0000 7f68a57fa640 1 -- 172.21.15.71:0/3881777434 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6548 (secure 0 0 0) 0x7f68a8157bf0 con 0x7f68a81542d0 2023-08-25T09:56:07.970 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:07.971 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:47.853332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:07.971 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 45 2023-08-25T09:56:07.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.970+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f688c06a080 msgr2=0x7f688c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.970+0000 7f68adc85640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f688c06a080 0x7f688c06c540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f689c036000 tx=0x7f689c008030 comp rx=0 tx=0).stop 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.970+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 msgr2=0x7f68a81f01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.970+0000 7f68adc85640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 0x7f68a81f01a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f68a000c7b0 tx=0x7f68a000cc70 comp rx=0 tx=0).stop 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 shutdown_connections 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f688c06a080 0x7f688c06c540 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68a81542d0 0x7f68a81f01a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f68a80c75c0 0x7f68a81f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 --2- 172.21.15.71:0/3881777434 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68a8069850 0x7f68a806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:07.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 >> 172.21.15.71:0/3881777434 conn(0x7f68a806c280 msgr2=0x7f68a80c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:07.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 shutdown_connections 2023-08-25T09:56:07.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:07.971+0000 7f68adc85640 1 -- 172.21.15.71:0/3881777434 wait complete. 2023-08-25T09:56:07.990 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-08-25T09:56:09.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:10.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.214+0000 7f1722fc9640 1 Processor -- start 2023-08-25T09:56:10.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.214+0000 7f1722fc9640 1 -- start start 2023-08-25T09:56:10.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1722fc9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f171c0c3660 0x7f171c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:10.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1722fc9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f171c1509d0 0x7f171c150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:10.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1722fc9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171c0c1dc0 con 0x7f171c0c3a40 2023-08-25T09:56:10.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1722fc9640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c157a80 con 0x7f171c1509d0 2023-08-25T09:56:10.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1722fc9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c151320 con 0x7f171c0c3660 2023-08-25T09:56:10.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1721566640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f171c1509d0 0x7f171c150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:10.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1721566640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f171c1509d0 0x7f171c150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60028/0 (socket says 172.21.15.71:60028) 2023-08-25T09:56:10.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f1721566640 1 -- 172.21.15.71:0/3679950702 learned_addr learned my addr 172.21.15.71:0/3679950702 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:10.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.216+0000 7f171bfff640 1 --1- 172.21.15.71:0/3679950702 >> v1:172.21.15.71:6789/0 conn(0x7f171c0c3a40 0x7f171c06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48960/0 (socket says 172.21.15.71:48960) 2023-08-25T09:56:10.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.217+0000 7f1720d65640 1 --2- 172.21.15.71:0/3679950702 >> v2:172.21.15.164:3300/0 conn(0x7f171c0c3660 0x7f171c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.217+0000 7f1721566640 1 -- 172.21.15.71:0/3679950702 >> v2:172.21.15.164:3300/0 conn(0x7f171c0c3660 msgr2=0x7f171c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.217+0000 7f1721566640 1 --2- 172.21.15.71:0/3679950702 >> v2:172.21.15.164:3300/0 conn(0x7f171c0c3660 0x7f171c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.217+0000 7f1721566640 1 -- 172.21.15.71:0/3679950702 >> v1:172.21.15.71:6789/0 conn(0x7f171c0c3a40 legacy=0x7f171c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.217+0000 7f1721566640 1 -- 172.21.15.71:0/3679950702 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c156e00 con 0x7f171c1509d0 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.218+0000 7f1721566640 1 --2- 172.21.15.71:0/3679950702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c1509d0 0x7f171c150de0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f171c066440 tx=0x7f171402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7b8c065e61931d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.218+0000 7f171b7fe640 1 -- 172.21.15.71:0/3679950702 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171402cba0 con 0x7f171c1509d0 2023-08-25T09:56:10.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.218+0000 7f171b7fe640 1 -- 172.21.15.71:0/3679950702 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171402cd00 con 0x7f171c1509d0 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.218+0000 7f171b7fe640 1 -- 172.21.15.71:0/3679950702 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714034610 con 0x7f171c1509d0 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 -- 172.21.15.71:0/3679950702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c1509d0 msgr2=0x7f171c150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 --2- 172.21.15.71:0/3679950702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c1509d0 0x7f171c150de0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f171c066440 tx=0x7f171402c000 comp rx=0 tx=0).stop 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 -- 172.21.15.71:0/3679950702 shutdown_connections 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 --2- 172.21.15.71:0/3679950702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c1509d0 0x7f171c150de0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 --2- 172.21.15.71:0/3679950702 >> v2:172.21.15.164:3300/0 conn(0x7f171c0c3660 0x7f171c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 -- 172.21.15.71:0/3679950702 >> 172.21.15.71:0/3679950702 conn(0x7f171c0654f0 msgr2=0x7f171c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 -- 172.21.15.71:0/3679950702 shutdown_connections 2023-08-25T09:56:10.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.219+0000 7f1722fc9640 1 -- 172.21.15.71:0/3679950702 wait complete. 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.221+0000 7f1722fc9640 1 Processor -- start 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.221+0000 7f1722fc9640 1 -- start start 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1722fc9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 0x7f171c158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1722fc9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f171c0c3a40 0x7f171c158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1722fc9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f171c1509d0 0x7f171c145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1722fc9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c06b130 con 0x7f171c0c3660 2023-08-25T09:56:10.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1722fc9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c157e20 con 0x7f171c1509d0 2023-08-25T09:56:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1722fc9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c15a380 con 0x7f171c0c3a40 2023-08-25T09:56:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1720d65640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 0x7f171c158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1720d65640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 0x7f171c158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60030/0 (socket says 172.21.15.71:60030) 2023-08-25T09:56:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.222+0000 7f1720d65640 1 -- 172.21.15.71:0/1196817231 learned_addr learned my addr 172.21.15.71:0/1196817231 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1720d65640 1 -- 172.21.15.71:0/1196817231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f171c0c3a40 msgr2=0x7f171c158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1720d65640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f171c0c3a40 0x7f171c158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1720d65640 1 -- 172.21.15.71:0/1196817231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f171c1509d0 msgr2=0x7f171c145d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:10.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1720d65640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f171c1509d0 0x7f171c145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1720d65640 1 -- 172.21.15.71:0/1196817231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714009b30 con 0x7f171c0c3660 2023-08-25T09:56:10.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1720d65640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 0x7f171c158870 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f171000d5f0 tx=0x7f171000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:10.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17100125f0 con 0x7f171c0c3660 2023-08-25T09:56:10.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1710012750 con 0x7f171c0c3660 2023-08-25T09:56:10.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1710012b00 con 0x7f171c0c3660 2023-08-25T09:56:10.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c15a5e0 con 0x7f171c0c3660 2023-08-25T09:56:10.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.223+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c158350 con 0x7f171c0c3660 2023-08-25T09:56:10.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.224+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c158510 con 0x7f171c0c3660 2023-08-25T09:56:10.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.225+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1710012c60 con 0x7f171c0c3660 2023-08-25T09:56:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.227+0000 7f17197fa640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f170006a050 0x7f170006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.227+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 8115+0+0 (secure 0 0 0) 0x7f17100892d0 con 0x7f171c0c3660 2023-08-25T09:56:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.227+0000 7f171bfff640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f170006a050 0x7f170006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:10.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.228+0000 7f171bfff640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f170006a050 0x7f170006c510 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f171c0c3e50 tx=0x7f17080079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:10.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.231+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1710054110 con 0x7f171c0c3660 2023-08-25T09:56:10.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.462+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f171c0c29c0 con 0x7f171c0c3660 2023-08-25T09:56:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.464+0000 7f17197fa640 1 -- 172.21.15.71:0/1196817231 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+6548 (secure 0 0 0) 0x7f1710057730 con 0x7f171c0c3660 2023-08-25T09:56:10.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:10.467 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:55:47.853332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:10.467 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 45 2023-08-25T09:56:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f170006a050 msgr2=0x7f170006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f170006a050 0x7f170006c510 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f171c0c3e50 tx=0x7f17080079c0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 msgr2=0x7f171c158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:10.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 0x7f171c158870 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f171000d5f0 tx=0x7f171000dab0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 shutdown_connections 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f171c1509d0 0x7f171c145d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f170006a050 0x7f170006c510 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f171c0c3a40 0x7f171c158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 --2- 172.21.15.71:0/1196817231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f171c0c3660 0x7f171c158870 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 >> 172.21.15.71:0/1196817231 conn(0x7f171c0654f0 msgr2=0x7f171c0c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:10.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.466+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 shutdown_connections 2023-08-25T09:56:10.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:10.467+0000 7f1722fc9640 1 -- 172.21.15.71:0/1196817231 wait complete. 2023-08-25T09:56:10.488 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-08-25T09:56:12.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:12.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.715+0000 7f7649406640 1 Processor -- start 2023-08-25T09:56:12.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.716+0000 7f7649406640 1 -- start start 2023-08-25T09:56:12.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.717+0000 7f7649406640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 0x7f76440c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.717+0000 7f7649406640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f76440c3a40 0x7f764406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.717+0000 7f7649406640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76440c1dc0 con 0x7f76441509d0 2023-08-25T09:56:12.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.717+0000 7f7649406640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7644157a80 con 0x7f76440c3a40 2023-08-25T09:56:12.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.717+0000 7f7649406640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764406cf70 con 0x7f76440c3660 2023-08-25T09:56:12.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.718+0000 7f76437fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f76441509d0 0x7f7644150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48974/0 (socket says 172.21.15.71:48974) 2023-08-25T09:56:12.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.718+0000 7f76437fe640 1 -- 172.21.15.71:0/1546690845 learned_addr learned my addr 172.21.15.71:0/1546690845 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:12.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.718+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 0x7f76440c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.718+0000 7f76427fc640 1 --2- 172.21.15.71:0/1546690845 >> v2:172.21.15.164:3300/0 conn(0x7f76440c3a40 0x7f764406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.718+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251269161 0 0) 0x7f76440c1dc0 con 0x7f76441509d0 2023-08-25T09:56:12.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.718+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630003520 con 0x7f76441509d0 2023-08-25T09:56:12.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1660713749 0 0) 0x7f7630003520 con 0x7f76441509d0 2023-08-25T09:56:12.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 msgr2=0x7f76440c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 0x7f76440c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 >> v2:172.21.15.164:3300/0 conn(0x7f76440c3a40 msgr2=0x7f764406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 --2- 172.21.15.71:0/1546690845 >> v2:172.21.15.164:3300/0 conn(0x7f76440c3a40 0x7f764406ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644156e00 con 0x7f76441509d0 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f763c001670 con 0x7f76441509d0 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f76427fc640 1 --2- 172.21.15.71:0/1546690845 >> v2:172.21.15.164:3300/0 conn(0x7f76440c3a40 0x7f764406ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:56:12.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 >> v1:172.21.15.71:6789/0 conn(0x7f76441509d0 legacy=0x7f7644150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300039e0 0x7f7630003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 0x7f7630005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644157a80 con 0x7f76300039e0 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764406cf70 con 0x7f76300047c0 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763000ba90 con 0x7f76440c3660 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300039e0 0x7f7630003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.719+0000 7f76437fe640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 0x7f7630005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f76427fc640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f7642ffd640 1 -- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 msgr2=0x7f7630005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:12.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f76440c3660 0x7f7630005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f7642ffd640 1 -- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300047c0 msgr2=0x7f7630004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:12.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f7642ffd640 1 -- 172.21.15.71:0/1546690845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630003520 con 0x7f76300039e0 2023-08-25T09:56:12.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.720+0000 7f76427fc640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300047c0 0x7f7630004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:12.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.721+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300039e0 0x7f7630003e00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7634009fc0 tx=0x7f76300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6285edf303a4116 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:12.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.721+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763402cc20 con 0x7f76300039e0 2023-08-25T09:56:12.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.721+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763400f040 con 0x7f76300039e0 2023-08-25T09:56:12.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.721+0000 7f7641ffb640 1 -- 172.21.15.71:0/1546690845 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7634034440 con 0x7f76300039e0 2023-08-25T09:56:12.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f7649406640 1 -- 172.21.15.71:0/1546690845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300039e0 msgr2=0x7f7630003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:12.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f7649406640 1 --2- 172.21.15.71:0/1546690845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76300039e0 0x7f7630003e00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7634009fc0 tx=0x7f76300057f0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f76437fe640 1 -- 172.21.15.71:0/1546690845 reap_dead start 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f7649406640 1 -- 172.21.15.71:0/1546690845 shutdown_connections 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f7649406640 1 -- 172.21.15.71:0/1546690845 >> 172.21.15.71:0/1546690845 conn(0x7f76440654f0 msgr2=0x7f7644065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f7649406640 1 -- 172.21.15.71:0/1546690845 shutdown_connections 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.722+0000 7f7649406640 1 -- 172.21.15.71:0/1546690845 wait complete. 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.724+0000 7f7649406640 1 Processor -- start 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.724+0000 7f7649406640 1 -- start start 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f7649406640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300039e0 0x7f7644151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f7649406640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 0x7f764414eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f7649406640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f764414f0a0 0x7f764414cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f7649406640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76441f1bb0 con 0x7f76441f2040 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f7649406640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644152410 con 0x7f76300039e0 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f7649406640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7644151e70 con 0x7f764414f0a0 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f76427fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 0x7f764414eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f76427fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 0x7f764414eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60038/0 (socket says 172.21.15.71:60038) 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.725+0000 7f76427fc640 1 -- 172.21.15.71:0/1634498242 learned_addr learned my addr 172.21.15.71:0/1634498242 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:12.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76437fe640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f764414f0a0 0x7f764414cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300039e0 0x7f7644151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76427fc640 1 -- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f764414f0a0 msgr2=0x7f764414cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76427fc640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f764414f0a0 0x7f764414cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76427fc640 1 -- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300039e0 msgr2=0x7f7644151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76427fc640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300039e0 0x7f7644151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76427fc640 1 -- 172.21.15.71:0/1634498242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7644152130 con 0x7f76441f2040 2023-08-25T09:56:12.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300039e0 0x7f7644151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:56:12.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f76427fc640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 0x7f764414eb60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f762c00f600 tx=0x7f762c00fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.726+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c014890 con 0x7f76441f2040 2023-08-25T09:56:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.727+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762c00c030 con 0x7f76441f2040 2023-08-25T09:56:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.727+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76441f1560 con 0x7f76441f2040 2023-08-25T09:56:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.727+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762c014c30 con 0x7f76441f2040 2023-08-25T09:56:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.727+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76441534b0 con 0x7f76441f2040 2023-08-25T09:56:12.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.728+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76441f13c0 con 0x7f76441f2040 2023-08-25T09:56:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.729+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f762c00d490 con 0x7f76441f2040 2023-08-25T09:56:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.730+0000 7f763b7fe640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f762406a050 0x7f762406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.730+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 8115+0+0 (secure 0 0 0) 0x7f762c0892a0 con 0x7f76441f2040 2023-08-25T09:56:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.730+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f762406a050 0x7f762406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:12.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.731+0000 7f7642ffd640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f762406a050 0x7f762406c510 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7634037040 tx=0x7f763402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:12.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.734+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f76441f13c0 con 0x7f76441f2040 2023-08-25T09:56:12.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.964+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76441f13c0 con 0x7f76441f2040 2023-08-25T09:56:12.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.965+0000 7f763b7fe640 1 -- 172.21.15.71:0/1634498242 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f76441f13c0 con 0x7f76441f2040 2023-08-25T09:56:12.967 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:12.968 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:12.646338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:12.968 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 46 2023-08-25T09:56:12.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.967+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f762406a050 msgr2=0x7f762406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:12.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.967+0000 7f7649406640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f762406a050 0x7f762406c510 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7634037040 tx=0x7f763402e000 comp rx=0 tx=0).stop 2023-08-25T09:56:12.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.967+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 msgr2=0x7f764414eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:12.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.967+0000 7f7649406640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 0x7f764414eb60 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f762c00f600 tx=0x7f762c00fac0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 shutdown_connections 2023-08-25T09:56:12.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f762406a050 0x7f762406c510 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f764414f0a0 0x7f764414cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76441f2040 0x7f764414eb60 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 --2- 172.21.15.71:0/1634498242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f76300039e0 0x7f7644151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:12.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 >> 172.21.15.71:0/1634498242 conn(0x7f76440654f0 msgr2=0x7f76440c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:12.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 shutdown_connections 2023-08-25T09:56:12.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:12.968+0000 7f7649406640 1 -- 172.21.15.71:0/1634498242 wait complete. 2023-08-25T09:56:12.989 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-08-25T09:56:14.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:15.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.211+0000 7fbb20a7c640 1 Processor -- start 2023-08-25T09:56:15.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.212+0000 7fbb20a7c640 1 -- start start 2023-08-25T09:56:15.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.213+0000 7fbb20a7c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbb1c0c3660 0x7fbb1c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:15.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.213+0000 7fbb20a7c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbb1c1509d0 0x7fbb1c150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:15.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.213+0000 7fbb20a7c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1c0c1dc0 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.213+0000 7fbb20a7c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1c157a80 con 0x7fbb1c1509d0 2023-08-25T09:56:15.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb20a7c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1c151320 con 0x7fbb1c0c3660 2023-08-25T09:56:15.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbb1c1509d0 0x7fbb1c150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:15.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbb1c1509d0 0x7fbb1c150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60046/0 (socket says 172.21.15.71:60046) 2023-08-25T09:56:15.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 -- 172.21.15.71:0/2057836460 learned_addr learned my addr 172.21.15.71:0/2057836460 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:15.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1a59c640 1 --2- 172.21.15.71:0/2057836460 >> v2:172.21.15.164:3300/0 conn(0x7fbb1c0c3660 0x7fbb1c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:15.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 -- 172.21.15.71:0/2057836460 >> v2:172.21.15.164:3300/0 conn(0x7fbb1c0c3660 msgr2=0x7fbb1c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:15.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 --2- 172.21.15.71:0/2057836460 >> v2:172.21.15.164:3300/0 conn(0x7fbb1c0c3660 0x7fbb1c0c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 -- 172.21.15.71:0/2057836460 >> v1:172.21.15.164:6789/0 conn(0x7fbb1c0c3a40 legacy=0x7fbb1c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:15.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.214+0000 7fbb1ad9d640 1 -- 172.21.15.71:0/2057836460 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c156e00 con 0x7fbb1c1509d0 2023-08-25T09:56:15.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.215+0000 7fbb1ad9d640 1 --2- 172.21.15.71:0/2057836460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c150de0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c066440 tx=0x7fbb1402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcabbd6eab9801ee server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:15.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb19d9b640 1 -- 172.21.15.71:0/2057836460 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1402cb20 con 0x7fbb1c1509d0 2023-08-25T09:56:15.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb19d9b640 1 -- 172.21.15.71:0/2057836460 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1402cc80 con 0x7fbb1c1509d0 2023-08-25T09:56:15.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb19d9b640 1 -- 172.21.15.71:0/2057836460 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb140343f0 con 0x7fbb1c1509d0 2023-08-25T09:56:15.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb20a7c640 1 -- 172.21.15.71:0/2057836460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c1509d0 msgr2=0x7fbb1c150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:15.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/2057836460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c150de0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c066440 tx=0x7fbb1402bf80 comp rx=0 tx=0).stop 2023-08-25T09:56:15.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb20a7c640 1 -- 172.21.15.71:0/2057836460 shutdown_connections 2023-08-25T09:56:15.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/2057836460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c150de0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/2057836460 >> v2:172.21.15.164:3300/0 conn(0x7fbb1c0c3660 0x7fbb1c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.216+0000 7fbb20a7c640 1 -- 172.21.15.71:0/2057836460 >> 172.21.15.71:0/2057836460 conn(0x7fbb1c0654f0 msgr2=0x7fbb1c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:15.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.217+0000 7fbb20a7c640 1 -- 172.21.15.71:0/2057836460 shutdown_connections 2023-08-25T09:56:15.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.217+0000 7fbb20a7c640 1 -- 172.21.15.71:0/2057836460 wait complete. 2023-08-25T09:56:15.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.218+0000 7fbb20a7c640 1 Processor -- start 2023-08-25T09:56:15.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.218+0000 7fbb20a7c640 1 -- start start 2023-08-25T09:56:15.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.219+0000 7fbb20a7c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb1c0c3660 0x7fbb1c158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.219+0000 7fbb20a7c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 0x7fbb1c158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.219+0000 7fbb20a7c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.219+0000 7fbb20a7c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c157e20 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.219+0000 7fbb20a7c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c1583e0 con 0x7fbb1c1509d0 2023-08-25T09:56:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.219+0000 7fbb20a7c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c1eee30 con 0x7fbb1c0c3660 2023-08-25T09:56:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 0x7fbb1c158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:15.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 0x7fbb1c158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60050/0 (socket says 172.21.15.71:60050) 2023-08-25T09:56:15.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 -- 172.21.15.71:0/3811146526 learned_addr learned my addr 172.21.15.71:0/3811146526 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:15.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb1ad9d640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:15.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb1a59c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb1c0c3660 0x7fbb1c158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:15.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 -- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb1c0c3660 msgr2=0x7fbb1c158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:15.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb1c0c3660 0x7fbb1c158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 -- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb1c1509d0 msgr2=0x7fbb1c159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:15.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 -- 172.21.15.71:0/3811146526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb14009b30 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.220+0000 7fbb13fff640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 0x7fbb1c158db0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400c430 tx=0x7fbb0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:15.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.221+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04011790 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.221+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb040118f0 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.221+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04011ca0 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.221+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c0c0ea0 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.221+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c158070 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.222+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c0c29c0 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.223+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbb0401c460 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.224+0000 7fbb12ffd640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbafc06a050 0x7fbafc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:15.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.224+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 8115+0+0 (secure 0 0 0) 0x7fbb04088d70 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.225+0000 7fbb1a59c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbafc06a050 0x7fbafc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:15.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.225+0000 7fbb1a59c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbafc06a050 0x7fbafc06c510 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c006fd0 tx=0x7fbb0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:15.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.229+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbb04053c50 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.455+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb1c0710e0 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.456+0000 7fbb12ffd640 1 -- 172.21.15.71:0/3811146526 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+6546 (secure 0 0 0) 0x7fbb04057270 con 0x7fbb1c0c3a40 2023-08-25T09:56:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:15.460 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:12.646338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:15.462 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 46 2023-08-25T09:56:15.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbafc06a050 msgr2=0x7fbafc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:15.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbafc06a050 0x7fbafc06c510 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c006fd0 tx=0x7fbb0c008040 comp rx=0 tx=0).stop 2023-08-25T09:56:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 msgr2=0x7fbb1c158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 0x7fbb1c158db0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400c430 tx=0x7fbb0400c8f0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 shutdown_connections 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb1c1509d0 0x7fbb1c159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbafc06a050 0x7fbafc06c510 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbb1c0c3a40 0x7fbb1c158db0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 --2- 172.21.15.71:0/3811146526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb1c0c3660 0x7fbb1c158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 >> 172.21.15.71:0/3811146526 conn(0x7fbb1c0654f0 msgr2=0x7fbb1c069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 shutdown_connections 2023-08-25T09:56:15.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:15.459+0000 7fbb20a7c640 1 -- 172.21.15.71:0/3811146526 wait complete. 2023-08-25T09:56:15.481 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-08-25T09:56:17.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:17.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.709+0000 7f71ebeec640 1 Processor -- start 2023-08-25T09:56:17.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.709+0000 7f71ebeec640 1 -- start start 2023-08-25T09:56:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.710+0000 7f71ebeec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4152d80 0x7f71e406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.710+0000 7f71ebeec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f71e4070cf0 0x7f71e41637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.710+0000 7f71ebeec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e406eb60 con 0x7f71e40c4560 2023-08-25T09:56:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ebeec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e406ca30 con 0x7f71e4070cf0 2023-08-25T09:56:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ebeec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e406cf90 con 0x7f71e4152d80 2023-08-25T09:56:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f71e4070cf0 0x7f71e41637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f71e4070cf0 0x7f71e41637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60056/0 (socket says 172.21.15.71:60056) 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 -- 172.21.15.71:0/4219069478 learned_addr learned my addr 172.21.15.71:0/4219069478 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71e9c88640 1 --2- 172.21.15.71:0/4219069478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4152d80 0x7f71e406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 -- 172.21.15.71:0/4219069478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4152d80 msgr2=0x7f71e406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 --2- 172.21.15.71:0/4219069478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4152d80 0x7f71e406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 -- 172.21.15.71:0/4219069478 >> v1:172.21.15.71:6789/0 conn(0x7f71e40c4560 legacy=0x7f71e40c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.711+0000 7f71ea489640 1 -- 172.21.15.71:0/4219069478 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4158fd0 con 0x7f71e4070cf0 2023-08-25T09:56:17.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.713+0000 7f71ea489640 1 --2- 172.21.15.71:0/4219069478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4070cf0 0x7f71e41637b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009e50 tx=0x7f71dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c443fe0e06abf3db server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.713+0000 7f71e8c86640 1 -- 172.21.15.71:0/4219069478 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc02ca80 con 0x7f71e4070cf0 2023-08-25T09:56:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.713+0000 7f71e8c86640 1 -- 172.21.15.71:0/4219069478 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc02cbe0 con 0x7f71e4070cf0 2023-08-25T09:56:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.713+0000 7f71e8c86640 1 -- 172.21.15.71:0/4219069478 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc0343f0 con 0x7f71e4070cf0 2023-08-25T09:56:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.713+0000 7f71ebeec640 1 -- 172.21.15.71:0/4219069478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4070cf0 msgr2=0x7f71e41637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.713+0000 7f71ebeec640 1 --2- 172.21.15.71:0/4219069478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4070cf0 0x7f71e41637b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009e50 tx=0x7f71dc02bee0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.714+0000 7f71ebeec640 1 -- 172.21.15.71:0/4219069478 shutdown_connections 2023-08-25T09:56:17.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.714+0000 7f71ebeec640 1 --2- 172.21.15.71:0/4219069478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4070cf0 0x7f71e41637b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.714+0000 7f71ebeec640 1 --2- 172.21.15.71:0/4219069478 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4152d80 0x7f71e406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.714+0000 7f71ebeec640 1 -- 172.21.15.71:0/4219069478 >> 172.21.15.71:0/4219069478 conn(0x7f71e40654f0 msgr2=0x7f71e4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:17.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.714+0000 7f71ebeec640 1 -- 172.21.15.71:0/4219069478 shutdown_connections 2023-08-25T09:56:17.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.714+0000 7f71ebeec640 1 -- 172.21.15.71:0/4219069478 wait complete. 2023-08-25T09:56:17.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.715+0000 7f71ebeec640 1 Processor -- start 2023-08-25T09:56:17.721 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 -- start start 2023-08-25T09:56:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4070cf0 0x7f71e415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e40c4560 0x7f71e414ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 0x7f71e414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40bc920 con 0x7f71e4152d80 2023-08-25T09:56:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e415b7c0 con 0x7f71e40c4560 2023-08-25T09:56:17.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.716+0000 7f71ebeec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41eebc0 con 0x7f71e4070cf0 2023-08-25T09:56:17.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 0x7f71e414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:17.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 0x7f71e414b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60060/0 (socket says 172.21.15.71:60060) 2023-08-25T09:56:17.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 -- 172.21.15.71:0/2689839895 learned_addr learned my addr 172.21.15.71:0/2689839895 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:17.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71e9c88640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4070cf0 0x7f71e415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:17.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71e9487640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e40c4560 0x7f71e414ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:17.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 -- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4070cf0 msgr2=0x7f71e415af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:17.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4070cf0 0x7f71e415af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 -- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e40c4560 msgr2=0x7f71e414ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:17.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e40c4560 0x7f71e414ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.717+0000 7f71ea489640 1 -- 172.21.15.71:0/2689839895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc009b30 con 0x7f71e4152d80 2023-08-25T09:56:17.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.718+0000 7f71ea489640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 0x7f71e414b320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009e50 tx=0x7f71dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:17.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.718+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc009510 con 0x7f71e4152d80 2023-08-25T09:56:17.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.718+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc03e450 con 0x7f71e4152d80 2023-08-25T09:56:17.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.718+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e415ba60 con 0x7f71e4152d80 2023-08-25T09:56:17.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.718+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc03e800 con 0x7f71e4152d80 2023-08-25T09:56:17.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.718+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41eeea0 con 0x7f71e4152d80 2023-08-25T09:56:17.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.719+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e415bf80 con 0x7f71e4152d80 2023-08-25T09:56:17.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.720+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f71dc02cd50 con 0x7f71e4152d80 2023-08-25T09:56:17.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.721+0000 7f71da7fc640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f71c806a050 0x7f71c806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:17.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.722+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 8007+0+0 (secure 0 0 0) 0x7f71dc0ab950 con 0x7f71e4152d80 2023-08-25T09:56:17.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.722+0000 7f71e9c88640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f71c806a050 0x7f71c806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:17.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.722+0000 7f71e9c88640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f71c806a050 0x7f71c806c510 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:17.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.727+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f71e415bf80 con 0x7f71e4152d80 2023-08-25T09:56:17.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.956+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71e415bf80 con 0x7f71e4152d80 2023-08-25T09:56:17.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.957+0000 7f71da7fc640 1 -- 172.21.15.71:0/2689839895 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f71e415bf80 con 0x7f71e4152d80 2023-08-25T09:56:17.960 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:17.960 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:12.646338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:17.960 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 46 2023-08-25T09:56:17.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f71c806a050 msgr2=0x7f71c806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:17.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f71c806a050 0x7f71c806c510 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).stop 2023-08-25T09:56:17.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 msgr2=0x7f71e414b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:17.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 0x7f71e414b320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009e50 tx=0x7f71dc004a40 comp rx=0 tx=0).stop 2023-08-25T09:56:17.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 shutdown_connections 2023-08-25T09:56:17.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71e40c4560 0x7f71e414ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f71c806a050 0x7f71c806c510 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f71e4152d80 0x7f71e414b320 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 --2- 172.21.15.71:0/2689839895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71e4070cf0 0x7f71e415af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:17.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 >> 172.21.15.71:0/2689839895 conn(0x7f71e40654f0 msgr2=0x7f71e4150b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:17.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 shutdown_connections 2023-08-25T09:56:17.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:17.960+0000 7f71ebeec640 1 -- 172.21.15.71:0/2689839895 wait complete. 2023-08-25T09:56:17.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-08-25T09:56:19.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:20.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.206+0000 7f4d9ba1f640 1 Processor -- start 2023-08-25T09:56:20.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.206+0000 7f4d9ba1f640 1 -- start start 2023-08-25T09:56:20.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.207+0000 7f4d9ba1f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d940c4560 0x7f4d9406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.207+0000 7f4d9ba1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9406eb60 con 0x7f4d94070cf0 2023-08-25T09:56:20.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.207+0000 7f4d9ba1f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9406ca30 con 0x7f4d94152d80 2023-08-25T09:56:20.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.207+0000 7f4d9ba1f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9406cf90 con 0x7f4d940c4560 2023-08-25T09:56:20.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.208+0000 7f4d99fbc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4d94070cf0 0x7f4d941637b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56344/0 (socket says 172.21.15.71:56344) 2023-08-25T09:56:20.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.208+0000 7f4d99fbc640 1 -- 172.21.15.71:0/4048585946 learned_addr learned my addr 172.21.15.71:0/4048585946 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:20.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.208+0000 7f4d98fba640 1 --2- 172.21.15.71:0/4048585946 >> v2:172.21.15.164:3300/0 conn(0x7f4d940c4560 0x7f4d9406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.208+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534027983 0 0) 0x7f4d9406eb60 con 0x7f4d94070cf0 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.208+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d80003520 con 0x7f4d94070cf0 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15674763 0 0) 0x7f4d80003520 con 0x7f4d94070cf0 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 >> v2:172.21.15.164:3300/0 conn(0x7f4d940c4560 msgr2=0x7f4d9406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 --2- 172.21.15.71:0/4048585946 >> v2:172.21.15.164:3300/0 conn(0x7f4d940c4560 0x7f4d9406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 >> v1:172.21.15.164:6789/0 conn(0x7f4d94152d80 legacy=0x7f4d940c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94158fd0 con 0x7f4d94070cf0 2023-08-25T09:56:20.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4d8c0019f0 con 0x7f4d94070cf0 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 >> v1:172.21.15.71:6789/0 conn(0x7f4d94070cf0 legacy=0x7f4d941637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800039e0 0x7f4d80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d800047c0 0x7f4d80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d800051f0 0x7f4d8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9406cf90 con 0x7f4d800051f0 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8000c150 con 0x7f4d800039e0 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8000c310 con 0x7f4d800047c0 2023-08-25T09:56:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.209+0000 7f4d99fbc640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d800051f0 0x7f4d8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d997bb640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800039e0 0x7f4d80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d98fba640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d800047c0 0x7f4d80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d99fbc640 1 -- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d800047c0 msgr2=0x7f4d80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d99fbc640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d800047c0 0x7f4d80004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d99fbc640 1 -- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800039e0 msgr2=0x7f4d80003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d99fbc640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800039e0 0x7f4d80003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.210+0000 7f4d99fbc640 1 -- 172.21.15.71:0/4048585946 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80003520 con 0x7f4d800051f0 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.211+0000 7f4d99fbc640 1 -- 172.21.15.71:0/4048585946 reap_dead start 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.211+0000 7f4d99fbc640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d800051f0 0x7f4d8000b980 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00a820 tx=0x7f4d800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2cc20e983e3240 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.211+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0072d0 con 0x7f4d800051f0 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.211+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8c02f070 con 0x7f4d800051f0 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.211+0000 7f4d8bfff640 1 -- 172.21.15.71:0/4048585946 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0076b0 con 0x7f4d800051f0 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/4048585946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d800051f0 msgr2=0x7f4d8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d800051f0 0x7f4d8000b980 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00a820 tx=0x7f4d800057f0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/4048585946 shutdown_connections 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/4048585946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d800051f0 0x7f4d8000b980 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/4048585946 >> 172.21.15.71:0/4048585946 conn(0x7f4d940654f0 msgr2=0x7f4d94065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/4048585946 shutdown_connections 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.212+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/4048585946 wait complete. 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.214+0000 7f4d9ba1f640 1 Processor -- start 2023-08-25T09:56:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.214+0000 7f4d9ba1f640 1 -- start start 2023-08-25T09:56:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d9ba1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800051f0 0x7f4d941f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d9ba1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 0x7f4d9406b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d9ba1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d941581c0 0x7f4d94158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d9ba1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d94157d30 con 0x7f4d94157290 2023-08-25T09:56:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d9ba1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d941f1fd0 con 0x7f4d800051f0 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d9ba1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d94158b80 con 0x7f4d941581c0 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d98fba640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 0x7f4d9406b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d98fba640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 0x7f4d9406b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36804/0 (socket says 172.21.15.71:36804) 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d98fba640 1 -- 172.21.15.71:0/3924598441 learned_addr learned my addr 172.21.15.71:0/3924598441 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d997bb640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800051f0 0x7f4d941f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.215+0000 7f4d99fbc640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d941581c0 0x7f4d94158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d98fba640 1 -- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d941581c0 msgr2=0x7f4d94158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d98fba640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d941581c0 0x7f4d94158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d98fba640 1 -- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800051f0 msgr2=0x7f4d941f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d98fba640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800051f0 0x7f4d941f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d98fba640 1 -- 172.21.15.71:0/3924598441 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d941ede50 con 0x7f4d94157290 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d98fba640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 0x7f4d9406b8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4d840108c0 tx=0x7f4d84010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d84009610 con 0x7f4d94157290 2023-08-25T09:56:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8400eb00 con 0x7f4d94157290 2023-08-25T09:56:20.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8400ee90 con 0x7f4d94157290 2023-08-25T09:56:20.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.216+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d941ee110 con 0x7f4d94157290 2023-08-25T09:56:20.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.217+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d941ee6d0 con 0x7f4d94157290 2023-08-25T09:56:20.222 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.218+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d94157ee0 con 0x7f4d94157290 2023-08-25T09:56:20.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.219+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4d8401c400 con 0x7f4d94157290 2023-08-25T09:56:20.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.219+0000 7f4d89ffb640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d7806a080 0x7f4d7806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:20.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.220+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 8007+0+0 (secure 0 0 0) 0x7f4d84089360 con 0x7f4d94157290 2023-08-25T09:56:20.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.220+0000 7f4d997bb640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d7806a080 0x7f4d7806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:20.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.220+0000 7f4d997bb640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d7806a080 0x7f4d7806c540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4d90005490 tx=0x7f4d90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:20.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.224+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4d94157ee0 con 0x7f4d94157290 2023-08-25T09:56:20.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.448+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d94069160 con 0x7f4d94157290 2023-08-25T09:56:20.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.449+0000 7f4d89ffb640 1 -- 172.21.15.71:0/3924598441 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f4d84054180 con 0x7f4d94157290 2023-08-25T09:56:20.451 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:20.451 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:12.646338+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4287,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":3,"incarnation":14,"state":"up:active","state_seq":14,"addr":"172.21.15.71:6836/1556118065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1556118065},{"type":"v1","addr":"172.21.15.71:6836","nonce":1556118065}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:20.451 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 46 2023-08-25T09:56:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d7806a080 msgr2=0x7f4d7806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d7806a080 0x7f4d7806c540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4d90005490 tx=0x7f4d90009040 comp rx=0 tx=0).stop 2023-08-25T09:56:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 msgr2=0x7f4d9406b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:20.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 0x7f4d9406b8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4d840108c0 tx=0x7f4d84010d80 comp rx=0 tx=0).stop 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 shutdown_connections 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d7806a080 0x7f4d7806c540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d941581c0 0x7f4d94158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d94157290 0x7f4d9406b8f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 --2- 172.21.15.71:0/3924598441 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d800051f0 0x7f4d941f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 >> 172.21.15.71:0/3924598441 conn(0x7f4d940654f0 msgr2=0x7f4d940c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:20.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.451+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 shutdown_connections 2023-08-25T09:56:20.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:20.452+0000 7f4d9ba1f640 1 -- 172.21.15.71:0/3924598441 wait complete. 2023-08-25T09:56:20.470 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-08-25T09:56:22.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:22.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.698+0000 7fb673e3f640 1 Processor -- start 2023-08-25T09:56:22.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.698+0000 7fb673e3f640 1 -- start start 2023-08-25T09:56:22.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb673e3f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb66c0c3a40 0x7fb66c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:22.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb673e3f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c0c1dc0 con 0x7fb66c1509d0 2023-08-25T09:56:22.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb673e3f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c156e00 con 0x7fb66c0c3660 2023-08-25T09:56:22.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb673e3f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb66c157a80 con 0x7fb66c0c3a40 2023-08-25T09:56:22.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb6713da640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb66c0c3a40 0x7fb66c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:22.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb6723dc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb66c1509d0 0x7fb66c150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56360/0 (socket says 172.21.15.71:56360) 2023-08-25T09:56:22.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb6713da640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb66c0c3a40 0x7fb66c0c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36810/0 (socket says 172.21.15.71:36810) 2023-08-25T09:56:22.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb6723dc640 1 -- 172.21.15.71:0/1844928869 learned_addr learned my addr 172.21.15.71:0/1844928869 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:22.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.700+0000 7fb6713da640 1 -- 172.21.15.71:0/1844928869 >> v1:172.21.15.164:6789/0 conn(0x7fb66c0c3660 legacy=0x7fb66c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.701+0000 7fb6713da640 1 -- 172.21.15.71:0/1844928869 >> v1:172.21.15.71:6789/0 conn(0x7fb66c1509d0 legacy=0x7fb66c150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.701+0000 7fb6713da640 1 -- 172.21.15.71:0/1844928869 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c1570e0 con 0x7fb66c0c3a40 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.701+0000 7fb670bd9640 1 -- 172.21.15.71:0/1844928869 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476572939 0 0) 0x7fb66c0c1dc0 con 0x7fb66c1509d0 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.702+0000 7fb6713da640 1 --2- 172.21.15.71:0/1844928869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c0c1710 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0097a0 tx=0x7fb65c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c3b16e245ffe87 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.702+0000 7fb670bd9640 1 -- 172.21.15.71:0/1844928869 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c030610 con 0x7fb66c0c3a40 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.702+0000 7fb670bd9640 1 -- 172.21.15.71:0/1844928869 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c030770 con 0x7fb66c0c3a40 2023-08-25T09:56:22.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.702+0000 7fb670bd9640 1 -- 172.21.15.71:0/1844928869 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c030b00 con 0x7fb66c0c3a40 2023-08-25T09:56:22.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 -- 172.21.15.71:0/1844928869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 msgr2=0x7fb66c0c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:22.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 --2- 172.21.15.71:0/1844928869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c0c1710 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0097a0 tx=0x7fb65c02b960 comp rx=0 tx=0).stop 2023-08-25T09:56:22.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 -- 172.21.15.71:0/1844928869 shutdown_connections 2023-08-25T09:56:22.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 --2- 172.21.15.71:0/1844928869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c0c1710 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 -- 172.21.15.71:0/1844928869 >> 172.21.15.71:0/1844928869 conn(0x7fb66c0654f0 msgr2=0x7fb66c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:22.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 -- 172.21.15.71:0/1844928869 shutdown_connections 2023-08-25T09:56:22.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.703+0000 7fb673e3f640 1 -- 172.21.15.71:0/1844928869 wait complete. 2023-08-25T09:56:22.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.704+0000 7fb673e3f640 1 Processor -- start 2023-08-25T09:56:22.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.705+0000 7fb673e3f640 1 -- start start 2023-08-25T09:56:22.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb673e3f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0c3660 0x7fb66c158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:22.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb673e3f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:22.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb673e3f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c1509d0 0x7fb66c06f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:22.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb673e3f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c15a830 con 0x7fb66c0c3a40 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb673e3f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c159700 con 0x7fb66c1509d0 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb673e3f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0c5c50 con 0x7fb66c0c3660 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c06ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36818/0 (socket says 172.21.15.71:36818) 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 -- 172.21.15.71:0/377967240 learned_addr learned my addr 172.21.15.71:0/377967240 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 -- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0c3660 msgr2=0x7fb66c158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:22.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb671bdb640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0c3660 0x7fb66c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:22.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6723dc640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c1509d0 0x7fb66c06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:22.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0c3660 0x7fb66c158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 -- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c1509d0 msgr2=0x7fb66c06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:22.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c1509d0 0x7fb66c06f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb6713da640 1 -- 172.21.15.71:0/377967240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c009400 con 0x7fb66c0c3a40 2023-08-25T09:56:22.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.706+0000 7fb671bdb640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0c3660 0x7fb66c158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:22.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.707+0000 7fb6713da640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c06ed10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0098c0 tx=0x7fb65c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:22.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.707+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c03e040 con 0x7fb66c0c3a40 2023-08-25T09:56:22.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.707+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c039c90 con 0x7fb66c0c3a40 2023-08-25T09:56:22.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.707+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c15aa90 con 0x7fb66c0c3a40 2023-08-25T09:56:22.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.707+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0443f0 con 0x7fb66c0c3a40 2023-08-25T09:56:22.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.707+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c1ee8c0 con 0x7fb66c0c3a40 2023-08-25T09:56:22.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.709+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c15b0f0 con 0x7fb66c0c3a40 2023-08-25T09:56:22.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.709+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb65c04d3b0 con 0x7fb66c0c3a40 2023-08-25T09:56:22.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.711+0000 7fb6627fc640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a080 0x7fb65006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:22.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.711+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fb65c0ab120 con 0x7fb66c0c3a40 2023-08-25T09:56:22.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.711+0000 7fb671bdb640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a080 0x7fb65006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:22.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.711+0000 7fb671bdb640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a080 0x7fb65006c540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb66800a830 tx=0x7fb668008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:22.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.716+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb65c076150 con 0x7fb66c0c3a40 2023-08-25T09:56:22.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.944+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb66c0c4e80 con 0x7fb66c0c3a40 2023-08-25T09:56:22.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.946+0000 7fb6627fc640 1 -- 172.21.15.71:0/377967240 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5748 (secure 0 0 0) 0x7fb65c049090 con 0x7fb66c0c3a40 2023-08-25T09:56:22.947 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:22.948 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:20.645939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:22.948 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 47 2023-08-25T09:56:22.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a080 msgr2=0x7fb65006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:22.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a080 0x7fb65006c540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb66800a830 tx=0x7fb668008040 comp rx=0 tx=0).stop 2023-08-25T09:56:22.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 msgr2=0x7fb66c06ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:22.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c06ed10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0098c0 tx=0x7fb65c004730 comp rx=0 tx=0).stop 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 shutdown_connections 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c1509d0 0x7fb66c06f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a080 0x7fb65006c540 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0c3a40 0x7fb66c06ed10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 --2- 172.21.15.71:0/377967240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0c3660 0x7fb66c158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.948+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 >> 172.21.15.71:0/377967240 conn(0x7fb66c0654f0 msgr2=0x7fb66c0c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:22.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.949+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 shutdown_connections 2023-08-25T09:56:22.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:22.949+0000 7fb673e3f640 1 -- 172.21.15.71:0/377967240 wait complete. 2023-08-25T09:56:22.967 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2023-08-25T09:56:22.967 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-08-25T09:56:22.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:23.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.180+0000 7fed3d058640 1 Processor -- start 2023-08-25T09:56:23.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.180+0000 7fed3d058640 1 -- start start 2023-08-25T09:56:23.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed3d058640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed38152c50 0x7fed380c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed3d058640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed38153030 0x7fed380c1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed3d058640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed3806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed3d058640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed38159ae0 con 0x7fed38152c50 2023-08-25T09:56:23.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed3d058640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed38071dd0 con 0x7fed38153030 2023-08-25T09:56:23.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed3d058640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed38158f60 con 0x7fed380c0260 2023-08-25T09:56:23.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed37fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed38152c50 0x7fed380c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:23.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed37fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fed38152c50 0x7fed380c0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36826/0 (socket says 172.21.15.71:36826) 2023-08-25T09:56:23.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.182+0000 7fed37fff640 1 -- 172.21.15.71:0/3268657078 learned_addr learned my addr 172.21.15.71:0/3268657078 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:23.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed3c857640 1 --2- 172.21.15.71:0/3268657078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed3806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:23.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed377fe640 1 --2- 172.21.15.71:0/3268657078 >> v2:172.21.15.164:3300/0 conn(0x7fed38153030 0x7fed380c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:23.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed37fff640 1 -- 172.21.15.71:0/3268657078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 msgr2=0x7fed3806fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:23.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed37fff640 1 --2- 172.21.15.71:0/3268657078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed3806fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed37fff640 1 -- 172.21.15.71:0/3268657078 >> v2:172.21.15.164:3300/0 conn(0x7fed38153030 msgr2=0x7fed380c1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:23.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed37fff640 1 --2- 172.21.15.71:0/3268657078 >> v2:172.21.15.164:3300/0 conn(0x7fed38153030 0x7fed380c1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.183+0000 7fed37fff640 1 -- 172.21.15.71:0/3268657078 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed38159240 con 0x7fed38152c50 2023-08-25T09:56:23.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.184+0000 7fed37fff640 1 --2- 172.21.15.71:0/3268657078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed380c0f00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fed380c5900 tx=0x7fed2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f552286bf4888d2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:23.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed36ffd640 1 -- 172.21.15.71:0/3268657078 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed28031870 con 0x7fed38152c50 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed36ffd640 1 -- 172.21.15.71:0/3268657078 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed280319d0 con 0x7fed38152c50 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed36ffd640 1 -- 172.21.15.71:0/3268657078 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed28031d60 con 0x7fed38152c50 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 -- 172.21.15.71:0/3268657078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 msgr2=0x7fed380c0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 --2- 172.21.15.71:0/3268657078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed380c0f00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fed380c5900 tx=0x7fed2802b910 comp rx=0 tx=0).stop 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 -- 172.21.15.71:0/3268657078 shutdown_connections 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 --2- 172.21.15.71:0/3268657078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed3806fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 --2- 172.21.15.71:0/3268657078 >> v2:172.21.15.164:3300/0 conn(0x7fed38153030 0x7fed380c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 --2- 172.21.15.71:0/3268657078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed380c0f00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.185+0000 7fed3d058640 1 -- 172.21.15.71:0/3268657078 >> 172.21.15.71:0/3268657078 conn(0x7fed380cb260 msgr2=0x7fed380cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:23.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.186+0000 7fed3d058640 1 -- 172.21.15.71:0/3268657078 shutdown_connections 2023-08-25T09:56:23.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.186+0000 7fed3d058640 1 -- 172.21.15.71:0/3268657078 wait complete. 2023-08-25T09:56:23.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.187+0000 7fed3d058640 1 Processor -- start 2023-08-25T09:56:23.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.188+0000 7fed3d058640 1 -- start start 2023-08-25T09:56:23.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.188+0000 7fed3d058640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed381f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.188+0000 7fed3d058640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed3814bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.188+0000 7fed3d058640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed38153030 0x7fed3814c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed3d058640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed38155b20 con 0x7fed38152c50 2023-08-25T09:56:23.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed3d058640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3815a4a0 con 0x7fed38153030 2023-08-25T09:56:23.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed3d058640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed381f22d0 con 0x7fed380c0260 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed3814bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed3814bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36840/0 (socket says 172.21.15.71:36840) 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 -- 172.21.15.71:0/1641585182 learned_addr learned my addr 172.21.15.71:0/1641585182 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 -- 172.21.15.71:0/1641585182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 msgr2=0x7fed381f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed381f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 -- 172.21.15.71:0/1641585182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed38153030 msgr2=0x7fed3814c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:23.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed38153030 0x7fed3814c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 -- 172.21.15.71:0/1641585182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280094d0 con 0x7fed38152c50 2023-08-25T09:56:23.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.189+0000 7fed377fe640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed3814bfc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00d620 tx=0x7fed2c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:23.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.190+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c0125f0 con 0x7fed38152c50 2023-08-25T09:56:23.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.190+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c012750 con 0x7fed38152c50 2023-08-25T09:56:23.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.190+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed380c69a0 con 0x7fed38152c50 2023-08-25T09:56:23.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.190+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c012b00 con 0x7fed38152c50 2023-08-25T09:56:23.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.190+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed381e4230 con 0x7fed38152c50 2023-08-25T09:56:23.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.191+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3804e420 con 0x7fed38152c50 2023-08-25T09:56:23.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.192+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fed2c012c60 con 0x7fed38152c50 2023-08-25T09:56:23.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.193+0000 7fed34ff9640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2406a050 0x7fed2406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:23.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.193+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fed2c088f30 con 0x7fed38152c50 2023-08-25T09:56:23.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.193+0000 7fed37fff640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2406a050 0x7fed2406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:23.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.194+0000 7fed37fff640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2406a050 0x7fed2406c510 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fed28036000 tx=0x7fed28030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:23.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.197+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fed2c054090 con 0x7fed38152c50 2023-08-25T09:56:23.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.427+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed3814fc40 con 0x7fed38152c50 2023-08-25T09:56:23.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.428+0000 7fed34ff9640 1 -- 172.21.15.71:0/1641585182 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5748 (secure 0 0 0) 0x7fed2c0576b0 con 0x7fed38152c50 2023-08-25T09:56:23.430 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:23.430 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:20.645939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:23.431 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 47 2023-08-25T09:56:23.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2406a050 msgr2=0x7fed2406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2406a050 0x7fed2406c510 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fed28036000 tx=0x7fed28030000 comp rx=0 tx=0).stop 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 msgr2=0x7fed3814bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed3814bfc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00d620 tx=0x7fed2c00dae0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 shutdown_connections 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed38153030 0x7fed3814c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed2406a050 0x7fed2406c510 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed38152c50 0x7fed3814bfc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 --2- 172.21.15.71:0/1641585182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed380c0260 0x7fed381f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:23.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 >> 172.21.15.71:0/1641585182 conn(0x7fed380cb260 msgr2=0x7fed381577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:23.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 shutdown_connections 2023-08-25T09:56:23.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:23.431+0000 7fed3d058640 1 -- 172.21.15.71:0/1641585182 wait complete. 2023-08-25T09:56:23.450 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 47, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 47, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4290": { "addr": "172.21.15.71:6837/3404832976", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3404832976, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3404832976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:56:20.645939+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 4290, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [] } 2023-08-25T09:56:24.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:56:24.014 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:56:25.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:25.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.693+0000 7f8001396640 1 Processor -- start 2023-08-25T09:56:25.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.694+0000 7f8001396640 1 -- start start 2023-08-25T09:56:25.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.696+0000 7f8001396640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7ffc066010 0x7f7ffc0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:25.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.696+0000 7f8001396640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ffc070cf0 0x7f7ffc0c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:25.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.696+0000 7f8001396640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc15db40 con 0x7f7ffc157450 2023-08-25T09:56:25.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.696+0000 7f8001396640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc15d230 con 0x7f7ffc066010 2023-08-25T09:56:25.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.696+0000 7f8001396640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc06cb40 con 0x7f7ffc070cf0 2023-08-25T09:56:25.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ffc070cf0 0x7f7ffc0c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffa7fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7ffc066010 0x7f7ffc0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ffc070cf0 0x7f7ffc0c5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:53164/0 (socket says 172.21.15.71:53164) 2023-08-25T09:56:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffb7fe640 1 -- 172.21.15.71:0/4207450411 learned_addr learned my addr 172.21.15.71:0/4207450411 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffa7fc640 1 -- 172.21.15.71:0/4207450411 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc070cf0 msgr2=0x7f7ffc0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:25.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffa7fc640 1 --2- 172.21.15.71:0/4207450411 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc070cf0 0x7f7ffc0c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffa7fc640 1 -- 172.21.15.71:0/4207450411 >> v1:172.21.15.71:6789/0 conn(0x7f7ffc157450 legacy=0x7f7ffc06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.697+0000 7f7ffa7fc640 1 -- 172.21.15.71:0/4207450411 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc06ce20 con 0x7f7ffc066010 2023-08-25T09:56:25.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.699+0000 7f7ffa7fc640 1 --2- 172.21.15.71:0/4207450411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc066010 0x7f7ffc0c56a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc06dbe0 tx=0x7f7ff002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c1dca329f1e881 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:25.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.700+0000 7f7ff9ffb640 1 -- 172.21.15.71:0/4207450411 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0031830 con 0x7f7ffc066010 2023-08-25T09:56:25.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.700+0000 7f7ff9ffb640 1 -- 172.21.15.71:0/4207450411 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff0031990 con 0x7f7ffc066010 2023-08-25T09:56:25.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.700+0000 7f7ff9ffb640 1 -- 172.21.15.71:0/4207450411 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0031d20 con 0x7f7ffc066010 2023-08-25T09:56:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.701+0000 7f8001396640 1 -- 172.21.15.71:0/4207450411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc066010 msgr2=0x7f7ffc0c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:25.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.701+0000 7f8001396640 1 --2- 172.21.15.71:0/4207450411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc066010 0x7f7ffc0c56a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc06dbe0 tx=0x7f7ff002b8d0 comp rx=0 tx=0).stop 2023-08-25T09:56:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.703+0000 7f8001396640 1 -- 172.21.15.71:0/4207450411 shutdown_connections 2023-08-25T09:56:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.703+0000 7f8001396640 1 --2- 172.21.15.71:0/4207450411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc066010 0x7f7ffc0c56a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:25.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.703+0000 7f8001396640 1 --2- 172.21.15.71:0/4207450411 >> v2:172.21.15.164:3300/0 conn(0x7f7ffc070cf0 0x7f7ffc0c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:25.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.703+0000 7f8001396640 1 -- 172.21.15.71:0/4207450411 >> 172.21.15.71:0/4207450411 conn(0x7f7ffc0654f0 msgr2=0x7f7ffc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:25.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.704+0000 7f8001396640 1 -- 172.21.15.71:0/4207450411 shutdown_connections 2023-08-25T09:56:25.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.704+0000 7f8001396640 1 -- 172.21.15.71:0/4207450411 wait complete. 2023-08-25T09:56:25.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.708+0000 7f8001396640 1 Processor -- start 2023-08-25T09:56:25.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.709+0000 7f8001396640 1 -- start start 2023-08-25T09:56:25.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.711+0000 7f8001396640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc066010 0x7f7ffc151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:25.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.711+0000 7f8001396640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc070cf0 0x7f7ffc14f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:25.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.711+0000 7f8001396640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 0x7f7ffc1f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:25.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.711+0000 7f8001396640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc14f280 con 0x7f7ffc157450 2023-08-25T09:56:25.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.711+0000 7f8001396640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc0c80f0 con 0x7f7ffc070cf0 2023-08-25T09:56:25.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.711+0000 7f8001396640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc0c3590 con 0x7f7ffc066010 2023-08-25T09:56:25.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.712+0000 7f7ffb7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 0x7f7ffc1f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:25.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.712+0000 7f7ffb7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 0x7f7ffc1f5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36844/0 (socket says 172.21.15.71:36844) 2023-08-25T09:56:25.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.712+0000 7f7ffb7fe640 1 -- 172.21.15.71:0/2960171778 learned_addr learned my addr 172.21.15.71:0/2960171778 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:25.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.712+0000 7f7ffa7fc640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc070cf0 0x7f7ffc14f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:25.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.713+0000 7f7ffb7fe640 1 -- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc066010 msgr2=0x7f7ffc151c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T09:56:25.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.713+0000 7f7ffaffd640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc066010 0x7f7ffc151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:25.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.713+0000 7f7ffb7fe640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc066010 0x7f7ffc151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:25.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.713+0000 7f7ffb7fe640 1 -- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc070cf0 msgr2=0x7f7ffc14f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:25.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.713+0000 7f7ffb7fe640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc070cf0 0x7f7ffc14f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:25.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.713+0000 7f7ffb7fe640 1 -- 172.21.15.71:0/2960171778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0009400 con 0x7f7ffc157450 2023-08-25T09:56:25.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.714+0000 7f7ffaffd640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc066010 0x7f7ffc151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:25.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.714+0000 7f7ffb7fe640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 0x7f7ffc1f5830 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00e8a0 tx=0x7f7fec00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:25.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.717+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec00c940 con 0x7f7ffc157450 2023-08-25T09:56:25.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.718+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec00caa0 con 0x7f7ffc157450 2023-08-25T09:56:25.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.718+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc0c2a10 con 0x7f7ffc157450 2023-08-25T09:56:25.722 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.718+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc0c2fd0 con 0x7f7ffc157450 2023-08-25T09:56:25.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.720+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec00ce50 con 0x7f7ffc157450 2023-08-25T09:56:25.723 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.721+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc005250 con 0x7f7ffc157450 2023-08-25T09:56:25.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.724+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7fec013470 con 0x7f7ffc157450 2023-08-25T09:56:25.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.731+0000 7f7fe37fe640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fe806a120 0x7f7fe806c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:25.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.731+0000 7f7ffaffd640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fe806a120 0x7f7fe806c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:25.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.731+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8101+0+0 (secure 0 0 0) 0x7f7fec088f60 con 0x7f7ffc157450 2023-08-25T09:56:25.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.732+0000 7f7ffaffd640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fe806a120 0x7f7fe806c5e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4004520 tx=0x7f7ff4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:25.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:25.735+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7fec057970 con 0x7f7ffc157450 2023-08-25T09:56:26.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.054+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fdc005a80 con 0x7f7ffc157450 2023-08-25T09:56:26.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.056+0000 7f7fe37fe640 1 -- 172.21.15.71:0/2960171778 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f7fec017090 con 0x7f7ffc157450 2023-08-25T09:56:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:26.058 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:20.645939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:26.058 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 47 2023-08-25T09:56:26.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fe806a120 msgr2=0x7f7fe806c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:26.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fe806a120 0x7f7fe806c5e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4004520 tx=0x7f7ff4009040 comp rx=0 tx=0).stop 2023-08-25T09:56:26.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 msgr2=0x7f7ffc1f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:26.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 0x7f7ffc1f5830 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00e8a0 tx=0x7f7fec00ed60 comp rx=0 tx=0).stop 2023-08-25T09:56:26.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 shutdown_connections 2023-08-25T09:56:26.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ffc070cf0 0x7f7ffc14f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:26.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fe806a120 0x7f7fe806c5e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:26.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ffc157450 0x7f7ffc1f5830 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:26.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 --2- 172.21.15.71:0/2960171778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ffc066010 0x7f7ffc151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:26.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 >> 172.21.15.71:0/2960171778 conn(0x7f7ffc0654f0 msgr2=0x7f7ffc0c5170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:26.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 shutdown_connections 2023-08-25T09:56:26.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:26.058+0000 7f8001396640 1 -- 172.21.15.71:0/2960171778 wait complete. 2023-08-25T09:56:26.078 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:28.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:28.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.301+0000 7ff04c0d3640 1 Processor -- start 2023-08-25T09:56:28.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.301+0000 7ff04c0d3640 1 -- start start 2023-08-25T09:56:28.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04c0d3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff044152d80 0x7ff04406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04c0d3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff0440c4560 0x7ff0440c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04c0d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044070cf0 0x7ff0440c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04c0d3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04406eb60 con 0x7ff0440c4560 2023-08-25T09:56:28.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04c0d3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff04406ca30 con 0x7ff044152d80 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04c0d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04406cf90 con 0x7ff044070cf0 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04966e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff0440c4560 0x7ff0440c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04966e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff0440c4560 0x7ff0440c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36850/0 (socket says 172.21.15.71:36850) 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.303+0000 7ff04966e640 1 -- 172.21.15.71:0/1237323941 learned_addr learned my addr 172.21.15.71:0/1237323941 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff049e6f640 1 --2- 172.21.15.71:0/1237323941 >> v2:172.21.15.164:3300/0 conn(0x7ff044152d80 0x7ff04406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff04a670640 1 --2- 172.21.15.71:0/1237323941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044070cf0 0x7ff0440c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff04966e640 1 -- 172.21.15.71:0/1237323941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044070cf0 msgr2=0x7ff0440c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:28.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff04966e640 1 --2- 172.21.15.71:0/1237323941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044070cf0 0x7ff0440c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff04966e640 1 -- 172.21.15.71:0/1237323941 >> v2:172.21.15.164:3300/0 conn(0x7ff044152d80 msgr2=0x7ff04406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:28.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff04966e640 1 --2- 172.21.15.71:0/1237323941 >> v2:172.21.15.164:3300/0 conn(0x7ff044152d80 0x7ff04406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff04966e640 1 -- 172.21.15.71:0/1237323941 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044158fd0 con 0x7ff0440c4560 2023-08-25T09:56:28.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.304+0000 7ff049e6f640 1 --2- 172.21.15.71:0/1237323941 >> v2:172.21.15.164:3300/0 conn(0x7ff044152d80 0x7ff04406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:56:28.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.305+0000 7ff04966e640 1 --2- 172.21.15.71:0/1237323941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff0440c6080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff04406dee0 tx=0x7ff03402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20796ec786cac0c7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:28.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.306+0000 7ff048e6d640 1 -- 172.21.15.71:0/1237323941 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340317d0 con 0x7ff0440c4560 2023-08-25T09:56:28.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.306+0000 7ff048e6d640 1 -- 172.21.15.71:0/1237323941 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034031930 con 0x7ff0440c4560 2023-08-25T09:56:28.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.306+0000 7ff048e6d640 1 -- 172.21.15.71:0/1237323941 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034031cc0 con 0x7ff0440c4560 2023-08-25T09:56:28.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.306+0000 7ff04c0d3640 1 -- 172.21.15.71:0/1237323941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 msgr2=0x7ff0440c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:28.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.306+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/1237323941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff0440c6080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff04406dee0 tx=0x7ff03402b870 comp rx=0 tx=0).stop 2023-08-25T09:56:28.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 -- 172.21.15.71:0/1237323941 shutdown_connections 2023-08-25T09:56:28.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/1237323941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044070cf0 0x7ff0440c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/1237323941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff0440c6080 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/1237323941 >> v2:172.21.15.164:3300/0 conn(0x7ff044152d80 0x7ff04406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 -- 172.21.15.71:0/1237323941 >> 172.21.15.71:0/1237323941 conn(0x7ff0440654f0 msgr2=0x7ff044065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:28.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 -- 172.21.15.71:0/1237323941 shutdown_connections 2023-08-25T09:56:28.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.307+0000 7ff04c0d3640 1 -- 172.21.15.71:0/1237323941 wait complete. 2023-08-25T09:56:28.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.308+0000 7ff04c0d3640 1 Processor -- start 2023-08-25T09:56:28.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.309+0000 7ff04c0d3640 1 -- start start 2023-08-25T09:56:28.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04c0d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff044070cf0 0x7ff044158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04c0d3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff044156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04c0d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044152d80 0x7ff044149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04c0d3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440c2070 con 0x7ff0440c4560 2023-08-25T09:56:28.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04c0d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04414c440 con 0x7ff044070cf0 2023-08-25T09:56:28.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04c0d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441571e0 con 0x7ff044152d80 2023-08-25T09:56:28.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff044156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff044156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36860/0 (socket says 172.21.15.71:36860) 2023-08-25T09:56:28.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 -- 172.21.15.71:0/3409608413 learned_addr learned my addr 172.21.15.71:0/3409608413 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:28.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04a670640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044152d80 0x7ff044149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff049e6f640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff044070cf0 0x7ff044158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 -- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044152d80 msgr2=0x7ff044149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:28.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044152d80 0x7ff044149970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 -- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff044070cf0 msgr2=0x7ff044158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:28.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff044070cf0 0x7ff044158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.310+0000 7ff04966e640 1 -- 172.21.15.71:0/3409608413 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340094d0 con 0x7ff0440c4560 2023-08-25T09:56:28.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.311+0000 7ff04966e640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff044156ca0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff034000c00 tx=0x7ff0340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:28.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.311+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034032870 con 0x7ff0440c4560 2023-08-25T09:56:28.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.311+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034032a10 con 0x7ff0440c4560 2023-08-25T09:56:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.311+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0440c22a0 con 0x7ff0440c4560 2023-08-25T09:56:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.311+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff034032dc0 con 0x7ff0440c4560 2023-08-25T09:56:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.311+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0440c2db0 con 0x7ff0440c4560 2023-08-25T09:56:28.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.312+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff044069160 con 0x7ff0440c4560 2023-08-25T09:56:28.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.313+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff034030070 con 0x7ff0440c4560 2023-08-25T09:56:28.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.314+0000 7ff03a7fc640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff02806a050 0x7ff02806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:28.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.314+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 7529+0+0 (secure 0 0 0) 0x7ff0340aaa10 con 0x7ff0440c4560 2023-08-25T09:56:28.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.314+0000 7ff049e6f640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff02806a050 0x7ff02806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:28.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.315+0000 7ff049e6f640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff02806a050 0x7ff02806c510 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff04000d8b0 tx=0x7ff040008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:28.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.318+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff034075b30 con 0x7ff0440c4560 2023-08-25T09:56:28.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.544+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0441524b0 con 0x7ff0440c4560 2023-08-25T09:56:28.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.545+0000 7ff03a7fc640 1 -- 172.21.15.71:0/3409608413 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5750 (secure 0 0 0) 0x7ff034079150 con 0x7ff0440c4560 2023-08-25T09:56:28.547 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:28.547 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:26.670426+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:resolve","state_seq":14,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:28.548 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 48 2023-08-25T09:56:28.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff02806a050 msgr2=0x7ff02806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:28.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff02806a050 0x7ff02806c510 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff04000d8b0 tx=0x7ff040008040 comp rx=0 tx=0).stop 2023-08-25T09:56:28.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 msgr2=0x7ff044156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:28.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff044156ca0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff034000c00 tx=0x7ff0340046c0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 shutdown_connections 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff02806a050 0x7ff02806c510 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff044152d80 0x7ff044149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff0440c4560 0x7ff044156ca0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 --2- 172.21.15.71:0/3409608413 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff044070cf0 0x7ff044158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 >> 172.21.15.71:0/3409608413 conn(0x7ff0440654f0 msgr2=0x7ff04406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 shutdown_connections 2023-08-25T09:56:28.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:28.548+0000 7ff04c0d3640 1 -- 172.21.15.71:0/3409608413 wait complete. 2023-08-25T09:56:28.568 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:30.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:30.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.796+0000 7f13fbfff640 1 Processor -- start 2023-08-25T09:56:30.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.796+0000 7f13fbfff640 1 -- start start 2023-08-25T09:56:30.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.797+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc1542d0 0x7f13fc1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.797+0000 7f13fbfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f13fc069850 0x7f13fc0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.797+0000 7f13fbfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f13fc0c75c0 0x7f13fc0c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.797+0000 7f13fbfff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13fc0c4b60 con 0x7f13fc0c75c0 2023-08-25T09:56:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.797+0000 7f13fbfff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13fc157bf0 con 0x7f13fc069850 2023-08-25T09:56:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.797+0000 7f13fbfff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc06a130 con 0x7f13fc1542d0 2023-08-25T09:56:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f13fc0c75c0 0x7f13fc0c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f13fc0c75c0 0x7f13fc0c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54916/0 (socket says 172.21.15.71:54916) 2023-08-25T09:56:30.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 -- 172.21.15.71:0/151295659 learned_addr learned my addr 172.21.15.71:0/151295659 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:30.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13faffd640 1 --2- 172.21.15.71:0/151295659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc1542d0 0x7f13fc1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:30.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fa7fc640 1 --2- 172.21.15.71:0/151295659 >> v2:172.21.15.164:3300/0 conn(0x7f13fc069850 0x7f13fc0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:30.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 -- 172.21.15.71:0/151295659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc1542d0 msgr2=0x7f13fc1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 --2- 172.21.15.71:0/151295659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc1542d0 0x7f13fc1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 -- 172.21.15.71:0/151295659 >> v2:172.21.15.164:3300/0 conn(0x7f13fc069850 msgr2=0x7f13fc0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 --2- 172.21.15.71:0/151295659 >> v2:172.21.15.164:3300/0 conn(0x7f13fc069850 0x7f13fc0c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.798+0000 7f13fb7fe640 1 -- 172.21.15.71:0/151295659 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc1546b0 con 0x7f13fc0c75c0 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.799+0000 7f13fb7fe640 1 --2- 172.21.15.71:0/151295659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0c4fb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f13fc06dca0 tx=0x7f13f402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e3fde0c8d4f2f4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.800+0000 7f13f9ffb640 1 -- 172.21.15.71:0/151295659 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f402cb00 con 0x7f13fc0c75c0 2023-08-25T09:56:30.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.800+0000 7f13f9ffb640 1 -- 172.21.15.71:0/151295659 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f402cc60 con 0x7f13fc0c75c0 2023-08-25T09:56:30.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.800+0000 7f13f9ffb640 1 -- 172.21.15.71:0/151295659 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f40343f0 con 0x7f13fc0c75c0 2023-08-25T09:56:30.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.800+0000 7f13fbfff640 1 -- 172.21.15.71:0/151295659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 msgr2=0x7f13fc0c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:30.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.800+0000 7f13fbfff640 1 --2- 172.21.15.71:0/151295659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0c4fb0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f13fc06dca0 tx=0x7f13f402bf60 comp rx=0 tx=0).stop 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 -- 172.21.15.71:0/151295659 shutdown_connections 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 --2- 172.21.15.71:0/151295659 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0c4fb0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 --2- 172.21.15.71:0/151295659 >> v2:172.21.15.164:3300/0 conn(0x7f13fc069850 0x7f13fc0c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 --2- 172.21.15.71:0/151295659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc1542d0 0x7f13fc1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 -- 172.21.15.71:0/151295659 >> 172.21.15.71:0/151295659 conn(0x7f13fc06c280 msgr2=0x7f13fc06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 -- 172.21.15.71:0/151295659 shutdown_connections 2023-08-25T09:56:30.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.801+0000 7f13fbfff640 1 -- 172.21.15.71:0/151295659 wait complete. 2023-08-25T09:56:30.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.803+0000 7f13fbfff640 1 Processor -- start 2023-08-25T09:56:30.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.803+0000 7f13fbfff640 1 -- start start 2023-08-25T09:56:30.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc069850 0x7f13fc156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13fc1542d0 0x7f13fc14fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fbfff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc150d20 con 0x7f13fc0c75c0 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fbfff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc151f10 con 0x7f13fc1542d0 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fbfff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc155ce0 con 0x7f13fc069850 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fa7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fa7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54926/0 (socket says 172.21.15.71:54926) 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.804+0000 7f13fa7fc640 1 -- 172.21.15.71:0/3552355074 learned_addr learned my addr 172.21.15.71:0/3552355074 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:30.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13faffd640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc069850 0x7f13fc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fa7fc640 1 -- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc069850 msgr2=0x7f13fc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fa7fc640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc069850 0x7f13fc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fa7fc640 1 -- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13fc1542d0 msgr2=0x7f13fc14fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fa7fc640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13fc1542d0 0x7f13fc14fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:30.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fa7fc640 1 -- 172.21.15.71:0/3552355074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4009b30 con 0x7f13fc0c75c0 2023-08-25T09:56:30.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fa7fc640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0ca450 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f13e400d830 tx=0x7f13e400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:30.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e400be10 con 0x7f13fc0c75c0 2023-08-25T09:56:30.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e40098b0 con 0x7f13fc0c75c0 2023-08-25T09:56:30.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc155f60 con 0x7f13fc0c75c0 2023-08-25T09:56:30.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.805+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4009a40 con 0x7f13fc0c75c0 2023-08-25T09:56:30.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.806+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc153940 con 0x7f13fc0c75c0 2023-08-25T09:56:30.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.807+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13fc152050 con 0x7f13fc0c75c0 2023-08-25T09:56:30.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.808+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f13e4009ba0 con 0x7f13fc0c75c0 2023-08-25T09:56:30.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.809+0000 7f13f37fe640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f13dc06a080 0x7f13dc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:30.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.809+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 7529+0+0 (secure 0 0 0) 0x7f13e4088e60 con 0x7f13fc0c75c0 2023-08-25T09:56:30.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.809+0000 7f13faffd640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f13dc06a080 0x7f13dc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:30.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.810+0000 7f13faffd640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f13dc06a080 0x7f13dc06c540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f13ec006fd0 tx=0x7f13ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:30.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:30.816+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f13e4054020 con 0x7f13fc0c75c0 2023-08-25T09:56:31.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.042+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13fc04e420 con 0x7f13fc0c75c0 2023-08-25T09:56:31.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.043+0000 7f13f37fe640 1 -- 172.21.15.71:0/3552355074 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f13e4057640 con 0x7f13fc0c75c0 2023-08-25T09:56:31.045 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:31.045 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:30.707943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:resolve","state_seq":14,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:31.046 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 49 2023-08-25T09:56:31.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.045+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f13dc06a080 msgr2=0x7f13dc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:31.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.045+0000 7f13fbfff640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f13dc06a080 0x7f13dc06c540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f13ec006fd0 tx=0x7f13ec008040 comp rx=0 tx=0).stop 2023-08-25T09:56:31.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.045+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 msgr2=0x7f13fc0ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:31.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.045+0000 7f13fbfff640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0ca450 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f13e400d830 tx=0x7f13e400dcf0 comp rx=0 tx=0).stop 2023-08-25T09:56:31.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 shutdown_connections 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13fc1542d0 0x7f13fc14fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f13dc06a080 0x7f13dc06c540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13fc0c75c0 0x7f13fc0ca450 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 --2- 172.21.15.71:0/3552355074 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13fc069850 0x7f13fc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 >> 172.21.15.71:0/3552355074 conn(0x7f13fc06c280 msgr2=0x7f13fc06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 shutdown_connections 2023-08-25T09:56:31.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:31.046+0000 7f13fbfff640 1 -- 172.21.15.71:0/3552355074 wait complete. 2023-08-25T09:56:31.067 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:33.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:33.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.293+0000 7ff19aaa6640 1 Processor -- start 2023-08-25T09:56:33.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.293+0000 7ff19aaa6640 1 -- start start 2023-08-25T09:56:33.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff19aaa6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1940c3660 0x7ff1940c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:33.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff19aaa6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1940c1dc0 con 0x7ff1941509d0 2023-08-25T09:56:33.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff19aaa6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff194156e00 con 0x7ff1940c3a40 2023-08-25T09:56:33.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff19aaa6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff194157a80 con 0x7ff1940c3660 2023-08-25T09:56:33.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff198842640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1940c3660 0x7ff1940c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:33.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff199043640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff1941509d0 0x7ff194150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51128/0 (socket says 172.21.15.71:51128) 2023-08-25T09:56:33.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff199043640 1 -- 172.21.15.71:0/2558914655 learned_addr learned my addr 172.21.15.71:0/2558914655 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:33.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff198842640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1940c3660 0x7ff1940c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54936/0 (socket says 172.21.15.71:54936) 2023-08-25T09:56:33.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff198842640 1 -- 172.21.15.71:0/2558914655 >> v1:172.21.15.164:6789/0 conn(0x7ff1940c3a40 legacy=0x7ff19406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:33.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.295+0000 7ff198842640 1 -- 172.21.15.71:0/2558914655 >> v1:172.21.15.71:6789/0 conn(0x7ff1941509d0 legacy=0x7ff194150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:33.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.296+0000 7ff198842640 1 -- 172.21.15.71:0/2558914655 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1941570e0 con 0x7ff1940c3660 2023-08-25T09:56:33.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.296+0000 7ff1937fe640 1 -- 172.21.15.71:0/2558914655 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722325868 0 0) 0x7ff1940c1dc0 con 0x7ff1941509d0 2023-08-25T09:56:33.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.297+0000 7ff198842640 1 --2- 172.21.15.71:0/2558914655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1940c1710 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff1880097a0 tx=0x7ff18802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0711acc416bdb53 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:33.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.297+0000 7ff1937fe640 1 -- 172.21.15.71:0/2558914655 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188030610 con 0x7ff1940c3660 2023-08-25T09:56:33.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.297+0000 7ff1937fe640 1 -- 172.21.15.71:0/2558914655 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff188030770 con 0x7ff1940c3660 2023-08-25T09:56:33.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.297+0000 7ff1937fe640 1 -- 172.21.15.71:0/2558914655 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188030b00 con 0x7ff1940c3660 2023-08-25T09:56:33.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2558914655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 msgr2=0x7ff1940c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:33.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2558914655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1940c1710 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff1880097a0 tx=0x7ff18802b960 comp rx=0 tx=0).stop 2023-08-25T09:56:33.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2558914655 shutdown_connections 2023-08-25T09:56:33.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2558914655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1940c1710 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2558914655 >> 172.21.15.71:0/2558914655 conn(0x7ff1940654f0 msgr2=0x7ff194065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:33.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2558914655 shutdown_connections 2023-08-25T09:56:33.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.298+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2558914655 wait complete. 2023-08-25T09:56:33.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.299+0000 7ff19aaa6640 1 Processor -- start 2023-08-25T09:56:33.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.300+0000 7ff19aaa6640 1 -- start start 2023-08-25T09:56:33.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff19aaa6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1941e1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff19aaa6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1940c3a40 0x7ff1941f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff19aaa6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1941509d0 0x7ff1941f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff19aaa6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1941e1730 con 0x7ff1940c3660 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff19aaa6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff194151a20 con 0x7ff1941509d0 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff19aaa6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff194154480 con 0x7ff1940c3a40 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1941e1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:33.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1941e1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54944/0 (socket says 172.21.15.71:54944) 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 -- 172.21.15.71:0/2837385688 learned_addr learned my addr 172.21.15.71:0/2837385688 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 -- 172.21.15.71:0/2837385688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1940c3a40 msgr2=0x7ff1941f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1940c3a40 0x7ff1941f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 -- 172.21.15.71:0/2837385688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1941509d0 msgr2=0x7ff1941f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1941509d0 0x7ff1941f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.301+0000 7ff198842640 1 -- 172.21.15.71:0/2837385688 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff188009400 con 0x7ff1940c3660 2023-08-25T09:56:33.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.302+0000 7ff198842640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1941e1b50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff18802bf70 tx=0x7ff1880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.302+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18803e070 con 0x7ff1940c3660 2023-08-25T09:56:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.302+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff188039a10 con 0x7ff1940c3660 2023-08-25T09:56:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.302+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19406e620 con 0x7ff1940c3660 2023-08-25T09:56:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.302+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188039da0 con 0x7ff1940c3660 2023-08-25T09:56:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.302+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1941541a0 con 0x7ff1940c3660 2023-08-25T09:56:33.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.303+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1941e1870 con 0x7ff1940c3660 2023-08-25T09:56:33.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.304+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff1880423f0 con 0x7ff1940c3660 2023-08-25T09:56:33.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.306+0000 7ff1917fa640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff17806a050 0x7ff17806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:33.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.306+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 7529+0+0 (secure 0 0 0) 0x7ff1880aaa90 con 0x7ff1940c3660 2023-08-25T09:56:33.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.306+0000 7ff193fff640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff17806a050 0x7ff17806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:33.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.307+0000 7ff193fff640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff17806a050 0x7ff17806c510 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff19406c650 tx=0x7ff1800079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:33.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.310+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff1941e1870 con 0x7ff1940c3660 2023-08-25T09:56:33.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.535+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1941e1870 con 0x7ff1940c3660 2023-08-25T09:56:33.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.536+0000 7ff1917fa640 1 -- 172.21.15.71:0/2837385688 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5750 (secure 0 0 0) 0x7ff1941e1870 con 0x7ff1940c3660 2023-08-25T09:56:33.538 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:33.538 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:32.721265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:33.538 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 50 2023-08-25T09:56:33.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff17806a050 msgr2=0x7ff17806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:33.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff17806a050 0x7ff17806c510 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff19406c650 tx=0x7ff1800079b0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 msgr2=0x7ff1941e1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1941e1b50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff18802bf70 tx=0x7ff1880046c0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 shutdown_connections 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1941509d0 0x7ff1941f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff17806a050 0x7ff17806c510 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1940c3a40 0x7ff1941f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 --2- 172.21.15.71:0/2837385688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1940c3660 0x7ff1941e1b50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:33.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 >> 172.21.15.71:0/2837385688 conn(0x7ff1940654f0 msgr2=0x7ff194069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:33.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 shutdown_connections 2023-08-25T09:56:33.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:33.539+0000 7ff19aaa6640 1 -- 172.21.15.71:0/2837385688 wait complete. 2023-08-25T09:56:33.559 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:35.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:35.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.807+0000 7f87f4ca6640 1 Processor -- start 2023-08-25T09:56:35.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.807+0000 7f87f4ca6640 1 -- start start 2023-08-25T09:56:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87f4ca6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f87f00c3660 0x7f87f00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87f4ca6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 0x7f87f0150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87f4ca6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f00c1dc0 con 0x7f87f00c3a40 2023-08-25T09:56:35.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87f4ca6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f0157a80 con 0x7f87f00c3660 2023-08-25T09:56:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87f4ca6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0151360 con 0x7f87f01509d0 2023-08-25T09:56:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87ee59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f87f00c3660 0x7f87f00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87ee59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f87f00c3660 0x7f87f00c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54958/0 (socket says 172.21.15.71:54958) 2023-08-25T09:56:35.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.808+0000 7f87ee59c640 1 -- 172.21.15.71:0/615838725 learned_addr learned my addr 172.21.15.71:0/615838725 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.809+0000 7f87ee59c640 1 -- 172.21.15.71:0/615838725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 msgr2=0x7f87f0150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.809+0000 7f87ee59c640 1 --2- 172.21.15.71:0/615838725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 0x7f87f0150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.809+0000 7f87ee59c640 1 -- 172.21.15.71:0/615838725 >> v1:172.21.15.164:6789/0 conn(0x7f87f00c3a40 legacy=0x7f87f006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:35.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.809+0000 7f87ee59c640 1 -- 172.21.15.71:0/615838725 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0156e00 con 0x7f87f00c3660 2023-08-25T09:56:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.810+0000 7f87ee59c640 1 --2- 172.21.15.71:0/615838725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3660 0x7f87f00c1710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009870 tx=0x7f87e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc2a4f934b1c9fd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.810+0000 7f87edd9b640 1 -- 172.21.15.71:0/615838725 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0030840 con 0x7f87f00c3660 2023-08-25T09:56:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.810+0000 7f87edd9b640 1 -- 172.21.15.71:0/615838725 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e00309a0 con 0x7f87f00c3660 2023-08-25T09:56:35.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.810+0000 7f87edd9b640 1 -- 172.21.15.71:0/615838725 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0030d30 con 0x7f87f00c3660 2023-08-25T09:56:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 -- 172.21.15.71:0/615838725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3660 msgr2=0x7f87f00c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/615838725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3660 0x7f87f00c1710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009870 tx=0x7f87e002b980 comp rx=0 tx=0).stop 2023-08-25T09:56:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 -- 172.21.15.71:0/615838725 shutdown_connections 2023-08-25T09:56:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/615838725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 0x7f87f0150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/615838725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3660 0x7f87f00c1710 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 -- 172.21.15.71:0/615838725 >> 172.21.15.71:0/615838725 conn(0x7f87f00654f0 msgr2=0x7f87f0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 -- 172.21.15.71:0/615838725 shutdown_connections 2023-08-25T09:56:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.811+0000 7f87f4ca6640 1 -- 172.21.15.71:0/615838725 wait complete. 2023-08-25T09:56:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.813+0000 7f87f4ca6640 1 Processor -- start 2023-08-25T09:56:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.813+0000 7f87f4ca6640 1 -- start start 2023-08-25T09:56:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87f4ca6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f00c3660 0x7f87f0147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87f4ca6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 0x7f87f01ee8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87f4ca6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 0x7f87f01eede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87f4ca6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0147ab0 con 0x7f87f00c3a40 2023-08-25T09:56:35.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87f4ca6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f01515f0 con 0x7f87f00c3660 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87f4ca6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f01518d0 con 0x7f87f01509d0 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 0x7f87f01ee8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 0x7f87f01ee8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54962/0 (socket says 172.21.15.71:54962) 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 -- 172.21.15.71:0/3629615725 learned_addr learned my addr 172.21.15.71:0/3629615725 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 -- 172.21.15.71:0/3629615725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 msgr2=0x7f87f01eede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 0x7f87f01eede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:35.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 -- 172.21.15.71:0/3629615725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f00c3660 msgr2=0x7f87f0147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:35.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f00c3660 0x7f87f0147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:35.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.814+0000 7f87e7fff640 1 -- 172.21.15.71:0/3629615725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e00094d0 con 0x7f87f00c3a40 2023-08-25T09:56:35.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.815+0000 7f87e7fff640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 0x7f87f01ee8a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f87d800b3c0 tx=0x7f87d800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:35.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.815+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8011830 con 0x7f87f00c3a40 2023-08-25T09:56:35.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.815+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8011990 con 0x7f87f00c3a40 2023-08-25T09:56:35.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.815+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f01edae0 con 0x7f87f00c3a40 2023-08-25T09:56:35.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.815+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8011d60 con 0x7f87f00c3a40 2023-08-25T09:56:35.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.815+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f01ee0a0 con 0x7f87f00c3a40 2023-08-25T09:56:35.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.816+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f0147210 con 0x7f87f00c3a40 2023-08-25T09:56:35.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.817+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f87d801b460 con 0x7f87f00c3a40 2023-08-25T09:56:35.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.818+0000 7f87e6ffd640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d006a080 0x7f87d006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:35.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.818+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 7617+0+0 (secure 0 0 0) 0x7f87d80888f0 con 0x7f87f00c3a40 2023-08-25T09:56:35.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.818+0000 7f87ee59c640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d006a080 0x7f87d006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:35.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.819+0000 7f87ee59c640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d006a080 0x7f87d006c540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f87e0036000 tx=0x7f87e003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:35.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:35.823+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f87f0147210 con 0x7f87f00c3a40 2023-08-25T09:56:36.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.050+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f0147210 con 0x7f87f00c3a40 2023-08-25T09:56:36.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.051+0000 7f87e6ffd640 1 -- 172.21.15.71:0/3629615725 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f87f0147210 con 0x7f87f00c3a40 2023-08-25T09:56:36.053 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:36.053 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:35.877146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:36.053 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 51 2023-08-25T09:56:36.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.053+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d006a080 msgr2=0x7f87d006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:36.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.053+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d006a080 0x7f87d006c540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f87e0036000 tx=0x7f87e003b040 comp rx=0 tx=0).stop 2023-08-25T09:56:36.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.053+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 msgr2=0x7f87f01ee8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:36.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.053+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 0x7f87f01ee8a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f87d800b3c0 tx=0x7f87d800b880 comp rx=0 tx=0).stop 2023-08-25T09:56:36.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 shutdown_connections 2023-08-25T09:56:36.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f87d006a080 0x7f87d006c540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:36.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87f01509d0 0x7f87f01eede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:36.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87f00c3a40 0x7f87f01ee8a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:36.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 --2- 172.21.15.71:0/3629615725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87f00c3660 0x7f87f0147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:36.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 >> 172.21.15.71:0/3629615725 conn(0x7f87f00654f0 msgr2=0x7f87f00c9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:36.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 shutdown_connections 2023-08-25T09:56:36.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:36.054+0000 7f87f4ca6640 1 -- 172.21.15.71:0/3629615725 wait complete. 2023-08-25T09:56:36.076 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 51, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 51, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4290": { "addr": "172.21.15.71:6837/3404832976", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3404832976, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3404832976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 12, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 13 }, "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:56:35.877146+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 4290, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [] } 2023-08-25T09:56:38.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:38.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.295+0000 7fb1ca9dc640 1 Processor -- start 2023-08-25T09:56:38.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.295+0000 7fb1ca9dc640 1 -- start start 2023-08-25T09:56:38.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1ca9dc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb1c40c3a40 0x7fb1c40c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1ca9dc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb1c41509d0 0x7fb1c4150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1ca9dc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c40c1dc0 con 0x7fb1c40c3660 2023-08-25T09:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1ca9dc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c4157a80 con 0x7fb1c40c3a40 2023-08-25T09:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1ca9dc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1c4151320 con 0x7fb1c41509d0 2023-08-25T09:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1bb7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb1c40c3a40 0x7fb1c40c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:38.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1c3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb1c40c3660 0x7fb1c406ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51134/0 (socket says 172.21.15.71:51134) 2023-08-25T09:56:38.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1bb7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb1c40c3a40 0x7fb1c40c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54966/0 (socket says 172.21.15.71:54966) 2023-08-25T09:56:38.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1c3fff640 1 -- 172.21.15.71:0/1461528097 learned_addr learned my addr 172.21.15.71:0/1461528097 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:38.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.297+0000 7fb1c8f79640 1 --2- 172.21.15.71:0/1461528097 >> v2:172.21.15.164:3300/0 conn(0x7fb1c41509d0 0x7fb1c4150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.298+0000 7fb1bb7fe640 1 -- 172.21.15.71:0/1461528097 >> v2:172.21.15.164:3300/0 conn(0x7fb1c41509d0 msgr2=0x7fb1c4150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.298+0000 7fb1bb7fe640 1 --2- 172.21.15.71:0/1461528097 >> v2:172.21.15.164:3300/0 conn(0x7fb1c41509d0 0x7fb1c4150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.298+0000 7fb1bb7fe640 1 -- 172.21.15.71:0/1461528097 >> v1:172.21.15.71:6789/0 conn(0x7fb1c40c3660 legacy=0x7fb1c406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.298+0000 7fb1bb7fe640 1 -- 172.21.15.71:0/1461528097 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4156e00 con 0x7fb1c40c3a40 2023-08-25T09:56:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.299+0000 7fb1bb7fe640 1 --2- 172.21.15.71:0/1461528097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c40c3a40 0x7fb1c40c1710 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40663f0 tx=0x7fb1ac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56361d2d9c9416ae server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:38.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.299+0000 7fb1c37fe640 1 -- 172.21.15.71:0/1461528097 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac02cd10 con 0x7fb1c40c3a40 2023-08-25T09:56:38.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1c37fe640 1 -- 172.21.15.71:0/1461528097 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ac02ce70 con 0x7fb1c40c3a40 2023-08-25T09:56:38.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1c37fe640 1 -- 172.21.15.71:0/1461528097 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac034640 con 0x7fb1c40c3a40 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/1461528097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c40c3a40 msgr2=0x7fb1c40c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/1461528097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c40c3a40 0x7fb1c40c1710 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40663f0 tx=0x7fb1ac02bf60 comp rx=0 tx=0).stop 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/1461528097 shutdown_connections 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/1461528097 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c40c3a40 0x7fb1c40c1710 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/1461528097 >> v2:172.21.15.164:3300/0 conn(0x7fb1c41509d0 0x7fb1c4150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/1461528097 >> 172.21.15.71:0/1461528097 conn(0x7fb1c40654f0 msgr2=0x7fb1c4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/1461528097 shutdown_connections 2023-08-25T09:56:38.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.300+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/1461528097 wait complete. 2023-08-25T09:56:38.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.302+0000 7fb1ca9dc640 1 Processor -- start 2023-08-25T09:56:38.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.302+0000 7fb1ca9dc640 1 -- start start 2023-08-25T09:56:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.303+0000 7fb1ca9dc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40c3660 0x7fb1c406a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.303+0000 7fb1ca9dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c40c3a40 0x7fb1c4159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.303+0000 7fb1ca9dc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 0x7fb1c41596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.303+0000 7fb1ca9dc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c414db70 con 0x7fb1c41509d0 2023-08-25T09:56:38.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.303+0000 7fb1ca9dc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c414cf60 con 0x7fb1c40c3a40 2023-08-25T09:56:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.303+0000 7fb1ca9dc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c414d240 con 0x7fb1c40c3660 2023-08-25T09:56:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 0x7fb1c41596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 0x7fb1c41596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54978/0 (socket says 172.21.15.71:54978) 2023-08-25T09:56:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 -- 172.21.15.71:0/2880938491 learned_addr learned my addr 172.21.15.71:0/2880938491 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c3fff640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40c3660 0x7fb1c406a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1bb7fe640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c40c3a40 0x7fb1c4159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 -- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40c3660 msgr2=0x7fb1c406a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40c3660 0x7fb1c406a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 -- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c40c3a40 msgr2=0x7fb1c4159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c40c3a40 0x7fb1c4159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 -- 172.21.15.71:0/2880938491 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac009a70 con 0x7fb1c41509d0 2023-08-25T09:56:38.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1bb7fe640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c40c3a40 0x7fb1c4159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.304+0000 7fb1c8f79640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 0x7fb1c41596c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00d5d0 tx=0x7fb1bc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.305+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0125f0 con 0x7fb1c41509d0 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.305+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc012770 con 0x7fb1c41509d0 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.305+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc012b20 con 0x7fb1c41509d0 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.305+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c414d520 con 0x7fb1c41509d0 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.305+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c406aca0 con 0x7fb1c41509d0 2023-08-25T09:56:38.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.306+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c40c29c0 con 0x7fb1c41509d0 2023-08-25T09:56:38.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.307+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb1bc012c80 con 0x7fb1c41509d0 2023-08-25T09:56:38.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.308+0000 7fb1c17fa640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1a406a050 0x7fb1a406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:38.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.309+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 7617+0+0 (secure 0 0 0) 0x7fb1bc089110 con 0x7fb1c41509d0 2023-08-25T09:56:38.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.309+0000 7fb1c3fff640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1a406a050 0x7fb1a406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:38.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.309+0000 7fb1c3fff640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1a406a050 0x7fb1a406c510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4005880 tx=0x7fb1b4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:38.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.313+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb1bc054150 con 0x7fb1c41509d0 2023-08-25T09:56:38.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.539+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1c4069160 con 0x7fb1c41509d0 2023-08-25T09:56:38.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.540+0000 7fb1c17fa640 1 -- 172.21.15.71:0/2880938491 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fb1bc057770 con 0x7fb1c41509d0 2023-08-25T09:56:38.542 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:38.543 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:37.897227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:38.543 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 52 2023-08-25T09:56:38.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.542+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1a406a050 msgr2=0x7fb1a406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:38.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.542+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1a406a050 0x7fb1a406c510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4005880 tx=0x7fb1b4009040 comp rx=0 tx=0).stop 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 msgr2=0x7fb1c41596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 0x7fb1c41596c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00d5d0 tx=0x7fb1bc00da90 comp rx=0 tx=0).stop 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 shutdown_connections 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb1c40c3a40 0x7fb1c4159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1a406a050 0x7fb1a406c510 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb1c41509d0 0x7fb1c41596c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 --2- 172.21.15.71:0/2880938491 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb1c40c3660 0x7fb1c406a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:38.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 >> 172.21.15.71:0/2880938491 conn(0x7fb1c40654f0 msgr2=0x7fb1c414efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:38.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 shutdown_connections 2023-08-25T09:56:38.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:38.543+0000 7fb1ca9dc640 1 -- 172.21.15.71:0/2880938491 wait complete. 2023-08-25T09:56:38.562 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:40.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:40.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.777+0000 7f93d0606640 1 Processor -- start 2023-08-25T09:56:40.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.777+0000 7f93d0606640 1 -- start start 2023-08-25T09:56:40.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93d0606640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93c81530c0 0x7f93c80c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:40.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93d0606640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f93c81592d0 0x7f93c8159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:40.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93d0606640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c8158e40 con 0x7f93c80c7740 2023-08-25T09:56:40.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93d0606640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c806bc80 con 0x7f93c81592d0 2023-08-25T09:56:40.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93d0606640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c81534a0 con 0x7f93c81530c0 2023-08-25T09:56:40.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93ceba3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f93c81592d0 0x7f93c8159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:40.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93ceba3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f93c81592d0 0x7f93c8159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45336/0 (socket says 172.21.15.71:45336) 2023-08-25T09:56:40.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93ceba3640 1 -- 172.21.15.71:0/2232314060 learned_addr learned my addr 172.21.15.71:0/2232314060 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:40.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.779+0000 7f93cdba1640 1 --1- 172.21.15.71:0/2232314060 >> v1:172.21.15.71:6789/0 conn(0x7f93c80c7740 0x7f93c806a960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58058/0 (socket says 172.21.15.71:58058) 2023-08-25T09:56:40.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.780+0000 7f93ce3a2640 1 --2- 172.21.15.71:0/2232314060 >> v2:172.21.15.164:3300/0 conn(0x7f93c81530c0 0x7f93c80c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:40.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.780+0000 7f93ceba3640 1 -- 172.21.15.71:0/2232314060 >> v2:172.21.15.164:3300/0 conn(0x7f93c81530c0 msgr2=0x7f93c80c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.780+0000 7f93ceba3640 1 --2- 172.21.15.71:0/2232314060 >> v2:172.21.15.164:3300/0 conn(0x7f93c81530c0 0x7f93c80c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.780+0000 7f93ceba3640 1 -- 172.21.15.71:0/2232314060 >> v1:172.21.15.71:6789/0 conn(0x7f93c80c7740 legacy=0x7f93c806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.780+0000 7f93ceba3640 1 -- 172.21.15.71:0/2232314060 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80c7380 con 0x7f93c81592d0 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.781+0000 7f93ceba3640 1 --2- 172.21.15.71:0/2232314060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81592d0 0x7f93c8159760 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f93c8071b20 tx=0x7f93c002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb122da660db6f8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.781+0000 7f93cd3a0640 1 -- 172.21.15.71:0/2232314060 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c002f850 con 0x7f93c81592d0 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.781+0000 7f93cd3a0640 1 -- 172.21.15.71:0/2232314060 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c002f9b0 con 0x7f93c81592d0 2023-08-25T09:56:40.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.781+0000 7f93cd3a0640 1 -- 172.21.15.71:0/2232314060 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c002fd40 con 0x7f93c81592d0 2023-08-25T09:56:40.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 -- 172.21.15.71:0/2232314060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81592d0 msgr2=0x7f93c8159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:40.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 --2- 172.21.15.71:0/2232314060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81592d0 0x7f93c8159760 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f93c8071b20 tx=0x7f93c002ba30 comp rx=0 tx=0).stop 2023-08-25T09:56:40.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 -- 172.21.15.71:0/2232314060 shutdown_connections 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 --2- 172.21.15.71:0/2232314060 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81592d0 0x7f93c8159760 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 --2- 172.21.15.71:0/2232314060 >> v2:172.21.15.164:3300/0 conn(0x7f93c81530c0 0x7f93c80c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 -- 172.21.15.71:0/2232314060 >> 172.21.15.71:0/2232314060 conn(0x7f93c80654f0 msgr2=0x7f93c8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 -- 172.21.15.71:0/2232314060 shutdown_connections 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.782+0000 7f93d0606640 1 -- 172.21.15.71:0/2232314060 wait complete. 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.784+0000 7f93d0606640 1 Processor -- start 2023-08-25T09:56:40.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.784+0000 7f93d0606640 1 -- start start 2023-08-25T09:56:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93d0606640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80c7740 0x7f93c8155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93d0606640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 0x7f93c8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93d0606640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81592d0 0x7f93c81f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93d0606640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80c4de0 con 0x7f93c81530c0 2023-08-25T09:56:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93d0606640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c814e530 con 0x7f93c80c7740 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93d0606640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c814ddc0 con 0x7f93c81592d0 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93cdba1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 0x7f93c8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93cdba1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 0x7f93c8070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45338/0 (socket says 172.21.15.71:45338) 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.785+0000 7f93cdba1640 1 -- 172.21.15.71:0/2652451955 learned_addr learned my addr 172.21.15.71:0/2652451955 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93ce3a2640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80c7740 0x7f93c8155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93cdba1640 1 -- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81592d0 msgr2=0x7f93c81f12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:40.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93cdba1640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81592d0 0x7f93c81f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93cdba1640 1 -- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80c7740 msgr2=0x7f93c8155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93cdba1640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80c7740 0x7f93c8155ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93cdba1640 1 -- 172.21.15.71:0/2652451955 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c00094d0 con 0x7f93c81530c0 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93cdba1640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 0x7f93c8070a30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00c410 tx=0x7f93bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0074e0 con 0x7f93c81530c0 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc007640 con 0x7f93c81530c0 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c8156ce0 con 0x7f93c81530c0 2023-08-25T09:56:40.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.786+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0049e0 con 0x7f93c81530c0 2023-08-25T09:56:40.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.787+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c80c5860 con 0x7f93c81530c0 2023-08-25T09:56:40.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.788+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8069160 con 0x7f93c81530c0 2023-08-25T09:56:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.788+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f93bc004b40 con 0x7f93c81530c0 2023-08-25T09:56:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.789+0000 7f93c6ffd640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f93b406a050 0x7f93b406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.790+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 7617+0+0 (secure 0 0 0) 0x7f93bc0886f0 con 0x7f93c81530c0 2023-08-25T09:56:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.790+0000 7f93ce3a2640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f93b406a050 0x7f93b406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:40.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.790+0000 7f93ce3a2640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f93b406a050 0x7f93b406c510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009f30 tx=0x7f93b800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:40.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:40.794+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f93bc053690 con 0x7f93c81530c0 2023-08-25T09:56:41.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.025+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93c806f250 con 0x7f93c81530c0 2023-08-25T09:56:41.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.026+0000 7f93c6ffd640 1 -- 172.21.15.71:0/2652451955 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f93bc056cb0 con 0x7f93c81530c0 2023-08-25T09:56:41.028 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:41.028 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:37.897227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:41.029 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 52 2023-08-25T09:56:41.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f93b406a050 msgr2=0x7f93b406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f93b406a050 0x7f93b406c510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f93b8009f30 tx=0x7f93b800b040 comp rx=0 tx=0).stop 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 msgr2=0x7f93c8070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 0x7f93c8070a30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00c410 tx=0x7f93bc00c8d0 comp rx=0 tx=0).stop 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 shutdown_connections 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f93b406a050 0x7f93b406c510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f93c81592d0 0x7f93c81f12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f93c81530c0 0x7f93c8070a30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:41.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 --2- 172.21.15.71:0/2652451955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93c80c7740 0x7f93c8155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:41.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 >> 172.21.15.71:0/2652451955 conn(0x7f93c80654f0 msgr2=0x7f93c80658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:41.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 shutdown_connections 2023-08-25T09:56:41.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:41.031+0000 7f93d0606640 1 -- 172.21.15.71:0/2652451955 wait complete. 2023-08-25T09:56:41.051 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:43.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:43.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.276+0000 7f4a81245640 1 Processor -- start 2023-08-25T09:56:43.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.276+0000 7f4a81245640 1 -- start start 2023-08-25T09:56:43.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a81245640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4a7c1510e0 0x7f4a7c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:43.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a81245640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0c1dc0 con 0x7f4a7c0c3660 2023-08-25T09:56:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a81245640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c156e00 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a81245640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7c157a80 con 0x7f4a7c1510e0 2023-08-25T09:56:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a7ad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4a7c0c3660 0x7f4a7c06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58068/0 (socket says 172.21.15.71:58068) 2023-08-25T09:56:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a7ad9d640 1 -- 172.21.15.71:0/2783743950 learned_addr learned my addr 172.21.15.71:0/2783743950 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a7b59e640 1 --2- 172.21.15.71:0/2783743950 >> v2:172.21.15.71:3300/0 conn(0x7f4a7c1510e0 0x7f4a7c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a7b59e640 1 -- 172.21.15.71:0/2783743950 >> v1:172.21.15.164:6789/0 conn(0x7f4a7c0c3a40 legacy=0x7f4a7c1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a7b59e640 1 -- 172.21.15.71:0/2783743950 >> v1:172.21.15.71:6789/0 conn(0x7f4a7c0c3660 legacy=0x7f4a7c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a7b59e640 1 -- 172.21.15.71:0/2783743950 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c1570e0 con 0x7f4a7c1510e0 2023-08-25T09:56:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.278+0000 7f4a79d9b640 1 -- 172.21.15.71:0/2783743950 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885558581 0 0) 0x7f4a7c0c1dc0 con 0x7f4a7c0c3660 2023-08-25T09:56:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.279+0000 7f4a7b59e640 1 --2- 172.21.15.71:0/2783743950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c1510e0 0x7f4a7c0c1710 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31da2e638ecf09b2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a79d9b640 1 -- 172.21.15.71:0/2783743950 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7402cd00 con 0x7f4a7c1510e0 2023-08-25T09:56:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a79d9b640 1 -- 172.21.15.71:0/2783743950 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7402ce60 con 0x7f4a7c1510e0 2023-08-25T09:56:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a79d9b640 1 -- 172.21.15.71:0/2783743950 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74034660 con 0x7f4a7c1510e0 2023-08-25T09:56:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a81245640 1 -- 172.21.15.71:0/2783743950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c1510e0 msgr2=0x7f4a7c0c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a81245640 1 --2- 172.21.15.71:0/2783743950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c1510e0 0x7f4a7c0c1710 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7402bf50 comp rx=0 tx=0).stop 2023-08-25T09:56:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a81245640 1 -- 172.21.15.71:0/2783743950 shutdown_connections 2023-08-25T09:56:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.280+0000 7f4a81245640 1 --2- 172.21.15.71:0/2783743950 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c1510e0 0x7f4a7c0c1710 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.281+0000 7f4a81245640 1 -- 172.21.15.71:0/2783743950 >> 172.21.15.71:0/2783743950 conn(0x7f4a7c0654f0 msgr2=0x7f4a7c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.281+0000 7f4a81245640 1 -- 172.21.15.71:0/2783743950 shutdown_connections 2023-08-25T09:56:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.281+0000 7f4a81245640 1 -- 172.21.15.71:0/2783743950 wait complete. 2023-08-25T09:56:43.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.283+0000 7f4a81245640 1 Processor -- start 2023-08-25T09:56:43.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.283+0000 7f4a81245640 1 -- start start 2023-08-25T09:56:43.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a81245640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a7c0c3660 0x7f4a7c1e5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:43.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a81245640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 0x7f4a7c14bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:43.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a81245640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a7c1510e0 0x7f4a7c14c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:43.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a81245640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1e0980 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a81245640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c14ca00 con 0x7f4a7c0c3660 2023-08-25T09:56:43.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a81245640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1e0020 con 0x7f4a7c1510e0 2023-08-25T09:56:43.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a7a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 0x7f4a7c14bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:43.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a7a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 0x7f4a7c14bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45362/0 (socket says 172.21.15.71:45362) 2023-08-25T09:56:43.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.284+0000 7f4a7a59c640 1 -- 172.21.15.71:0/3633910079 learned_addr learned my addr 172.21.15.71:0/3633910079 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:43.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7b59e640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a7c1510e0 0x7f4a7c14c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:43.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7ad9d640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a7c0c3660 0x7f4a7c1e5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:43.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7a59c640 1 -- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a7c1510e0 msgr2=0x7f4a7c14c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:43.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7a59c640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a7c1510e0 0x7f4a7c14c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7a59c640 1 -- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a7c0c3660 msgr2=0x7f4a7c1e5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7a59c640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a7c0c3660 0x7f4a7c1e5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7a59c640 1 -- 172.21.15.71:0/3633910079 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a74009a70 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7ad9d640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a7c0c3660 0x7f4a7c1e5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a7a59c640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 0x7f4a7c14bf80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00c400 tx=0x7f4a6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c011790 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c011930 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c011ce0 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.285+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c15ab60 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.286+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c1e02c0 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.287+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c0c4e80 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.288+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4a6c01c460 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.288+0000 7f4a737fe640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a5c06a050 0x7f4a5c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:43.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.289+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 7617+0+0 (secure 0 0 0) 0x7f4a6c088c30 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.289+0000 7f4a7ad9d640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a5c06a050 0x7f4a5c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:43.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.289+0000 7f4a7ad9d640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a5c06a050 0x7f4a5c06c510 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4a64000f60 tx=0x7f4a64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:43.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.294+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4a6c053c60 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.523+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a7c069160 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.525+0000 7f4a737fe640 1 -- 172.21.15.71:0/3633910079 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f4a6c057280 con 0x7f4a7c0c3a40 2023-08-25T09:56:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:43.527 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:37.897227+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:43.527 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 52 2023-08-25T09:56:43.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a5c06a050 msgr2=0x7f4a5c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a5c06a050 0x7f4a5c06c510 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4a64000f60 tx=0x7f4a64009040 comp rx=0 tx=0).stop 2023-08-25T09:56:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 msgr2=0x7f4a7c14bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 0x7f4a7c14bf80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00c400 tx=0x7f4a6c00c8c0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 shutdown_connections 2023-08-25T09:56:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a5c06a050 0x7f4a5c06c510 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4a7c1510e0 0x7f4a7c14c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4a7c0c3a40 0x7f4a7c14bf80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 --2- 172.21.15.71:0/3633910079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4a7c0c3660 0x7f4a7c1e5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:43.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 >> 172.21.15.71:0/3633910079 conn(0x7f4a7c0654f0 msgr2=0x7f4a7c0c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:43.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 shutdown_connections 2023-08-25T09:56:43.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:43.528+0000 7f4a81245640 1 -- 172.21.15.71:0/3633910079 wait complete. 2023-08-25T09:56:43.547 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:56:45.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:45.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.778+0000 7f1b4e762640 1 Processor -- start 2023-08-25T09:56:45.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.778+0000 7f1b4e762640 1 -- start start 2023-08-25T09:56:45.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4e762640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3a40 0x7f1b480c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:45.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4e762640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1b481509d0 0x7f1b48150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:45.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4e762640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b480c1dc0 con 0x7f1b480c3660 2023-08-25T09:56:45.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4e762640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b48157a80 con 0x7f1b481509d0 2023-08-25T09:56:45.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4e762640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b481513a0 con 0x7f1b480c3a40 2023-08-25T09:56:45.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4ccff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1b481509d0 0x7f1b48150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4ccff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1b481509d0 0x7f1b48150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45364/0 (socket says 172.21.15.71:45364) 2023-08-25T09:56:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b47fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1b480c3660 0x7f1b4806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58082/0 (socket says 172.21.15.71:58082) 2023-08-25T09:56:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4ccff640 1 -- 172.21.15.71:0/1451694329 learned_addr learned my addr 172.21.15.71:0/1451694329 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b3f7fe640 1 --2- 172.21.15.71:0/1451694329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3a40 0x7f1b480c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:45.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.780+0000 7f1b4ccff640 1 -- 172.21.15.71:0/1451694329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3a40 msgr2=0x7f1b480c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:45.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.781+0000 7f1b4ccff640 1 --2- 172.21.15.71:0/1451694329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3a40 0x7f1b480c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:45.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.781+0000 7f1b4ccff640 1 -- 172.21.15.71:0/1451694329 >> v1:172.21.15.71:6789/0 conn(0x7f1b480c3660 legacy=0x7f1b4806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:45.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.781+0000 7f1b4ccff640 1 -- 172.21.15.71:0/1451694329 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b48156e00 con 0x7f1b481509d0 2023-08-25T09:56:45.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.782+0000 7f1b4ccff640 1 --2- 172.21.15.71:0/1451694329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b481509d0 0x7f1b48150e60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1b40009870 tx=0x7f1b4002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33857a6e327346c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:45.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.782+0000 7f1b477fe640 1 -- 172.21.15.71:0/1451694329 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4002f830 con 0x7f1b481509d0 2023-08-25T09:56:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.782+0000 7f1b477fe640 1 -- 172.21.15.71:0/1451694329 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b4002f990 con 0x7f1b481509d0 2023-08-25T09:56:45.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.782+0000 7f1b477fe640 1 -- 172.21.15.71:0/1451694329 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4002fd20 con 0x7f1b481509d0 2023-08-25T09:56:45.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.782+0000 7f1b4e762640 1 -- 172.21.15.71:0/1451694329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b481509d0 msgr2=0x7f1b48150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.782+0000 7f1b4e762640 1 --2- 172.21.15.71:0/1451694329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b481509d0 0x7f1b48150e60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1b40009870 tx=0x7f1b4002ba10 comp rx=0 tx=0).stop 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.783+0000 7f1b4e762640 1 -- 172.21.15.71:0/1451694329 shutdown_connections 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.783+0000 7f1b4e762640 1 --2- 172.21.15.71:0/1451694329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b481509d0 0x7f1b48150e60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.783+0000 7f1b4e762640 1 --2- 172.21.15.71:0/1451694329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3a40 0x7f1b480c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.783+0000 7f1b4e762640 1 -- 172.21.15.71:0/1451694329 >> 172.21.15.71:0/1451694329 conn(0x7f1b480654f0 msgr2=0x7f1b48065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.783+0000 7f1b4e762640 1 -- 172.21.15.71:0/1451694329 shutdown_connections 2023-08-25T09:56:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.783+0000 7f1b4e762640 1 -- 172.21.15.71:0/1451694329 wait complete. 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.784+0000 7f1b4e762640 1 Processor -- start 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.784+0000 7f1b4e762640 1 -- start start 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.785+0000 7f1b4e762640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3660 0x7f1b48148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.785+0000 7f1b4e762640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 0x7f1b48155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.785+0000 7f1b4e762640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b481509d0 0x7f1b48156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.785+0000 7f1b4e762640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4806f6e0 con 0x7f1b480c3a40 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.785+0000 7f1b4e762640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4814a580 con 0x7f1b481509d0 2023-08-25T09:56:45.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.785+0000 7f1b4e762640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4814a990 con 0x7f1b480c3660 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 0x7f1b48155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 0x7f1b48155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45380/0 (socket says 172.21.15.71:45380) 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 -- 172.21.15.71:0/4087757279 learned_addr learned my addr 172.21.15.71:0/4087757279 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b4ccff640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b481509d0 0x7f1b48156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b47fff640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3660 0x7f1b48148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 -- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3660 msgr2=0x7f1b48148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:45.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3660 0x7f1b48148510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 -- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b481509d0 msgr2=0x7f1b48156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b481509d0 0x7f1b48156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 -- 172.21.15.71:0/4087757279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400094d0 con 0x7f1b480c3a40 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b4ccff640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b481509d0 0x7f1b48156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.786+0000 7f1b3f7fe640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 0x7f1b48155f50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007e40 tx=0x7f1b3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.787+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30014620 con 0x7f1b480c3a40 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.787+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3000ee90 con 0x7f1b480c3a40 2023-08-25T09:56:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.787+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b300149c0 con 0x7f1b480c3a40 2023-08-25T09:56:45.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.787+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b48070080 con 0x7f1b480c3a40 2023-08-25T09:56:45.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.787+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4806f400 con 0x7f1b480c3a40 2023-08-25T09:56:45.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.788+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b480c29c0 con 0x7f1b480c3a40 2023-08-25T09:56:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.789+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1b30014b20 con 0x7f1b480c3a40 2023-08-25T09:56:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.790+0000 7f1b457fa640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b2806a050 0x7f1b2806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.790+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7217+0+0 (secure 0 0 0) 0x7f1b30089410 con 0x7f1b480c3a40 2023-08-25T09:56:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.790+0000 7f1b47fff640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b2806a050 0x7f1b2806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.791+0000 7f1b47fff640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b2806a050 0x7f1b2806c510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1b38007fd0 tx=0x7f1b38009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:45.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:45.795+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1b300545d0 con 0x7f1b480c3a40 2023-08-25T09:56:46.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.026+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b480c9760 con 0x7f1b480c3a40 2023-08-25T09:56:46.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.027+0000 7f1b457fa640 1 -- 172.21.15.71:0/4087757279 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f1b30057bf0 con 0x7f1b480c3a40 2023-08-25T09:56:46.029 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:46.029 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:45.895430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:46.029 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 54 2023-08-25T09:56:46.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.032+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b2806a050 msgr2=0x7f1b2806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:46.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.032+0000 7f1b4e762640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b2806a050 0x7f1b2806c510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1b38007fd0 tx=0x7f1b38009040 comp rx=0 tx=0).stop 2023-08-25T09:56:46.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.032+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 msgr2=0x7f1b48155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:46.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.032+0000 7f1b4e762640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 0x7f1b48155f50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007e40 tx=0x7f1b3000e4b0 comp rx=0 tx=0).stop 2023-08-25T09:56:46.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.035+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 shutdown_connections 2023-08-25T09:56:46.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.036+0000 7f1b4e762640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b481509d0 0x7f1b48156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:46.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.036+0000 7f1b4e762640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1b2806a050 0x7f1b2806c510 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:46.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.036+0000 7f1b4e762640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1b480c3a40 0x7f1b48155f50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:46.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.036+0000 7f1b4e762640 1 --2- 172.21.15.71:0/4087757279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b480c3660 0x7f1b48148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:46.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.036+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 >> 172.21.15.71:0/4087757279 conn(0x7f1b480654f0 msgr2=0x7f1b480c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:46.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.037+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 shutdown_connections 2023-08-25T09:56:46.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:46.037+0000 7f1b4e762640 1 -- 172.21.15.71:0/4087757279 wait complete. 2023-08-25T09:56:46.057 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=3, replacing gid=4287 2023-08-25T09:56:46.058 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2023-08-25T09:56:54.060 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2023-08-25T09:56:54.061 INFO:tasks.ceph.mds.a:Restarting daemon 2023-08-25T09:56:54.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-08-25T09:56:54.073 INFO:tasks.ceph.mds.a:Started 2023-08-25T09:56:54.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:54.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:56:54.202 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:56:54.425 INFO:tasks.ceph.mds.a.smithi071.stdout:starting mds.a at 2023-08-25T09:56:54.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.449+0000 7f3eafc1f640 1 Processor -- start 2023-08-25T09:56:54.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.449+0000 7f3eafc1f640 1 -- start start 2023-08-25T09:56:54.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.450+0000 7f3eafc1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3ea806cc70 0x7f3ea80c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:54.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.450+0000 7f3eafc1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea806a960 con 0x7f3ea81594d0 2023-08-25T09:56:54.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.450+0000 7f3eafc1f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea806ac20 con 0x7f3ea80c3940 2023-08-25T09:56:54.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.450+0000 7f3eafc1f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea806af00 con 0x7f3ea806cc70 2023-08-25T09:56:54.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3ead9bb640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3ea81594d0 0x7f3ea814f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37518/0 (socket says 172.21.15.71:37518) 2023-08-25T09:56:54.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3ead9bb640 1 -- 172.21.15.71:0/4270656521 learned_addr learned my addr 172.21.15.71:0/4270656521 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:56:54.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3ead1ba640 1 --2- 172.21.15.71:0/4270656521 >> v2:172.21.15.71:3300/0 conn(0x7f3ea806cc70 0x7f3ea80c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:54.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3eac9b9640 1 -- 172.21.15.71:0/4270656521 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827903492 0 0) 0x7f3ea806a960 con 0x7f3ea81594d0 2023-08-25T09:56:54.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3eac9b9640 1 -- 172.21.15.71:0/4270656521 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e94005730 con 0x7f3ea81594d0 2023-08-25T09:56:54.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3ead1ba640 1 -- 172.21.15.71:0/4270656521 >> v1:172.21.15.164:6789/0 conn(0x7f3ea80c3940 legacy=0x7f3ea81598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3ead1ba640 1 -- 172.21.15.71:0/4270656521 >> v1:172.21.15.71:6789/0 conn(0x7f3ea81594d0 legacy=0x7f3ea814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.451+0000 7f3ead1ba640 1 -- 172.21.15.71:0/4270656521 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80c2e60 con 0x7f3ea806cc70 2023-08-25T09:56:54.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.452+0000 7f3ead1ba640 1 --2- 172.21.15.71:0/4270656521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea806cc70 0x7f3ea80c4c90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3e980097a0 tx=0x7f3e9802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb26afaa25533ec server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:54.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eac9b9640 1 -- 172.21.15.71:0/4270656521 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98030610 con 0x7f3ea806cc70 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eac9b9640 1 -- 172.21.15.71:0/4270656521 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98030770 con 0x7f3ea806cc70 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eac9b9640 1 -- 172.21.15.71:0/4270656521 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98030b00 con 0x7f3ea806cc70 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eafc1f640 1 -- 172.21.15.71:0/4270656521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea806cc70 msgr2=0x7f3ea80c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/4270656521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea806cc70 0x7f3ea80c4c90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3e980097a0 tx=0x7f3e9802b960 comp rx=0 tx=0).stop 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eafc1f640 1 -- 172.21.15.71:0/4270656521 shutdown_connections 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/4270656521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea806cc70 0x7f3ea80c4c90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.453+0000 7f3eafc1f640 1 -- 172.21.15.71:0/4270656521 >> 172.21.15.71:0/4270656521 conn(0x7f3ea80654f0 msgr2=0x7f3ea8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:54.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.454+0000 7f3eafc1f640 1 -- 172.21.15.71:0/4270656521 shutdown_connections 2023-08-25T09:56:54.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.454+0000 7f3eafc1f640 1 -- 172.21.15.71:0/4270656521 wait complete. 2023-08-25T09:56:54.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.455+0000 7f3eafc1f640 1 Processor -- start 2023-08-25T09:56:54.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.456+0000 7f3eafc1f640 1 -- start start 2023-08-25T09:56:54.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.456+0000 7f3eafc1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ea806cc70 0x7f3ea814b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:54.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eafc1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea80c3940 0x7f3ea8158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:54.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eafc1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 0x7f3ea8158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:54.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eafc1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea806fc10 con 0x7f3ea81594d0 2023-08-25T09:56:54.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eafc1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8157cc0 con 0x7f3ea80c3940 2023-08-25T09:56:54.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eafc1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8156e30 con 0x7f3ea806cc70 2023-08-25T09:56:54.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 0x7f3ea8158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:54.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 0x7f3ea8158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48750/0 (socket says 172.21.15.71:48750) 2023-08-25T09:56:54.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 -- 172.21.15.71:0/3150918617 learned_addr learned my addr 172.21.15.71:0/3150918617 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:54.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3ead9bb640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ea806cc70 0x7f3ea814b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3ead1ba640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea80c3940 0x7f3ea8158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 -- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ea806cc70 msgr2=0x7f3ea814b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ea806cc70 0x7f3ea814b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 -- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea80c3940 msgr2=0x7f3ea8158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea80c3940 0x7f3ea8158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.457+0000 7f3eae1bc640 1 -- 172.21.15.71:0/3150918617 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e98009400 con 0x7f3ea81594d0 2023-08-25T09:56:54.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.458+0000 7f3eae1bc640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 0x7f3ea8158690 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000c6b0 tx=0x7f3ea000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:54.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.458+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0003b70 con 0x7f3ea81594d0 2023-08-25T09:56:54.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.458+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0003cf0 con 0x7f3ea81594d0 2023-08-25T09:56:54.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.458+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea806f0b0 con 0x7f3ea81594d0 2023-08-25T09:56:54.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.458+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0003ea0 con 0x7f3ea81594d0 2023-08-25T09:56:54.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.458+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea81570f0 con 0x7f3ea81594d0 2023-08-25T09:56:54.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.459+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea8157e30 con 0x7f3ea81594d0 2023-08-25T09:56:54.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.460+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3ea0005310 con 0x7f3ea81594d0 2023-08-25T09:56:54.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.461+0000 7f3e9e7fc640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e8c06a050 0x7f3e8c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:54.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.461+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 9257+0+0 (secure 0 0 0) 0x7f3ea0089010 con 0x7f3ea81594d0 2023-08-25T09:56:54.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.461+0000 7f3ead9bb640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e8c06a050 0x7f3e8c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:54.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.462+0000 7f3ead9bb640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e8c06a050 0x7f3e8c06c510 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40011e0 tx=0x7f3ea4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:54.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.466+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3ea0053a30 con 0x7f3ea81594d0 2023-08-25T09:56:54.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.762+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ea8069160 con 0x7f3ea81594d0 2023-08-25T09:56:54.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.762+0000 7f3e9e7fc640 1 -- 172.21.15.71:0/3150918617 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f3ea0057050 con 0x7f3ea81594d0 2023-08-25T09:56:54.764 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:54.765 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:52.666963+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:54.765 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 55 2023-08-25T09:56:54.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.764+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e8c06a050 msgr2=0x7f3e8c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.764+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e8c06a050 0x7f3e8c06c510 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40011e0 tx=0x7f3ea4009040 comp rx=0 tx=0).stop 2023-08-25T09:56:54.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.764+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 msgr2=0x7f3ea8158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:54.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.764+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 0x7f3ea8158690 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000c6b0 tx=0x7f3ea000cb70 comp rx=0 tx=0).stop 2023-08-25T09:56:54.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 shutdown_connections 2023-08-25T09:56:54.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ea80c3940 0x7f3ea8158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e8c06a050 0x7f3e8c06c510 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ea81594d0 0x7f3ea8158690 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 --2- 172.21.15.71:0/3150918617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ea806cc70 0x7f3ea814b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:54.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 >> 172.21.15.71:0/3150918617 conn(0x7f3ea80654f0 msgr2=0x7f3ea80658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:54.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 shutdown_connections 2023-08-25T09:56:54.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:54.765+0000 7f3eafc1f640 1 -- 172.21.15.71:0/3150918617 wait complete. 2023-08-25T09:56:54.782 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-08-25T09:56:56.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:56:56.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.977+0000 7f39ee5dd640 1 Processor -- start 2023-08-25T09:56:56.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.977+0000 7f39ee5dd640 1 -- start start 2023-08-25T09:56:56.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.978+0000 7f39ee5dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e80cb570 0x7f39e80ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:56.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.978+0000 7f39ee5dd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39e815cd00 0x7f39e8158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:56.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.978+0000 7f39ee5dd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e80b1af0 con 0x7f39e80cb190 2023-08-25T09:56:56.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39ee5dd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80c98f0 con 0x7f39e815cd00 2023-08-25T09:56:56.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39ee5dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e815d190 con 0x7f39e80cb570 2023-08-25T09:56:56.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39e815cd00 0x7f39e8158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:56.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f39e815cd00 0x7f39e8158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48762/0 (socket says 172.21.15.71:48762) 2023-08-25T09:56:56.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 -- 172.21.15.71:0/1036517639 learned_addr learned my addr 172.21.15.71:0/1036517639 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:56.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39ecdda640 1 --2- 172.21.15.71:0/1036517639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e80cb570 0x7f39e80ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:56.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 -- 172.21.15.71:0/1036517639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e80cb570 msgr2=0x7f39e80ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:56:56.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 --2- 172.21.15.71:0/1036517639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e80cb570 0x7f39e80ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:56.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 -- 172.21.15.71:0/1036517639 >> v1:172.21.15.164:6789/0 conn(0x7f39e80cb190 legacy=0x7f39e80c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:56.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.979+0000 7f39edddc640 1 -- 172.21.15.71:0/1036517639 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80b0fb0 con 0x7f39e815cd00 2023-08-25T09:56:56.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39edddc640 1 --2- 172.21.15.71:0/1036517639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e815cd00 0x7f39e8158d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009e50 tx=0x7f39e002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f22a457ba1ceb07 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:56.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39e7fff640 1 -- 172.21.15.71:0/1036517639 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e002cb90 con 0x7f39e815cd00 2023-08-25T09:56:56.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39e7fff640 1 -- 172.21.15.71:0/1036517639 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e002ccf0 con 0x7f39e815cd00 2023-08-25T09:56:56.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39e7fff640 1 -- 172.21.15.71:0/1036517639 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0034610 con 0x7f39e815cd00 2023-08-25T09:56:56.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1036517639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e815cd00 msgr2=0x7f39e8158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:56.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1036517639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e815cd00 0x7f39e8158d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009e50 tx=0x7f39e002bff0 comp rx=0 tx=0).stop 2023-08-25T09:56:56.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1036517639 shutdown_connections 2023-08-25T09:56:56.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1036517639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e815cd00 0x7f39e8158d80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:56.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1036517639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e80cb570 0x7f39e80ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:56.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.981+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1036517639 >> 172.21.15.71:0/1036517639 conn(0x7f39e806ae20 msgr2=0x7f39e806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:56.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.982+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1036517639 shutdown_connections 2023-08-25T09:56:56.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.982+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1036517639 wait complete. 2023-08-25T09:56:56.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.983+0000 7f39ee5dd640 1 Processor -- start 2023-08-25T09:56:56.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.983+0000 7f39ee5dd640 1 -- start start 2023-08-25T09:56:56.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ee5dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39e80cb190 0x7f39e81eddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:56.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ee5dd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 0x7f39e814bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:56.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ee5dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e815cd00 0x7f39e8154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:56.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ee5dd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81ee300 con 0x7f39e80cb570 2023-08-25T09:56:56.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ee5dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e81eed80 con 0x7f39e80cb190 2023-08-25T09:56:56.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ee5dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8150f00 con 0x7f39e815cd00 2023-08-25T09:56:56.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ecdda640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 0x7f39e814bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:56.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ecdda640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 0x7f39e814bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48770/0 (socket says 172.21.15.71:48770) 2023-08-25T09:56:56.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.984+0000 7f39ecdda640 1 -- 172.21.15.71:0/1764855359 learned_addr learned my addr 172.21.15.71:0/1764855359 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:56:56.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ed5db640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39e80cb190 0x7f39e81eddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:56.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ecdda640 1 -- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e815cd00 msgr2=0x7f39e8154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:56.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ecdda640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e815cd00 0x7f39e8154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:56.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ecdda640 1 -- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39e80cb190 msgr2=0x7f39e81eddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:56.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ecdda640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39e80cb190 0x7f39e81eddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:56.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ecdda640 1 -- 172.21.15.71:0/1764855359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009b30 con 0x7f39e80cb570 2023-08-25T09:56:56.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ed5db640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39e80cb190 0x7f39e81eddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:56:56.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ecdda640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 0x7f39e814bb90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00d800 tx=0x7f39dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:56.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc00b660 con 0x7f39e80cb570 2023-08-25T09:56:56.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc00b7c0 con 0x7f39e80cb570 2023-08-25T09:56:56.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc00bb50 con 0x7f39e80cb570 2023-08-25T09:56:56.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e81ee4e0 con 0x7f39e80cb570 2023-08-25T09:56:56.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.985+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e81eeaa0 con 0x7f39e80cb570 2023-08-25T09:56:56.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.987+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e80c9c40 con 0x7f39e80cb570 2023-08-25T09:56:56.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.987+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f39dc00bcb0 con 0x7f39e80cb570 2023-08-25T09:56:56.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.988+0000 7f39e5ffb640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f39d406a050 0x7f39d406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:56:57.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.988+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 8841+0+0 (secure 0 0 0) 0x7f39dc0896a0 con 0x7f39e80cb570 2023-08-25T09:56:57.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.988+0000 7f39ed5db640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f39d406a050 0x7f39d406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:56:57.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.989+0000 7f39ed5db640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f39d406a050 0x7f39d406c510 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f39d8000f60 tx=0x7f39d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:56:57.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:56.993+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f39dc054170 con 0x7f39e80cb570 2023-08-25T09:56:57.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.199+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39e80b1530 con 0x7f39e80cb570 2023-08-25T09:56:57.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.200+0000 7f39e5ffb640 1 -- 172.21.15.71:0/1764855359 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6534 (secure 0 0 0) 0x7f39dc057790 con 0x7f39e80cb570 2023-08-25T09:56:57.202 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:56:57.202 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:56:52.666963+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:56:57.203 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 56 2023-08-25T09:56:57.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f39d406a050 msgr2=0x7f39d406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:57.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f39d406a050 0x7f39d406c510 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f39d8000f60 tx=0x7f39d8009040 comp rx=0 tx=0).stop 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 msgr2=0x7f39e814bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 0x7f39e814bb90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00d800 tx=0x7f39dc00dcc0 comp rx=0 tx=0).stop 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 shutdown_connections 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f39d406a050 0x7f39d406c510 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39e815cd00 0x7f39e8154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f39e80cb570 0x7f39e814bb90 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:57.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 --2- 172.21.15.71:0/1764855359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39e80cb190 0x7f39e81eddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:56:57.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 >> 172.21.15.71:0/1764855359 conn(0x7f39e806ae20 msgr2=0x7f39e80c7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:56:57.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 shutdown_connections 2023-08-25T09:56:57.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:56:57.203+0000 7f39ee5dd640 1 -- 172.21.15.71:0/1764855359 wait complete. 2023-08-25T09:56:57.222 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2023-08-25T09:56:57.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2023-08-25T09:57:00.204 INFO:tasks.workunit.client.0.smithi071.stdout:Delete all the files and directories ... 2023-08-25T09:57:03.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/19' 2023-08-25T09:57:03.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/15' 2023-08-25T09:57:03.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/31' 2023-08-25T09:57:03.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/36' 2023-08-25T09:57:03.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/20' 2023-08-25T09:57:03.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/12' 2023-08-25T09:57:03.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/39' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/10' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/34' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/38' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/11' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/16' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/32' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/30' 2023-08-25T09:57:03.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/14' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/18' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/13' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/33' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/17' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/35' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './100/37' 2023-08-25T09:57:03.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './100' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/19' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/15' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/31' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/36' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/20' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/12' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/39' 2023-08-25T09:57:03.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/10' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/34' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/38' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/11' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/16' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/32' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/30' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/14' 2023-08-25T09:57:03.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/18' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/13' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/33' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/17' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/35' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './101/37' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './101' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/19' 2023-08-25T09:57:03.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/15' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/31' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/36' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/20' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/12' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/39' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/10' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/34' 2023-08-25T09:57:03.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/38' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/11' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/16' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/32' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/30' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/14' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/18' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/13' 2023-08-25T09:57:03.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/33' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/17' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/35' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './102/37' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './102' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/19' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/15' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/31' 2023-08-25T09:57:03.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/36' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/20' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/12' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/39' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/10' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/34' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/38' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/11' 2023-08-25T09:57:03.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/16' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/32' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/30' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/14' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/18' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/13' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/33' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/17' 2023-08-25T09:57:03.102 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/35' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed './103/37' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './103' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/19' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/15' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/31' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/36' 2023-08-25T09:57:03.103 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/20' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/12' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/39' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/10' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/34' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/38' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/11' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/16' 2023-08-25T09:57:03.104 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/32' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/30' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/14' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/18' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/13' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/33' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/17' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/35' 2023-08-25T09:57:03.105 INFO:tasks.workunit.client.0.smithi071.stdout:removed './104/37' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './104' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/19' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/15' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/31' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/36' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/20' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/12' 2023-08-25T09:57:03.106 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/39' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/10' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/34' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/38' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/11' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/16' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/32' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/30' 2023-08-25T09:57:03.107 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/14' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/18' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/13' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/33' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/17' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/35' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './105/37' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './105' 2023-08-25T09:57:03.108 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/19' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/15' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/31' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/36' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/20' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/12' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/39' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/10' 2023-08-25T09:57:03.109 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/34' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/38' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/11' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/16' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/32' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/30' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/14' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/18' 2023-08-25T09:57:03.110 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/13' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/33' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/17' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/35' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './106/37' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './106' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/19' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/15' 2023-08-25T09:57:03.111 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/31' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/36' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/20' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/12' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/39' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/10' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/34' 2023-08-25T09:57:03.112 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/38' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/11' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/16' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/32' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/30' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/14' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/18' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/13' 2023-08-25T09:57:03.113 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/33' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/17' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/35' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './107/37' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './107' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/19' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/15' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/31' 2023-08-25T09:57:03.114 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/36' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/20' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/12' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/39' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/10' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/34' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/38' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/11' 2023-08-25T09:57:03.115 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/16' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/32' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/30' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/14' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/18' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/13' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/33' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/17' 2023-08-25T09:57:03.116 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/35' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './108/37' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './108' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/19' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/15' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/31' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/36' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/20' 2023-08-25T09:57:03.117 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/12' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/39' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/10' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/34' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/38' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/11' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/16' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/32' 2023-08-25T09:57:03.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/30' 2023-08-25T09:57:09.476 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/14' 2023-08-25T09:57:09.476 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/18' 2023-08-25T09:57:09.476 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/13' 2023-08-25T09:57:09.476 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/33' 2023-08-25T09:57:09.476 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/17' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/35' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed './109/37' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './109' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/19' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/15' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/31' 2023-08-25T09:57:09.477 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/36' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/20' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/12' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/39' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/10' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/34' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/38' 2023-08-25T09:57:09.478 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/11' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/16' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/32' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/30' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/14' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/18' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/13' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/33' 2023-08-25T09:57:09.479 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/17' 2023-08-25T09:57:09.480 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/35' 2023-08-25T09:57:09.480 INFO:tasks.workunit.client.0.smithi071.stdout:removed './110/37' 2023-08-25T09:57:09.480 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './110' 2023-08-25T09:57:09.480 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/19' 2023-08-25T09:57:09.480 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/15' 2023-08-25T09:57:09.480 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/31' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/36' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/20' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/12' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/39' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/10' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/34' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/38' 2023-08-25T09:57:09.481 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/11' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/16' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/32' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/30' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/14' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/18' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/13' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/33' 2023-08-25T09:57:09.482 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/17' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/35' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed './111/37' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './111' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/19' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/15' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/31' 2023-08-25T09:57:09.483 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/36' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/20' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/12' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/39' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/10' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/34' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/38' 2023-08-25T09:57:09.484 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/11' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/16' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/32' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/30' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/14' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/18' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/13' 2023-08-25T09:57:09.485 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/33' 2023-08-25T09:57:09.486 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/17' 2023-08-25T09:57:09.486 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/35' 2023-08-25T09:57:09.486 INFO:tasks.workunit.client.0.smithi071.stdout:removed './112/37' 2023-08-25T09:57:09.486 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './112' 2023-08-25T09:57:09.486 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/19' 2023-08-25T09:57:09.486 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/15' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/31' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/36' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/20' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/12' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/39' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/10' 2023-08-25T09:57:09.487 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/34' 2023-08-25T09:57:09.488 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/38' 2023-08-25T09:57:09.488 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/11' 2023-08-25T09:57:09.488 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/16' 2023-08-25T09:57:09.488 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/32' 2023-08-25T09:57:09.488 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/30' 2023-08-25T09:57:09.488 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/14' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/18' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/13' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/33' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/17' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/35' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed './113/37' 2023-08-25T09:57:09.489 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './113' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/19' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/15' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/31' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/36' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/20' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/12' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/39' 2023-08-25T09:57:09.490 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/10' 2023-08-25T09:57:09.491 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/34' 2023-08-25T09:57:09.491 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/38' 2023-08-25T09:57:09.491 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/11' 2023-08-25T09:57:09.491 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/16' 2023-08-25T09:57:09.491 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/32' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/30' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/14' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/18' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/13' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/33' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/17' 2023-08-25T09:57:09.492 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/35' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './114/37' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './114' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/19' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/15' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/31' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/36' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/20' 2023-08-25T09:57:09.493 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/12' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/39' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/10' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/34' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/38' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/11' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/16' 2023-08-25T09:57:09.494 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/32' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/30' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/14' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/18' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/13' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/33' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/17' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/35' 2023-08-25T09:57:09.495 INFO:tasks.workunit.client.0.smithi071.stdout:removed './115/37' 2023-08-25T09:57:09.496 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './115' 2023-08-25T09:57:09.496 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/19' 2023-08-25T09:57:09.496 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/15' 2023-08-25T09:57:09.496 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/31' 2023-08-25T09:57:09.496 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/36' 2023-08-25T09:57:09.497 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/20' 2023-08-25T09:57:09.497 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/12' 2023-08-25T09:57:09.497 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/39' 2023-08-25T09:57:09.497 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/10' 2023-08-25T09:57:09.497 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/34' 2023-08-25T09:57:09.497 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/38' 2023-08-25T09:57:09.498 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/11' 2023-08-25T09:57:09.498 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/16' 2023-08-25T09:57:09.498 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/32' 2023-08-25T09:57:09.498 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/30' 2023-08-25T09:57:09.498 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/14' 2023-08-25T09:57:09.498 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/18' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/13' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/33' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/17' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/35' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed './116/37' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './116' 2023-08-25T09:57:09.499 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/19' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/15' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/31' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/36' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/20' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/12' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/39' 2023-08-25T09:57:09.500 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/10' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/34' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/38' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/11' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/16' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/32' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/30' 2023-08-25T09:57:09.501 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/14' 2023-08-25T09:57:09.502 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/18' 2023-08-25T09:57:09.502 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/13' 2023-08-25T09:57:09.502 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/33' 2023-08-25T09:57:09.502 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/17' 2023-08-25T09:57:09.502 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/35' 2023-08-25T09:57:09.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed './117/37' 2023-08-25T09:57:09.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './117' 2023-08-25T09:57:09.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/19' 2023-08-25T09:57:09.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/15' 2023-08-25T09:57:09.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/31' 2023-08-25T09:57:09.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/36' 2023-08-25T09:57:09.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/20' 2023-08-25T09:57:09.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/12' 2023-08-25T09:57:09.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/39' 2023-08-25T09:57:09.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/10' 2023-08-25T09:57:09.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/34' 2023-08-25T09:57:09.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/38' 2023-08-25T09:57:09.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/11' 2023-08-25T09:57:09.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/16' 2023-08-25T09:57:09.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/32' 2023-08-25T09:57:09.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/30' 2023-08-25T09:57:09.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/14' 2023-08-25T09:57:09.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/18' 2023-08-25T09:57:09.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/13' 2023-08-25T09:57:09.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/33' 2023-08-25T09:57:09.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/17' 2023-08-25T09:57:09.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/35' 2023-08-25T09:57:09.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './118/37' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './118' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/19' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/15' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/31' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/36' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/20' 2023-08-25T09:57:09.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/12' 2023-08-25T09:57:14.141 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/39' 2023-08-25T09:57:14.141 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/10' 2023-08-25T09:57:14.141 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/34' 2023-08-25T09:57:14.141 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/38' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/11' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/16' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/32' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/30' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/14' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/18' 2023-08-25T09:57:14.142 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/13' 2023-08-25T09:57:14.143 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/33' 2023-08-25T09:57:14.143 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/17' 2023-08-25T09:57:14.143 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/35' 2023-08-25T09:57:14.143 INFO:tasks.workunit.client.0.smithi071.stdout:removed './119/37' 2023-08-25T09:57:14.143 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './119' 2023-08-25T09:57:14.144 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/19' 2023-08-25T09:57:14.144 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/15' 2023-08-25T09:57:14.144 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/31' 2023-08-25T09:57:14.144 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/36' 2023-08-25T09:57:14.144 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/20' 2023-08-25T09:57:14.145 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/12' 2023-08-25T09:57:14.145 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/39' 2023-08-25T09:57:14.145 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/10' 2023-08-25T09:57:14.145 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/34' 2023-08-25T09:57:14.145 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/38' 2023-08-25T09:57:14.145 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/11' 2023-08-25T09:57:14.146 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/16' 2023-08-25T09:57:14.146 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/32' 2023-08-25T09:57:14.146 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/30' 2023-08-25T09:57:14.147 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/14' 2023-08-25T09:57:14.147 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/18' 2023-08-25T09:57:14.147 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/13' 2023-08-25T09:57:14.147 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/33' 2023-08-25T09:57:14.148 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/17' 2023-08-25T09:57:14.148 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/35' 2023-08-25T09:57:14.148 INFO:tasks.workunit.client.0.smithi071.stdout:removed './120/37' 2023-08-25T09:57:14.148 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './120' 2023-08-25T09:57:14.149 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/19' 2023-08-25T09:57:14.149 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/15' 2023-08-25T09:57:14.149 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/31' 2023-08-25T09:57:14.150 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/36' 2023-08-25T09:57:14.150 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/20' 2023-08-25T09:57:14.151 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/12' 2023-08-25T09:57:14.151 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/39' 2023-08-25T09:57:14.151 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/10' 2023-08-25T09:57:14.151 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/34' 2023-08-25T09:57:14.151 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/38' 2023-08-25T09:57:14.151 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/11' 2023-08-25T09:57:14.152 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/16' 2023-08-25T09:57:14.152 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/32' 2023-08-25T09:57:14.152 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/30' 2023-08-25T09:57:14.153 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/14' 2023-08-25T09:57:14.153 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/18' 2023-08-25T09:57:14.154 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/13' 2023-08-25T09:57:14.154 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/33' 2023-08-25T09:57:14.154 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/17' 2023-08-25T09:57:14.155 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/35' 2023-08-25T09:57:14.155 INFO:tasks.workunit.client.0.smithi071.stdout:removed './121/37' 2023-08-25T09:57:14.155 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './121' 2023-08-25T09:57:14.155 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/19' 2023-08-25T09:57:14.155 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/15' 2023-08-25T09:57:14.156 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/31' 2023-08-25T09:57:14.156 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/36' 2023-08-25T09:57:14.156 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/20' 2023-08-25T09:57:14.156 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/12' 2023-08-25T09:57:14.157 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/39' 2023-08-25T09:57:14.157 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/10' 2023-08-25T09:57:14.157 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/34' 2023-08-25T09:57:14.157 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/38' 2023-08-25T09:57:14.158 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/11' 2023-08-25T09:57:14.158 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/16' 2023-08-25T09:57:14.158 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/32' 2023-08-25T09:57:14.158 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/30' 2023-08-25T09:57:14.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/14' 2023-08-25T09:57:14.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/18' 2023-08-25T09:57:14.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/13' 2023-08-25T09:57:14.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/33' 2023-08-25T09:57:14.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/17' 2023-08-25T09:57:14.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/35' 2023-08-25T09:57:14.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './122/37' 2023-08-25T09:57:14.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './122' 2023-08-25T09:57:14.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/19' 2023-08-25T09:57:14.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/15' 2023-08-25T09:57:14.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/31' 2023-08-25T09:57:14.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/36' 2023-08-25T09:57:14.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/20' 2023-08-25T09:57:14.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/12' 2023-08-25T09:57:14.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/39' 2023-08-25T09:57:14.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/10' 2023-08-25T09:57:14.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/34' 2023-08-25T09:57:14.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/38' 2023-08-25T09:57:14.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/11' 2023-08-25T09:57:14.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/16' 2023-08-25T09:57:14.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/32' 2023-08-25T09:57:14.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/30' 2023-08-25T09:57:14.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/14' 2023-08-25T09:57:14.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/18' 2023-08-25T09:57:14.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/13' 2023-08-25T09:57:14.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/33' 2023-08-25T09:57:14.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/17' 2023-08-25T09:57:14.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/35' 2023-08-25T09:57:14.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './123/37' 2023-08-25T09:57:14.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './123' 2023-08-25T09:57:14.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/19' 2023-08-25T09:57:14.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/15' 2023-08-25T09:57:14.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/31' 2023-08-25T09:57:14.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/36' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/20' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/12' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/39' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/10' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/34' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/38' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/11' 2023-08-25T09:57:14.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/16' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/32' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/30' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/14' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/18' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/13' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/33' 2023-08-25T09:57:14.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/17' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/35' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './124/37' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './124' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/19' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/15' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/31' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/36' 2023-08-25T09:57:14.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/20' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/12' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/39' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/10' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/34' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/38' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/11' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/16' 2023-08-25T09:57:14.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/32' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/30' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/14' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/18' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/13' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/33' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/17' 2023-08-25T09:57:14.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/35' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './125/37' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './125' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/19' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/15' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/31' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/36' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/20' 2023-08-25T09:57:14.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/12' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/39' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/10' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/34' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/38' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/11' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/16' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/32' 2023-08-25T09:57:14.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/30' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/14' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/18' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/13' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/33' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/17' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/35' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed './126/37' 2023-08-25T09:57:14.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './126' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/19' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/15' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/31' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/36' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/20' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/12' 2023-08-25T09:57:14.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/39' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/10' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/34' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/38' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/11' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/16' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/32' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/30' 2023-08-25T09:57:14.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/14' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/18' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/13' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/33' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/17' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/35' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './127/37' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './127' 2023-08-25T09:57:14.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/19' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/15' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/31' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/36' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/20' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/12' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/39' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/10' 2023-08-25T09:57:14.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/34' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/38' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/11' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/16' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/32' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/30' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/14' 2023-08-25T09:57:14.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/18' 2023-08-25T09:57:14.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/13' 2023-08-25T09:57:14.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/33' 2023-08-25T09:57:14.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/17' 2023-08-25T09:57:14.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/35' 2023-08-25T09:57:16.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed './128/37' 2023-08-25T09:57:16.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './128' 2023-08-25T09:57:16.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/19' 2023-08-25T09:57:16.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/15' 2023-08-25T09:57:16.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/31' 2023-08-25T09:57:16.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/36' 2023-08-25T09:57:16.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/20' 2023-08-25T09:57:16.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/12' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/39' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/10' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/34' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/38' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/11' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/16' 2023-08-25T09:57:16.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/32' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/30' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/14' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/18' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/13' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/33' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/17' 2023-08-25T09:57:16.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/35' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './129/37' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './129' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/19' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/15' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/31' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/36' 2023-08-25T09:57:16.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/20' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/12' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/39' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/10' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/34' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/38' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/11' 2023-08-25T09:57:16.354 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/16' 2023-08-25T09:57:16.355 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/32' 2023-08-25T09:57:16.355 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/30' 2023-08-25T09:57:16.355 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/14' 2023-08-25T09:57:16.355 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/18' 2023-08-25T09:57:16.355 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/13' 2023-08-25T09:57:16.355 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/33' 2023-08-25T09:57:16.356 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/17' 2023-08-25T09:57:16.356 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/35' 2023-08-25T09:57:16.356 INFO:tasks.workunit.client.0.smithi071.stdout:removed './130/37' 2023-08-25T09:57:16.356 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './130' 2023-08-25T09:57:16.356 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/19' 2023-08-25T09:57:16.357 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/15' 2023-08-25T09:57:16.357 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/31' 2023-08-25T09:57:16.357 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/36' 2023-08-25T09:57:16.357 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/20' 2023-08-25T09:57:16.357 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/12' 2023-08-25T09:57:16.358 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/39' 2023-08-25T09:57:16.358 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/10' 2023-08-25T09:57:16.358 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/34' 2023-08-25T09:57:16.358 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/38' 2023-08-25T09:57:16.358 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/11' 2023-08-25T09:57:16.358 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/16' 2023-08-25T09:57:16.359 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/32' 2023-08-25T09:57:16.359 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/30' 2023-08-25T09:57:16.359 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/14' 2023-08-25T09:57:16.359 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/18' 2023-08-25T09:57:16.359 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/13' 2023-08-25T09:57:16.360 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/33' 2023-08-25T09:57:16.360 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/17' 2023-08-25T09:57:16.360 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/35' 2023-08-25T09:57:16.360 INFO:tasks.workunit.client.0.smithi071.stdout:removed './131/37' 2023-08-25T09:57:16.360 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './131' 2023-08-25T09:57:16.361 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/19' 2023-08-25T09:57:16.361 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/15' 2023-08-25T09:57:16.361 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/31' 2023-08-25T09:57:16.361 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/36' 2023-08-25T09:57:16.361 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/20' 2023-08-25T09:57:16.362 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/12' 2023-08-25T09:57:16.362 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/39' 2023-08-25T09:57:16.362 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/10' 2023-08-25T09:57:16.362 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/34' 2023-08-25T09:57:16.362 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/38' 2023-08-25T09:57:16.362 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/11' 2023-08-25T09:57:16.363 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/16' 2023-08-25T09:57:16.363 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/32' 2023-08-25T09:57:16.363 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/30' 2023-08-25T09:57:16.363 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/14' 2023-08-25T09:57:16.364 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/18' 2023-08-25T09:57:16.364 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/13' 2023-08-25T09:57:16.364 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/33' 2023-08-25T09:57:16.364 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/17' 2023-08-25T09:57:16.364 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/35' 2023-08-25T09:57:16.365 INFO:tasks.workunit.client.0.smithi071.stdout:removed './132/37' 2023-08-25T09:57:16.365 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './132' 2023-08-25T09:57:16.365 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/19' 2023-08-25T09:57:16.365 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/15' 2023-08-25T09:57:16.365 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/31' 2023-08-25T09:57:16.365 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/36' 2023-08-25T09:57:16.366 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/20' 2023-08-25T09:57:16.366 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/12' 2023-08-25T09:57:16.366 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/39' 2023-08-25T09:57:16.366 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/10' 2023-08-25T09:57:16.366 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/34' 2023-08-25T09:57:16.366 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/38' 2023-08-25T09:57:16.367 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/11' 2023-08-25T09:57:16.367 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/16' 2023-08-25T09:57:16.367 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/32' 2023-08-25T09:57:16.367 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/30' 2023-08-25T09:57:16.367 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/14' 2023-08-25T09:57:16.368 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/18' 2023-08-25T09:57:16.368 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/13' 2023-08-25T09:57:16.368 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/33' 2023-08-25T09:57:16.368 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/17' 2023-08-25T09:57:16.368 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/35' 2023-08-25T09:57:16.369 INFO:tasks.workunit.client.0.smithi071.stdout:removed './133/37' 2023-08-25T09:57:16.369 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './133' 2023-08-25T09:57:16.369 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/19' 2023-08-25T09:57:16.369 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/15' 2023-08-25T09:57:16.369 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/31' 2023-08-25T09:57:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/36' 2023-08-25T09:57:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/20' 2023-08-25T09:57:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/12' 2023-08-25T09:57:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/39' 2023-08-25T09:57:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/10' 2023-08-25T09:57:16.370 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/34' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/38' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/11' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/16' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/32' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/30' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/14' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/18' 2023-08-25T09:57:16.371 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/13' 2023-08-25T09:57:16.372 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/33' 2023-08-25T09:57:16.372 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/17' 2023-08-25T09:57:16.372 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/35' 2023-08-25T09:57:16.372 INFO:tasks.workunit.client.0.smithi071.stdout:removed './134/37' 2023-08-25T09:57:16.372 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './134' 2023-08-25T09:57:16.372 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/19' 2023-08-25T09:57:16.373 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/15' 2023-08-25T09:57:16.373 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/31' 2023-08-25T09:57:16.373 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/36' 2023-08-25T09:57:16.373 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/20' 2023-08-25T09:57:16.373 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/12' 2023-08-25T09:57:16.374 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/39' 2023-08-25T09:57:16.374 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/10' 2023-08-25T09:57:16.374 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/34' 2023-08-25T09:57:16.374 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/38' 2023-08-25T09:57:16.374 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/11' 2023-08-25T09:57:16.375 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/16' 2023-08-25T09:57:16.375 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/32' 2023-08-25T09:57:16.375 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/30' 2023-08-25T09:57:16.375 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/14' 2023-08-25T09:57:16.375 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/18' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/13' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/33' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/17' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/35' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed './135/37' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './135' 2023-08-25T09:57:16.376 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/19' 2023-08-25T09:57:16.377 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/15' 2023-08-25T09:57:16.377 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/31' 2023-08-25T09:57:16.377 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/36' 2023-08-25T09:57:16.377 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/20' 2023-08-25T09:57:16.377 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/12' 2023-08-25T09:57:16.377 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/39' 2023-08-25T09:57:16.378 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/10' 2023-08-25T09:57:16.378 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/34' 2023-08-25T09:57:16.378 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/38' 2023-08-25T09:57:16.378 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/11' 2023-08-25T09:57:16.378 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/16' 2023-08-25T09:57:16.379 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/32' 2023-08-25T09:57:16.379 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/30' 2023-08-25T09:57:16.379 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/14' 2023-08-25T09:57:16.379 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/18' 2023-08-25T09:57:16.380 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/13' 2023-08-25T09:57:16.380 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/33' 2023-08-25T09:57:16.380 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/17' 2023-08-25T09:57:16.380 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/35' 2023-08-25T09:57:16.381 INFO:tasks.workunit.client.0.smithi071.stdout:removed './136/37' 2023-08-25T09:57:16.381 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './136' 2023-08-25T09:57:16.381 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/19' 2023-08-25T09:57:16.382 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/15' 2023-08-25T09:57:16.382 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/31' 2023-08-25T09:57:16.382 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/36' 2023-08-25T09:57:16.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/20' 2023-08-25T09:57:16.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/12' 2023-08-25T09:57:16.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/39' 2023-08-25T09:57:16.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/10' 2023-08-25T09:57:16.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/34' 2023-08-25T09:57:16.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/38' 2023-08-25T09:57:16.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/11' 2023-08-25T09:57:16.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/16' 2023-08-25T09:57:16.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/32' 2023-08-25T09:57:16.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/30' 2023-08-25T09:57:16.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/14' 2023-08-25T09:57:16.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/18' 2023-08-25T09:57:16.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/13' 2023-08-25T09:57:16.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/33' 2023-08-25T09:57:16.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/17' 2023-08-25T09:57:16.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/35' 2023-08-25T09:57:16.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed './137/37' 2023-08-25T09:57:16.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './137' 2023-08-25T09:57:16.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/19' 2023-08-25T09:57:16.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/15' 2023-08-25T09:57:16.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/31' 2023-08-25T09:57:16.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/36' 2023-08-25T09:57:16.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/20' 2023-08-25T09:57:16.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/12' 2023-08-25T09:57:16.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/39' 2023-08-25T09:57:16.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/10' 2023-08-25T09:57:16.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/34' 2023-08-25T09:57:16.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/38' 2023-08-25T09:57:16.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/11' 2023-08-25T09:57:16.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/16' 2023-08-25T09:57:17.994 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/32' 2023-08-25T09:57:17.994 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/30' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/14' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/18' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/13' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/33' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/17' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/35' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed './138/37' 2023-08-25T09:57:17.995 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './138' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/19' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/15' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/31' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/36' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/20' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/12' 2023-08-25T09:57:17.996 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/39' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/10' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/34' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/38' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/11' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/16' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/32' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/30' 2023-08-25T09:57:17.997 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/14' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/18' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/13' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/33' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/17' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/35' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed './139/37' 2023-08-25T09:57:17.998 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './139' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/19' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/15' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/31' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/36' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/20' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/12' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/39' 2023-08-25T09:57:17.999 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/10' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/34' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/38' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/11' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/16' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/32' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/30' 2023-08-25T09:57:18.000 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/14' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/18' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/13' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/33' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/17' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/35' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './140/37' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './140' 2023-08-25T09:57:18.001 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/19' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/15' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/31' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/36' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/20' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/12' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/39' 2023-08-25T09:57:18.002 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/10' 2023-08-25T09:57:18.003 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/34' 2023-08-25T09:57:18.003 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/38' 2023-08-25T09:57:18.003 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/11' 2023-08-25T09:57:18.003 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/16' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/32' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/30' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/14' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/18' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/13' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/33' 2023-08-25T09:57:18.004 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/17' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/35' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed './141/37' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './141' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/19' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/15' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/31' 2023-08-25T09:57:18.005 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/36' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/20' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/12' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/39' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/10' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/34' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/38' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/11' 2023-08-25T09:57:18.006 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/16' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/32' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/30' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/14' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/18' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/13' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/33' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/17' 2023-08-25T09:57:18.007 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/35' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed './142/37' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './142' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/19' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/15' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/31' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/36' 2023-08-25T09:57:18.008 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/20' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/12' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/39' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/10' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/34' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/38' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/11' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/16' 2023-08-25T09:57:18.009 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/32' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/30' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/14' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/18' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/13' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/33' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/17' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/35' 2023-08-25T09:57:18.010 INFO:tasks.workunit.client.0.smithi071.stdout:removed './143/37' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './143' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/19' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/15' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/31' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/36' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/20' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/12' 2023-08-25T09:57:18.011 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/39' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/10' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/34' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/38' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/11' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/16' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/32' 2023-08-25T09:57:18.012 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/30' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/14' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/18' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/13' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/33' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/17' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/35' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed './144/37' 2023-08-25T09:57:18.013 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './144' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/19' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/15' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/31' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/36' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/20' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/12' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/39' 2023-08-25T09:57:18.014 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/10' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/34' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/38' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/11' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/16' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/32' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/30' 2023-08-25T09:57:18.015 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/14' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/18' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/13' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/33' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/17' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/35' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed './145/37' 2023-08-25T09:57:18.016 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './145' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/19' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/15' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/31' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/36' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/20' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/12' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/39' 2023-08-25T09:57:18.017 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/10' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/34' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/38' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/11' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/16' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/32' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/30' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/14' 2023-08-25T09:57:18.018 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/18' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/13' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/33' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/17' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/35' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './146/37' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './146' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/19' 2023-08-25T09:57:18.019 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/15' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/31' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/36' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/20' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/12' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/39' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/10' 2023-08-25T09:57:18.020 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/34' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/38' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/11' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/16' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/32' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/30' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/14' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/18' 2023-08-25T09:57:18.021 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/13' 2023-08-25T09:57:18.022 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/33' 2023-08-25T09:57:18.022 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/17' 2023-08-25T09:57:18.022 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/35' 2023-08-25T09:57:18.022 INFO:tasks.workunit.client.0.smithi071.stdout:removed './147/37' 2023-08-25T09:57:18.022 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './147' 2023-08-25T09:57:18.022 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/19' 2023-08-25T09:57:18.023 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/15' 2023-08-25T09:57:18.023 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/31' 2023-08-25T09:57:18.023 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/36' 2023-08-25T09:57:19.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/20' 2023-08-25T09:57:19.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/12' 2023-08-25T09:57:19.503 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/39' 2023-08-25T09:57:19.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/10' 2023-08-25T09:57:19.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/34' 2023-08-25T09:57:19.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/38' 2023-08-25T09:57:19.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/11' 2023-08-25T09:57:19.504 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/16' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/32' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/30' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/14' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/18' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/13' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/33' 2023-08-25T09:57:19.505 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/17' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/35' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './148/37' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './148' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/19' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/15' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/31' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/36' 2023-08-25T09:57:19.506 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/20' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/12' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/39' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/10' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/34' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/38' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/11' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/16' 2023-08-25T09:57:19.507 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/32' 2023-08-25T09:57:19.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/30' 2023-08-25T09:57:19.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/14' 2023-08-25T09:57:19.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/18' 2023-08-25T09:57:19.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/13' 2023-08-25T09:57:19.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/33' 2023-08-25T09:57:19.508 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/17' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/35' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './149/37' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './149' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/19' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/15' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/31' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/36' 2023-08-25T09:57:19.509 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/20' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/12' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/39' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/10' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/34' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/38' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/11' 2023-08-25T09:57:19.510 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/16' 2023-08-25T09:57:19.511 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/32' 2023-08-25T09:57:19.511 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/30' 2023-08-25T09:57:19.511 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/14' 2023-08-25T09:57:19.511 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/18' 2023-08-25T09:57:19.511 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/13' 2023-08-25T09:57:19.512 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/33' 2023-08-25T09:57:19.512 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/17' 2023-08-25T09:57:19.512 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/35' 2023-08-25T09:57:19.512 INFO:tasks.workunit.client.0.smithi071.stdout:removed './150/37' 2023-08-25T09:57:19.512 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './150' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/19' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/15' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/31' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/36' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/20' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/12' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/39' 2023-08-25T09:57:19.513 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/10' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/34' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/38' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/11' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/16' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/32' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/30' 2023-08-25T09:57:19.514 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/14' 2023-08-25T09:57:19.515 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/18' 2023-08-25T09:57:19.515 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/13' 2023-08-25T09:57:19.515 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/33' 2023-08-25T09:57:19.515 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/17' 2023-08-25T09:57:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/35' 2023-08-25T09:57:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:removed './151/37' 2023-08-25T09:57:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './151' 2023-08-25T09:57:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/19' 2023-08-25T09:57:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/15' 2023-08-25T09:57:19.516 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/31' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/36' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/20' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/12' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/39' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/10' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/34' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/38' 2023-08-25T09:57:19.517 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/11' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/16' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/32' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/30' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/14' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/18' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/13' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/33' 2023-08-25T09:57:19.518 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/17' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/35' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './152/37' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './152' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/19' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/15' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/31' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/36' 2023-08-25T09:57:19.519 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/20' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/12' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/39' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/10' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/34' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/38' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/11' 2023-08-25T09:57:19.520 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/16' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/32' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/30' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/14' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/18' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/13' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/33' 2023-08-25T09:57:19.521 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/17' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/35' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed './153/37' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './153' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/19' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/15' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/31' 2023-08-25T09:57:19.522 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/36' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/20' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/12' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/39' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/10' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/34' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/38' 2023-08-25T09:57:19.523 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/11' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/16' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/32' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/30' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/14' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/18' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/13' 2023-08-25T09:57:19.524 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/33' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/17' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/35' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './154/37' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './154' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/19' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/15' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/31' 2023-08-25T09:57:19.525 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/36' 2023-08-25T09:57:19.526 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/20' 2023-08-25T09:57:19.526 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/12' 2023-08-25T09:57:19.526 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/39' 2023-08-25T09:57:19.526 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/10' 2023-08-25T09:57:19.526 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/34' 2023-08-25T09:57:19.526 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/38' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/11' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/16' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/32' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/30' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/14' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/18' 2023-08-25T09:57:19.527 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/13' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/33' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/17' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/35' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed './155/37' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './155' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/19' 2023-08-25T09:57:19.528 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/15' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/31' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/36' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/20' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/12' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/39' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/10' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/34' 2023-08-25T09:57:19.529 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/38' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/11' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/16' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/32' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/30' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/14' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/18' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/13' 2023-08-25T09:57:19.530 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/33' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/17' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/35' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed './156/37' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './156' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/19' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/15' 2023-08-25T09:57:19.531 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/31' 2023-08-25T09:57:19.532 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/36' 2023-08-25T09:57:19.532 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/20' 2023-08-25T09:57:19.532 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/12' 2023-08-25T09:57:19.532 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/39' 2023-08-25T09:57:19.532 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/10' 2023-08-25T09:57:19.532 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/34' 2023-08-25T09:57:19.533 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/38' 2023-08-25T09:57:19.533 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/11' 2023-08-25T09:57:19.533 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/16' 2023-08-25T09:57:19.533 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/32' 2023-08-25T09:57:19.533 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/30' 2023-08-25T09:57:19.533 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/14' 2023-08-25T09:57:19.534 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/18' 2023-08-25T09:57:19.534 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/13' 2023-08-25T09:57:19.534 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/33' 2023-08-25T09:57:22.070 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/17' 2023-08-25T09:57:22.070 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/35' 2023-08-25T09:57:22.070 INFO:tasks.workunit.client.0.smithi071.stdout:removed './157/37' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './157' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/19' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/15' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/31' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/36' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/20' 2023-08-25T09:57:22.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/12' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/39' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/10' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/34' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/38' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/11' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/16' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/32' 2023-08-25T09:57:22.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/30' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/14' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/18' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/13' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/33' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/17' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/35' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './158/37' 2023-08-25T09:57:22.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './158' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/19' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/15' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/31' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/36' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/20' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/12' 2023-08-25T09:57:22.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/39' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/10' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/34' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/38' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/11' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/16' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/32' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/30' 2023-08-25T09:57:22.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/14' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/18' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/13' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/33' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/17' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/35' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './159/37' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './159' 2023-08-25T09:57:22.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/19' 2023-08-25T09:57:22.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/15' 2023-08-25T09:57:22.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/31' 2023-08-25T09:57:22.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/36' 2023-08-25T09:57:22.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/20' 2023-08-25T09:57:22.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/12' 2023-08-25T09:57:22.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/39' 2023-08-25T09:57:22.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/10' 2023-08-25T09:57:22.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/34' 2023-08-25T09:57:22.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/38' 2023-08-25T09:57:22.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/11' 2023-08-25T09:57:22.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/16' 2023-08-25T09:57:22.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/32' 2023-08-25T09:57:22.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/30' 2023-08-25T09:57:22.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/14' 2023-08-25T09:57:22.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/18' 2023-08-25T09:57:22.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/13' 2023-08-25T09:57:22.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/33' 2023-08-25T09:57:22.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/17' 2023-08-25T09:57:22.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/35' 2023-08-25T09:57:22.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed './160/37' 2023-08-25T09:57:22.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './160' 2023-08-25T09:57:22.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/19' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/15' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/31' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/36' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/20' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/12' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/39' 2023-08-25T09:57:22.081 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/10' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/34' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/38' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/11' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/16' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/32' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/30' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/14' 2023-08-25T09:57:22.082 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/18' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/13' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/33' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/17' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/35' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './161/37' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './161' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/19' 2023-08-25T09:57:22.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/15' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/31' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/36' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/20' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/12' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/39' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/10' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/34' 2023-08-25T09:57:22.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/38' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/11' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/16' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/32' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/30' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/14' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/18' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/13' 2023-08-25T09:57:22.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/33' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/17' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/35' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './162/37' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './162' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/19' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/15' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/31' 2023-08-25T09:57:22.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/36' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/20' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/12' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/39' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/10' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/34' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/38' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/11' 2023-08-25T09:57:22.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/16' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/32' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/30' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/14' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/18' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/13' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/33' 2023-08-25T09:57:22.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/17' 2023-08-25T09:57:22.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/35' 2023-08-25T09:57:22.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './163/37' 2023-08-25T09:57:22.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './163' 2023-08-25T09:57:22.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/19' 2023-08-25T09:57:22.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/15' 2023-08-25T09:57:22.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/31' 2023-08-25T09:57:22.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/36' 2023-08-25T09:57:22.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/20' 2023-08-25T09:57:22.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/12' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/39' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/10' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/34' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/38' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/11' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/16' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/32' 2023-08-25T09:57:22.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/30' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/14' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/18' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/13' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/33' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/17' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/35' 2023-08-25T09:57:22.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './164/37' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './164' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/19' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/15' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/31' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/36' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/20' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/12' 2023-08-25T09:57:22.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/39' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/10' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/34' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/38' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/11' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/16' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/32' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/30' 2023-08-25T09:57:22.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/14' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/18' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/13' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/33' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/17' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/35' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './165/37' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './165' 2023-08-25T09:57:22.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/19' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/15' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/31' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/36' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/20' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/12' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/39' 2023-08-25T09:57:22.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/10' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/34' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/38' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/11' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/16' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/32' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/30' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/14' 2023-08-25T09:57:22.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/18' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/13' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/33' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/17' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/35' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './166/37' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './166' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/19' 2023-08-25T09:57:22.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/15' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/31' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/36' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/20' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/12' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/39' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/10' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/34' 2023-08-25T09:57:22.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/38' 2023-08-25T09:57:24.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:57:24.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:57:24.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/11' 2023-08-25T09:57:24.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/16' 2023-08-25T09:57:24.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/32' 2023-08-25T09:57:24.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/30' 2023-08-25T09:57:24.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/14' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/18' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/13' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/33' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/17' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/35' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './167/37' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './167' 2023-08-25T09:57:24.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/19' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/15' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/31' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/36' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/20' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/12' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/39' 2023-08-25T09:57:24.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/10' 2023-08-25T09:57:24.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/34' 2023-08-25T09:57:24.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/38' 2023-08-25T09:57:24.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/11' 2023-08-25T09:57:24.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/16' 2023-08-25T09:57:24.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/32' 2023-08-25T09:57:24.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/30' 2023-08-25T09:57:24.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/14' 2023-08-25T09:57:24.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/18' 2023-08-25T09:57:24.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/13' 2023-08-25T09:57:24.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/33' 2023-08-25T09:57:24.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/17' 2023-08-25T09:57:24.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/35' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './168/37' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './168' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/19' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/15' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/31' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/36' 2023-08-25T09:57:24.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/20' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/12' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/39' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/10' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/34' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/38' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/11' 2023-08-25T09:57:24.571 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/16' 2023-08-25T09:57:24.572 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/32' 2023-08-25T09:57:24.572 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/30' 2023-08-25T09:57:24.572 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/14' 2023-08-25T09:57:24.572 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/18' 2023-08-25T09:57:24.572 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/13' 2023-08-25T09:57:24.572 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/33' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/17' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/35' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed './169/37' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './169' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/19' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/15' 2023-08-25T09:57:24.573 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/31' 2023-08-25T09:57:24.574 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/36' 2023-08-25T09:57:24.574 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/20' 2023-08-25T09:57:24.574 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/12' 2023-08-25T09:57:24.574 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/39' 2023-08-25T09:57:24.574 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/10' 2023-08-25T09:57:24.575 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/34' 2023-08-25T09:57:24.575 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/38' 2023-08-25T09:57:24.575 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/11' 2023-08-25T09:57:24.575 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/16' 2023-08-25T09:57:24.576 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/32' 2023-08-25T09:57:24.576 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/30' 2023-08-25T09:57:24.576 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/14' 2023-08-25T09:57:24.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/18' 2023-08-25T09:57:24.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/13' 2023-08-25T09:57:24.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/33' 2023-08-25T09:57:24.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/17' 2023-08-25T09:57:24.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/35' 2023-08-25T09:57:24.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './170/37' 2023-08-25T09:57:24.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './170' 2023-08-25T09:57:24.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/19' 2023-08-25T09:57:24.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/15' 2023-08-25T09:57:24.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/31' 2023-08-25T09:57:24.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/36' 2023-08-25T09:57:24.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/20' 2023-08-25T09:57:24.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/12' 2023-08-25T09:57:24.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/39' 2023-08-25T09:57:24.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/10' 2023-08-25T09:57:24.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/34' 2023-08-25T09:57:24.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/38' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/11' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/16' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/32' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/30' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/14' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/18' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/13' 2023-08-25T09:57:24.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/33' 2023-08-25T09:57:24.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/17' 2023-08-25T09:57:24.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/35' 2023-08-25T09:57:24.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './171/37' 2023-08-25T09:57:24.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './171' 2023-08-25T09:57:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/19' 2023-08-25T09:57:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/15' 2023-08-25T09:57:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/31' 2023-08-25T09:57:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/36' 2023-08-25T09:57:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/20' 2023-08-25T09:57:24.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/12' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/39' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/10' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/34' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/38' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/11' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/16' 2023-08-25T09:57:24.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/32' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/30' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/14' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/18' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/13' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/33' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/17' 2023-08-25T09:57:24.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/35' 2023-08-25T09:57:24.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './172/37' 2023-08-25T09:57:24.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './172' 2023-08-25T09:57:24.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/19' 2023-08-25T09:57:24.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/15' 2023-08-25T09:57:24.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/31' 2023-08-25T09:57:24.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/36' 2023-08-25T09:57:24.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/20' 2023-08-25T09:57:24.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/12' 2023-08-25T09:57:24.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/39' 2023-08-25T09:57:24.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/10' 2023-08-25T09:57:24.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/34' 2023-08-25T09:57:24.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/38' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/11' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/16' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/32' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/30' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/14' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/18' 2023-08-25T09:57:24.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/13' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/33' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/17' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/35' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './173/37' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './173' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/19' 2023-08-25T09:57:24.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/15' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/31' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/36' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/20' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/12' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/39' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/10' 2023-08-25T09:57:24.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/34' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/38' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/11' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/16' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/32' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/30' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/14' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/18' 2023-08-25T09:57:24.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/13' 2023-08-25T09:57:24.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/33' 2023-08-25T09:57:24.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/17' 2023-08-25T09:57:24.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/35' 2023-08-25T09:57:24.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './174/37' 2023-08-25T09:57:24.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './174' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/19' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/15' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/31' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/36' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/20' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/12' 2023-08-25T09:57:24.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/39' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/10' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/34' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/38' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/11' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/16' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/32' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/30' 2023-08-25T09:57:24.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/14' 2023-08-25T09:57:24.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/18' 2023-08-25T09:57:24.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/13' 2023-08-25T09:57:24.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/33' 2023-08-25T09:57:24.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/17' 2023-08-25T09:57:24.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/35' 2023-08-25T09:57:24.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './175/37' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './175' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/19' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/15' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/31' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/36' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/20' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/12' 2023-08-25T09:57:24.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/39' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/10' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/34' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/38' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/11' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/16' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/32' 2023-08-25T09:57:24.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/30' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/14' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/18' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/13' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/33' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/17' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/35' 2023-08-25T09:57:24.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './176/37' 2023-08-25T09:57:24.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './176' 2023-08-25T09:57:24.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/19' 2023-08-25T09:57:24.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/15' 2023-08-25T09:57:27.309 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/31' 2023-08-25T09:57:27.309 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/36' 2023-08-25T09:57:27.309 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/20' 2023-08-25T09:57:27.310 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/12' 2023-08-25T09:57:27.310 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/39' 2023-08-25T09:57:27.310 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/10' 2023-08-25T09:57:27.311 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/34' 2023-08-25T09:57:27.311 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/38' 2023-08-25T09:57:27.311 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/11' 2023-08-25T09:57:27.311 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/16' 2023-08-25T09:57:27.312 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/32' 2023-08-25T09:57:27.312 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/30' 2023-08-25T09:57:27.312 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/14' 2023-08-25T09:57:27.313 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/18' 2023-08-25T09:57:27.313 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/13' 2023-08-25T09:57:27.313 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/33' 2023-08-25T09:57:27.313 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/17' 2023-08-25T09:57:27.314 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/35' 2023-08-25T09:57:27.314 INFO:tasks.workunit.client.0.smithi071.stdout:removed './177/37' 2023-08-25T09:57:27.314 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './177' 2023-08-25T09:57:27.315 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/19' 2023-08-25T09:57:27.315 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/15' 2023-08-25T09:57:27.315 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/31' 2023-08-25T09:57:27.315 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/36' 2023-08-25T09:57:27.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/20' 2023-08-25T09:57:27.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/12' 2023-08-25T09:57:27.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/39' 2023-08-25T09:57:27.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/10' 2023-08-25T09:57:27.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/34' 2023-08-25T09:57:27.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/38' 2023-08-25T09:57:27.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/11' 2023-08-25T09:57:27.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/16' 2023-08-25T09:57:27.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/32' 2023-08-25T09:57:27.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/30' 2023-08-25T09:57:27.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/14' 2023-08-25T09:57:27.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/18' 2023-08-25T09:57:27.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/13' 2023-08-25T09:57:27.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/33' 2023-08-25T09:57:27.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/17' 2023-08-25T09:57:27.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/35' 2023-08-25T09:57:27.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './178/37' 2023-08-25T09:57:27.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './178' 2023-08-25T09:57:27.321 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/19' 2023-08-25T09:57:27.321 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/15' 2023-08-25T09:57:27.321 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/31' 2023-08-25T09:57:27.321 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/36' 2023-08-25T09:57:27.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/20' 2023-08-25T09:57:27.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/12' 2023-08-25T09:57:27.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/39' 2023-08-25T09:57:27.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/10' 2023-08-25T09:57:27.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/34' 2023-08-25T09:57:27.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/38' 2023-08-25T09:57:27.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/11' 2023-08-25T09:57:27.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/16' 2023-08-25T09:57:27.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/32' 2023-08-25T09:57:27.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/30' 2023-08-25T09:57:27.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/14' 2023-08-25T09:57:27.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/18' 2023-08-25T09:57:27.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/13' 2023-08-25T09:57:27.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/33' 2023-08-25T09:57:27.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/17' 2023-08-25T09:57:27.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/35' 2023-08-25T09:57:27.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './179/37' 2023-08-25T09:57:27.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './179' 2023-08-25T09:57:27.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/19' 2023-08-25T09:57:27.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/15' 2023-08-25T09:57:27.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/31' 2023-08-25T09:57:27.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/36' 2023-08-25T09:57:27.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/20' 2023-08-25T09:57:27.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/12' 2023-08-25T09:57:27.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/39' 2023-08-25T09:57:27.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/10' 2023-08-25T09:57:27.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/34' 2023-08-25T09:57:27.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/38' 2023-08-25T09:57:27.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/11' 2023-08-25T09:57:27.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/16' 2023-08-25T09:57:27.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/32' 2023-08-25T09:57:27.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/30' 2023-08-25T09:57:27.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/14' 2023-08-25T09:57:27.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/18' 2023-08-25T09:57:27.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/13' 2023-08-25T09:57:27.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/33' 2023-08-25T09:57:27.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/17' 2023-08-25T09:57:27.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/35' 2023-08-25T09:57:27.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './180/37' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './180' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/19' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/15' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/31' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/36' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/20' 2023-08-25T09:57:27.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/12' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/39' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/10' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/34' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/38' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/11' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/16' 2023-08-25T09:57:27.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/32' 2023-08-25T09:57:27.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/30' 2023-08-25T09:57:27.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/14' 2023-08-25T09:57:27.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/18' 2023-08-25T09:57:27.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/13' 2023-08-25T09:57:27.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/33' 2023-08-25T09:57:27.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/17' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/35' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './181/37' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './181' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/19' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/15' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/31' 2023-08-25T09:57:27.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/36' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/20' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/12' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/39' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/10' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/34' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/38' 2023-08-25T09:57:27.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/11' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/16' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/32' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/30' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/14' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/18' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/13' 2023-08-25T09:57:27.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/33' 2023-08-25T09:57:27.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/17' 2023-08-25T09:57:27.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/35' 2023-08-25T09:57:27.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './182/37' 2023-08-25T09:57:27.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './182' 2023-08-25T09:57:27.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/19' 2023-08-25T09:57:27.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/15' 2023-08-25T09:57:27.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/31' 2023-08-25T09:57:27.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/36' 2023-08-25T09:57:27.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/20' 2023-08-25T09:57:27.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/12' 2023-08-25T09:57:27.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/39' 2023-08-25T09:57:27.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/10' 2023-08-25T09:57:27.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/34' 2023-08-25T09:57:27.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/38' 2023-08-25T09:57:27.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/11' 2023-08-25T09:57:27.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/16' 2023-08-25T09:57:27.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/32' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/30' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/14' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/18' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/13' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/33' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/17' 2023-08-25T09:57:27.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/35' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './183/37' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './183' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/19' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/15' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/31' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/36' 2023-08-25T09:57:27.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/20' 2023-08-25T09:57:27.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/12' 2023-08-25T09:57:27.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/39' 2023-08-25T09:57:27.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/10' 2023-08-25T09:57:27.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/34' 2023-08-25T09:57:27.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/38' 2023-08-25T09:57:27.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/11' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/16' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/32' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/30' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/14' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/18' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/13' 2023-08-25T09:57:27.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/33' 2023-08-25T09:57:27.347 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/17' 2023-08-25T09:57:27.347 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/35' 2023-08-25T09:57:27.347 INFO:tasks.workunit.client.0.smithi071.stdout:removed './184/37' 2023-08-25T09:57:27.347 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './184' 2023-08-25T09:57:27.347 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/19' 2023-08-25T09:57:27.347 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/15' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/31' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/36' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/20' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/12' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/39' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/10' 2023-08-25T09:57:27.348 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/34' 2023-08-25T09:57:27.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/38' 2023-08-25T09:57:27.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/11' 2023-08-25T09:57:27.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/16' 2023-08-25T09:57:27.349 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/32' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/30' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/14' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/18' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/13' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/33' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/17' 2023-08-25T09:57:27.350 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/35' 2023-08-25T09:57:27.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './185/37' 2023-08-25T09:57:27.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './185' 2023-08-25T09:57:27.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/19' 2023-08-25T09:57:27.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/15' 2023-08-25T09:57:27.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/31' 2023-08-25T09:57:27.351 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/36' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/20' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/12' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/39' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/10' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/34' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/38' 2023-08-25T09:57:27.352 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/11' 2023-08-25T09:57:27.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/16' 2023-08-25T09:57:27.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/32' 2023-08-25T09:57:27.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/30' 2023-08-25T09:57:27.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/14' 2023-08-25T09:57:27.353 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/18' 2023-08-25T09:57:29.542 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/13' 2023-08-25T09:57:29.542 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/33' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/17' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/35' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed './186/37' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './186' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/19' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/15' 2023-08-25T09:57:29.543 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/31' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/36' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/20' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/12' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/39' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/10' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/34' 2023-08-25T09:57:29.544 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/38' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/11' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/16' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/32' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/30' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/14' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/18' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/13' 2023-08-25T09:57:29.545 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/33' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/17' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/35' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './187/37' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './187' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/19' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/15' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/31' 2023-08-25T09:57:29.546 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/36' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/20' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/12' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/39' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/10' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/34' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/38' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/11' 2023-08-25T09:57:29.547 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/16' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/32' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/30' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/14' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/18' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/13' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/33' 2023-08-25T09:57:29.548 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/17' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/35' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed './188/37' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './188' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/19' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/15' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/31' 2023-08-25T09:57:29.549 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/36' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/20' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/12' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/39' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/10' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/34' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/38' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/11' 2023-08-25T09:57:29.550 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/16' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/32' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/30' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/14' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/18' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/13' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/33' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/17' 2023-08-25T09:57:29.551 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/35' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './189/37' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './189' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/19' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/15' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/31' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/36' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/20' 2023-08-25T09:57:29.552 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/12' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/39' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/10' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/34' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/38' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/11' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/16' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/32' 2023-08-25T09:57:29.553 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/30' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/14' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/18' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/13' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/33' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/17' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/35' 2023-08-25T09:57:29.554 INFO:tasks.workunit.client.0.smithi071.stdout:removed './190/37' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './190' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/19' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/15' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/31' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/36' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/20' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/12' 2023-08-25T09:57:29.555 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/39' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/10' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/34' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/38' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/11' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/16' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/32' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/30' 2023-08-25T09:57:29.556 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/14' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/18' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/13' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/33' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/17' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/35' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './191/37' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './191' 2023-08-25T09:57:29.557 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/19' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/15' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/31' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/36' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/20' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/12' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/39' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/10' 2023-08-25T09:57:29.558 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/34' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/38' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/11' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/16' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/32' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/30' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/14' 2023-08-25T09:57:29.559 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/18' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/13' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/33' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/17' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/35' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './192/37' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './192' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/19' 2023-08-25T09:57:29.560 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/15' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/31' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/36' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/20' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/12' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/39' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/10' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/34' 2023-08-25T09:57:29.561 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/38' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/11' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/16' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/32' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/30' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/14' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/18' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/13' 2023-08-25T09:57:29.562 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/33' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/17' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/35' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed './193/37' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './193' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/19' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/15' 2023-08-25T09:57:29.563 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/31' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/36' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/20' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/12' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/39' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/10' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/34' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/38' 2023-08-25T09:57:29.564 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/11' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/16' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/32' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/30' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/14' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/18' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/13' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/33' 2023-08-25T09:57:29.565 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/17' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/35' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './194/37' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './194' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/19' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/15' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/31' 2023-08-25T09:57:29.566 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/36' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/20' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/12' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/39' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/10' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/34' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/38' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/11' 2023-08-25T09:57:29.567 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/16' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/32' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/30' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/14' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/18' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/13' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/33' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/17' 2023-08-25T09:57:29.568 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/35' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './195/37' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './195' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/19' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/15' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/31' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/36' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/20' 2023-08-25T09:57:29.569 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/12' 2023-08-25T09:57:29.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/39' 2023-08-25T09:57:29.570 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/10' 2023-08-25T09:57:33.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:33.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.442+0000 7efc083ab640 1 Processor -- start 2023-08-25T09:57:33.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.442+0000 7efc083ab640 1 -- start start 2023-08-25T09:57:33.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.443+0000 7efc083ab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc000c3660 0x7efc000c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.443+0000 7efc083ab640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc001509d0 0x7efc00150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc083ab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc000c1dc0 con 0x7efc000c3a40 2023-08-25T09:57:33.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc083ab640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc00157a80 con 0x7efc001509d0 2023-08-25T09:57:33.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc083ab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc001513a0 con 0x7efc000c3660 2023-08-25T09:57:33.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06948640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc001509d0 0x7efc00150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc05946640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efc000c3a40 0x7efc0006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50390/0 (socket says 172.21.15.71:50390) 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc05946640 1 -- 172.21.15.71:0/2009576614 learned_addr learned my addr 172.21.15.71:0/2009576614 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06948640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7efc001509d0 0x7efc00150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60918/0 (socket says 172.21.15.71:60918) 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06147640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc000c3660 0x7efc000c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06948640 1 -- 172.21.15.71:0/2009576614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc000c3660 msgr2=0x7efc000c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06948640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc000c3660 0x7efc000c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06948640 1 -- 172.21.15.71:0/2009576614 >> v1:172.21.15.71:6789/0 conn(0x7efc000c3a40 legacy=0x7efc0006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:33.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06948640 1 -- 172.21.15.71:0/2009576614 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00156e00 con 0x7efc001509d0 2023-08-25T09:57:33.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc06147640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc000c3660 0x7efc000c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:33.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.444+0000 7efc05145640 1 -- 172.21.15.71:0/2009576614 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828183203 0 0) 0x7efc000c1dc0 con 0x7efc000c3a40 2023-08-25T09:57:33.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc06948640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc001509d0 0x7efc00150e60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d90 tx=0x7efbf802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8669ee090fa3cdb server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:33.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc05145640 1 -- 172.21.15.71:0/2009576614 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf802cce0 con 0x7efc001509d0 2023-08-25T09:57:33.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc05145640 1 -- 172.21.15.71:0/2009576614 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf802ce40 con 0x7efc001509d0 2023-08-25T09:57:33.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc05145640 1 -- 172.21.15.71:0/2009576614 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8034660 con 0x7efc001509d0 2023-08-25T09:57:33.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc083ab640 1 -- 172.21.15.71:0/2009576614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc001509d0 msgr2=0x7efc00150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc083ab640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc001509d0 0x7efc00150e60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009d90 tx=0x7efbf802bf30 comp rx=0 tx=0).stop 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.446+0000 7efc083ab640 1 -- 172.21.15.71:0/2009576614 shutdown_connections 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.447+0000 7efc083ab640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc001509d0 0x7efc00150e60 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.447+0000 7efc083ab640 1 --2- 172.21.15.71:0/2009576614 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc000c3660 0x7efc000c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.447+0000 7efc083ab640 1 -- 172.21.15.71:0/2009576614 >> 172.21.15.71:0/2009576614 conn(0x7efc000654f0 msgr2=0x7efc00065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.447+0000 7efc083ab640 1 -- 172.21.15.71:0/2009576614 shutdown_connections 2023-08-25T09:57:33.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.447+0000 7efc083ab640 1 -- 172.21.15.71:0/2009576614 wait complete. 2023-08-25T09:57:33.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.448+0000 7efc083ab640 1 Processor -- start 2023-08-25T09:57:33.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.448+0000 7efc083ab640 1 -- start start 2023-08-25T09:57:33.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.449+0000 7efc083ab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 0x7efc0014d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.449+0000 7efc083ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc000c3a40 0x7efc0014d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.449+0000 7efc083ab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc001509d0 0x7efc00151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.449+0000 7efc083ab640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00159b10 con 0x7efc000c3660 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.449+0000 7efc083ab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00159d70 con 0x7efc000c3a40 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.449+0000 7efc083ab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0015a050 con 0x7efc001509d0 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 0x7efc0014d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 0x7efc0014d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60934/0 (socket says 172.21.15.71:60934) 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 -- 172.21.15.71:0/3500952299 learned_addr learned my addr 172.21.15.71:0/3500952299 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc05946640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc000c3a40 0x7efc0014d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06948640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc001509d0 0x7efc00151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 -- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc001509d0 msgr2=0x7efc00151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc001509d0 0x7efc00151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 -- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc000c3a40 msgr2=0x7efc0014d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc000c3a40 0x7efc0014d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06147640 1 -- 172.21.15.71:0/3500952299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8009a70 con 0x7efc000c3660 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.450+0000 7efc06948640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc001509d0 0x7efc00151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:33.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.451+0000 7efc06147640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 0x7efc0014d090 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c7f0 tx=0x7efbfc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:33.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.451+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc007820 con 0x7efc000c3660 2023-08-25T09:57:33.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.451+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc007980 con 0x7efc000c3660 2023-08-25T09:57:33.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.451+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc007cf0 con 0x7efc000c3660 2023-08-25T09:57:33.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.451+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc000c0b70 con 0x7efc000c3660 2023-08-25T09:57:33.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.451+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc000c1130 con 0x7efc000c3660 2023-08-25T09:57:33.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.452+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc000c29c0 con 0x7efc000c3660 2023-08-25T09:57:33.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.453+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efbfc0113f0 con 0x7efc000c3660 2023-08-25T09:57:33.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.454+0000 7efbf6ffd640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbe406a050 0x7efbe406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.454+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7efbfc089110 con 0x7efc000c3660 2023-08-25T09:57:33.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.454+0000 7efc05946640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbe406a050 0x7efbe406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.455+0000 7efc05946640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbe406a050 0x7efbe406c510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc001569e0 tx=0x7efbf0007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:33.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.458+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efbfc053f80 con 0x7efc000c3660 2023-08-25T09:57:33.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.688+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc000c09d0 con 0x7efc000c3660 2023-08-25T09:57:33.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.688+0000 7efbf6ffd640 1 -- 172.21.15.71:0/3500952299 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6540 (secure 0 0 0) 0x7efc000c09d0 con 0x7efc000c3660 2023-08-25T09:57:33.690 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:33.691 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:32.709489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:33.691 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 61 2023-08-25T09:57:33.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbe406a050 msgr2=0x7efbe406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:33.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbe406a050 0x7efbe406c510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc001569e0 tx=0x7efbf0007c00 comp rx=0 tx=0).stop 2023-08-25T09:57:33.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 msgr2=0x7efc0014d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:33.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 0x7efc0014d090 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c7f0 tx=0x7efbfc00ccb0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 shutdown_connections 2023-08-25T09:57:33.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc000c3a40 0x7efc0014d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbe406a050 0x7efbe406c510 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc001509d0 0x7efc00151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 --2- 172.21.15.71:0/3500952299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc000c3660 0x7efc0014d090 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 >> 172.21.15.71:0/3500952299 conn(0x7efc000654f0 msgr2=0x7efc000c9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:33.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 shutdown_connections 2023-08-25T09:57:33.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.692+0000 7efc083ab640 1 -- 172.21.15.71:0/3500952299 wait complete. 2023-08-25T09:57:33.713 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-08-25T09:57:33.714 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-08-25T09:57:33.714 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T09:57:33.734 INFO:tasks.ceph.mds.c:Stopped 2023-08-25T09:57:33.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:33.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.988+0000 7f6a61d55640 1 Processor -- start 2023-08-25T09:57:33.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.988+0000 7f6a61d55640 1 -- start start 2023-08-25T09:57:33.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a61d55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a61d55640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6a5c0c3a40 0x7f6a5c06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a61d55640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6a5c1509d0 0x7f6a5c150e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:33.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a61d55640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a5c157a80 con 0x7f6a5c0c3a40 2023-08-25T09:57:33.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a61d55640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a5c0c1dc0 con 0x7f6a5c1509d0 2023-08-25T09:57:33.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a61d55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c151380 con 0x7f6a5c0c3660 2023-08-25T09:57:33.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a5affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6a5c0c3a40 0x7f6a5c06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a5affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6a5c0c3a40 0x7f6a5c06ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60940/0 (socket says 172.21.15.71:60940) 2023-08-25T09:57:33.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a5affd640 1 -- 172.21.15.71:0/4252367978 learned_addr learned my addr 172.21.15.71:0/4252367978 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:33.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.990+0000 7f6a5b7fe640 1 --2- 172.21.15.71:0/4252367978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.991+0000 7f6a5bfff640 1 --2- 172.21.15.71:0/4252367978 >> v2:172.21.15.164:3300/0 conn(0x7f6a5c1509d0 0x7f6a5c150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:33.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.991+0000 7f6a5affd640 1 -- 172.21.15.71:0/4252367978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 msgr2=0x7f6a5c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:33.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.991+0000 7f6a5affd640 1 --2- 172.21.15.71:0/4252367978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.991+0000 7f6a5affd640 1 -- 172.21.15.71:0/4252367978 >> v2:172.21.15.164:3300/0 conn(0x7f6a5c1509d0 msgr2=0x7f6a5c150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:33.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.991+0000 7f6a5affd640 1 --2- 172.21.15.71:0/4252367978 >> v2:172.21.15.164:3300/0 conn(0x7f6a5c1509d0 0x7f6a5c150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:33.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.991+0000 7f6a5affd640 1 -- 172.21.15.71:0/4252367978 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c06cf70 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.992+0000 7f6a5affd640 1 --2- 172.21.15.71:0/4252367978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c06ca30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0c4090 tx=0x7f6a5002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd1a2da98f1a432 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:34.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.992+0000 7f6a5a7fc640 1 -- 172.21.15.71:0/4252367978 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50030850 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.992+0000 7f6a5a7fc640 1 -- 172.21.15.71:0/4252367978 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a500309b0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.992+0000 7f6a5a7fc640 1 -- 172.21.15.71:0/4252367978 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50030d40 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 -- 172.21.15.71:0/4252367978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 msgr2=0x7f6a5c06ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 --2- 172.21.15.71:0/4252367978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c06ca30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0c4090 tx=0x7f6a5002b990 comp rx=0 tx=0).stop 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 -- 172.21.15.71:0/4252367978 shutdown_connections 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 --2- 172.21.15.71:0/4252367978 >> v2:172.21.15.164:3300/0 conn(0x7f6a5c1509d0 0x7f6a5c150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 --2- 172.21.15.71:0/4252367978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c06ca30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 --2- 172.21.15.71:0/4252367978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 -- 172.21.15.71:0/4252367978 >> 172.21.15.71:0/4252367978 conn(0x7f6a5c0654f0 msgr2=0x7f6a5c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:34.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 -- 172.21.15.71:0/4252367978 shutdown_connections 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.993+0000 7f6a61d55640 1 -- 172.21.15.71:0/4252367978 wait complete. 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.995+0000 7f6a61d55640 1 Processor -- start 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.995+0000 7f6a61d55640 1 -- start start 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a61d55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a61d55640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a61d55640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a5c1509d0 0x7f6a5c14bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:34.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a61d55640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c0c62a0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a61d55640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c153fa0 con 0x7f6a5c1509d0 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a61d55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5c154260 con 0x7f6a5c0c3660 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a5affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a5affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60946/0 (socket says 172.21.15.71:60946) 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.996+0000 7f6a5affd640 1 -- 172.21.15.71:0/1711272648 learned_addr learned my addr 172.21.15.71:0/1711272648 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5b7fe640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5bfff640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a5c1509d0 0x7f6a5c14bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:34.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5affd640 1 -- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 msgr2=0x7f6a5c153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5affd640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5affd640 1 -- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a5c1509d0 msgr2=0x7f6a5c14bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5affd640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a5c1509d0 0x7f6a5c14bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5affd640 1 -- 172.21.15.71:0/1711272648 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a500094d0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5b7fe640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a5affd640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c151f00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6a5002b9d0 tx=0x7f6a50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:34.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50039b60 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a50039ce0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a50043650 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.997+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5c1eeec0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.998+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5c151550 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.999+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5c0c29c0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:33.999+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6a5002e070 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.000+0000 7f6a43fff640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6a4806a050 0x7f6a4806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:34.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.001+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7f6a500aacd0 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.001+0000 7f6a5b7fe640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6a4806a050 0x7f6a4806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:34.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.001+0000 7f6a5b7fe640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6a4806a050 0x7f6a4806c510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c006fd0 tx=0x7f6a4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:34.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.005+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6a50075c60 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.236+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a5c1ef080 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.237+0000 7f6a43fff640 1 -- 172.21.15.71:0/1711272648 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6540 (secure 0 0 0) 0x7f6a50048020 con 0x7f6a5c0c3a40 2023-08-25T09:57:34.239 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:34.239 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:32.709489+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:34.240 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 61 2023-08-25T09:57:34.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.239+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6a4806a050 msgr2=0x7f6a4806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:34.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.239+0000 7f6a61d55640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6a4806a050 0x7f6a4806c510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c006fd0 tx=0x7f6a4c008040 comp rx=0 tx=0).stop 2023-08-25T09:57:34.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.239+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 msgr2=0x7f6a5c151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:34.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.239+0000 7f6a61d55640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c151f00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6a5002b9d0 tx=0x7f6a50004730 comp rx=0 tx=0).stop 2023-08-25T09:57:34.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 shutdown_connections 2023-08-25T09:57:34.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a5c1509d0 0x7f6a5c14bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6a4806a050 0x7f6a4806c510 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6a5c0c3a40 0x7f6a5c151f00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 --2- 172.21.15.71:0/1711272648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a5c0c3660 0x7f6a5c153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:34.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 >> 172.21.15.71:0/1711272648 conn(0x7f6a5c0654f0 msgr2=0x7f6a5c0c5350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:34.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 shutdown_connections 2023-08-25T09:57:34.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:34.240+0000 7f6a61d55640 1 -- 172.21.15.71:0/1711272648 wait complete. 2023-08-25T09:57:34.262 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:36.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:36.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.488+0000 7f7fce9e8640 1 Processor -- start 2023-08-25T09:57:36.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.488+0000 7f7fce9e8640 1 -- start start 2023-08-25T09:57:36.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.490+0000 7f7fce9e8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.490+0000 7f7fce9e8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7fc80c3a40 0x7f7fc806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.490+0000 7f7fce9e8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc80c1dc0 con 0x7f7fc81509d0 2023-08-25T09:57:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.490+0000 7f7fce9e8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc8157a80 con 0x7f7fc80c3a40 2023-08-25T09:57:36.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.490+0000 7f7fce9e8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc806cf70 con 0x7f7fc80c3660 2023-08-25T09:57:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.490+0000 7f7fc7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7fc80c3a40 0x7f7fc806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc80c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:32780/0 (socket says 172.21.15.71:32780) 2023-08-25T09:57:36.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7fc80c3a40 0x7f7fc806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:35108/0 (socket says 172.21.15.71:35108) 2023-08-25T09:57:36.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc7fff640 1 -- 172.21.15.71:0/2886386876 learned_addr learned my addr 172.21.15.71:0/2886386876 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:36.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc77fe640 1 -- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 msgr2=0x7f7fc80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:36.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc77fe640 1 --2- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc77fe640 1 -- 172.21.15.71:0/2886386876 >> v1:172.21.15.164:6789/0 conn(0x7f7fc81509d0 legacy=0x7f7fc8150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:36.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc77fe640 1 -- 172.21.15.71:0/2886386876 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8156e00 con 0x7f7fc80c3a40 2023-08-25T09:57:36.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.491+0000 7f7fc7fff640 1 --2- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:57:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.493+0000 7f7fc77fe640 1 --2- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc806ca30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0097a0 tx=0x7f7fbc02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a9abaa89e6357c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.493+0000 7f7fc6ffd640 1 -- 172.21.15.71:0/2886386876 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc030800 con 0x7f7fc80c3a40 2023-08-25T09:57:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.493+0000 7f7fc6ffd640 1 -- 172.21.15.71:0/2886386876 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc030960 con 0x7f7fc80c3a40 2023-08-25T09:57:36.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.493+0000 7f7fc6ffd640 1 -- 172.21.15.71:0/2886386876 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc030cf0 con 0x7f7fc80c3a40 2023-08-25T09:57:36.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 -- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 msgr2=0x7f7fc806ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:36.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc806ca30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0097a0 tx=0x7f7fbc02b940 comp rx=0 tx=0).stop 2023-08-25T09:57:36.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 -- 172.21.15.71:0/2886386876 shutdown_connections 2023-08-25T09:57:36.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc806ca30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/2886386876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 -- 172.21.15.71:0/2886386876 >> 172.21.15.71:0/2886386876 conn(0x7f7fc80654f0 msgr2=0x7f7fc8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 -- 172.21.15.71:0/2886386876 shutdown_connections 2023-08-25T09:57:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.494+0000 7f7fce9e8640 1 -- 172.21.15.71:0/2886386876 wait complete. 2023-08-25T09:57:36.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.496+0000 7f7fce9e8640 1 Processor -- start 2023-08-25T09:57:36.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.496+0000 7f7fce9e8640 1 -- start start 2023-08-25T09:57:36.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fce9e8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc8148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:36.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fce9e8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc81ef860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fce9e8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 0x7f7fc8158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fce9e8640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc81492f0 con 0x7f7fc81509d0 2023-08-25T09:57:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fce9e8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc81498b0 con 0x7f7fc80c3a40 2023-08-25T09:57:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fce9e8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc8155d30 con 0x7f7fc80c3660 2023-08-25T09:57:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fccf85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 0x7f7fc8158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:36.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fccf85640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 0x7f7fc8158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60952/0 (socket says 172.21.15.71:60952) 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fccf85640 1 -- 172.21.15.71:0/183428581 learned_addr learned my addr 172.21.15.71:0/183428581 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fc77fe640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc81ef860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fc7fff640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc8148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fccf85640 1 -- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 msgr2=0x7f7fc8148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.498+0000 7f7fccf85640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc8148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fccf85640 1 -- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 msgr2=0x7f7fc81ef860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:36.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fccf85640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc81ef860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fccf85640 1 -- 172.21.15.71:0/183428581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb80094d0 con 0x7f7fc81509d0 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fccf85640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 0x7f7fc8158a00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000c7d0 tx=0x7f7fc000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0003b70 con 0x7f7fc81509d0 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc0003cf0 con 0x7f7fc81509d0 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fbc009400 con 0x7f7fc81509d0 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc000aa30 con 0x7f7fc81509d0 2023-08-25T09:57:36.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.499+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc81562f0 con 0x7f7fc81509d0 2023-08-25T09:57:36.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.501+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc80c29c0 con 0x7f7fc81509d0 2023-08-25T09:57:36.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.502+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7fc000ab90 con 0x7f7fc81509d0 2023-08-25T09:57:36.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.503+0000 7f7fc4ff9640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fb406a080 0x7f7fb406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:36.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.503+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7f7fc0088b80 con 0x7f7fc81509d0 2023-08-25T09:57:36.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.503+0000 7f7fc7fff640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fb406a080 0x7f7fb406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:36.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.504+0000 7f7fc7fff640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fb406a080 0x7f7fb406c540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009c50 tx=0x7f7fb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:36.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.509+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7fc0053a80 con 0x7f7fc81509d0 2023-08-25T09:57:36.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.741+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fc81564e0 con 0x7f7fc81509d0 2023-08-25T09:57:36.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.742+0000 7f7fc4ff9640 1 -- 172.21.15.71:0/183428581 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f7fc0003e60 con 0x7f7fc81509d0 2023-08-25T09:57:36.744 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:36.744 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:35.725682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:36.744 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2023-08-25T09:57:36.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.744+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fb406a080 msgr2=0x7f7fb406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:36.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.744+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fb406a080 0x7f7fb406c540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009c50 tx=0x7f7fb8008040 comp rx=0 tx=0).stop 2023-08-25T09:57:36.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.744+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 msgr2=0x7f7fc8158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:36.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.744+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 0x7f7fc8158a00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7fc000c7d0 tx=0x7f7fc000cc90 comp rx=0 tx=0).stop 2023-08-25T09:57:36.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 shutdown_connections 2023-08-25T09:57:36.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7fc80c3a40 0x7f7fc81ef860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7fb406a080 0x7f7fb406c540 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7fc81509d0 0x7f7fc8158a00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 --2- 172.21.15.71:0/183428581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7fc80c3660 0x7f7fc8148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:36.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 >> 172.21.15.71:0/183428581 conn(0x7f7fc80654f0 msgr2=0x7f7fc80658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:36.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 shutdown_connections 2023-08-25T09:57:36.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:36.745+0000 7f7fce9e8640 1 -- 172.21.15.71:0/183428581 wait complete. 2023-08-25T09:57:36.765 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:38.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:38.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.987+0000 7fbfd57e3640 1 Processor -- start 2023-08-25T09:57:38.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.988+0000 7fbfd57e3640 1 -- start start 2023-08-25T09:57:38.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.989+0000 7fbfd57e3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd006cc70 0x7fbfd00c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:38.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.989+0000 7fbfd57e3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbfd00c3940 0x7fbfd01598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:38.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.989+0000 7fbfd57e3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd006a960 con 0x7fbfd01594d0 2023-08-25T09:57:38.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.989+0000 7fbfd57e3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd006ac20 con 0x7fbfd00c3940 2023-08-25T09:57:38.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.989+0000 7fbfd57e3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd006af00 con 0x7fbfd006cc70 2023-08-25T09:57:38.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfceffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbfd01594d0 0x7fbfd014f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50398/0 (socket says 172.21.15.71:50398) 2023-08-25T09:57:38.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfceffd640 1 -- 172.21.15.71:0/4205104003 learned_addr learned my addr 172.21.15.71:0/4205104003 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:57:38.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfcf7fe640 1 --2- 172.21.15.71:0/4205104003 >> v2:172.21.15.71:3300/0 conn(0x7fbfd00c3940 0x7fbfd01598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:38.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfc7fff640 1 --2- 172.21.15.71:0/4205104003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd006cc70 0x7fbfd00c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:38.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfce7fc640 1 -- 172.21.15.71:0/4205104003 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974071318 0 0) 0x7fbfd006a960 con 0x7fbfd01594d0 2023-08-25T09:57:38.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/4205104003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd006cc70 msgr2=0x7fbfd00c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:38.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfcf7fe640 1 --2- 172.21.15.71:0/4205104003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd006cc70 0x7fbfd00c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:38.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/4205104003 >> v1:172.21.15.71:6789/0 conn(0x7fbfd01594d0 legacy=0x7fbfd014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:38.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.990+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/4205104003 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd00c2e60 con 0x7fbfd00c3940 2023-08-25T09:57:38.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.991+0000 7fbfcf7fe640 1 --2- 172.21.15.71:0/4205104003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd00c3940 0x7fbfd01598b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009d90 tx=0x7fbfc802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203d100f10def5b4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:38.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfce7fc640 1 -- 172.21.15.71:0/4205104003 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc802cc60 con 0x7fbfd00c3940 2023-08-25T09:57:38.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfce7fc640 1 -- 172.21.15.71:0/4205104003 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc802cdc0 con 0x7fbfd00c3940 2023-08-25T09:57:38.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfce7fc640 1 -- 172.21.15.71:0/4205104003 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8034660 con 0x7fbfd00c3940 2023-08-25T09:57:38.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 -- 172.21.15.71:0/4205104003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd00c3940 msgr2=0x7fbfd01598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:38.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/4205104003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd00c3940 0x7fbfd01598b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009d90 tx=0x7fbfc802beb0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 -- 172.21.15.71:0/4205104003 shutdown_connections 2023-08-25T09:57:39.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/4205104003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd00c3940 0x7fbfd01598b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/4205104003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd006cc70 0x7fbfd00c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 -- 172.21.15.71:0/4205104003 >> 172.21.15.71:0/4205104003 conn(0x7fbfd00654f0 msgr2=0x7fbfd0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:39.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 -- 172.21.15.71:0/4205104003 shutdown_connections 2023-08-25T09:57:39.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.992+0000 7fbfd57e3640 1 -- 172.21.15.71:0/4205104003 wait complete. 2023-08-25T09:57:39.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.994+0000 7fbfd57e3640 1 Processor -- start 2023-08-25T09:57:39.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.994+0000 7fbfd57e3640 1 -- start start 2023-08-25T09:57:39.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfd57e3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfd006cc70 0x7fbfd0158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:39.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfd57e3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd00c3940 0x7fbfd014a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:39.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfd57e3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 0x7fbfd00c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfd57e3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0157d00 con 0x7fbfd01594d0 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfd57e3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd006eaa0 con 0x7fbfd006cc70 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfd57e3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0156e30 con 0x7fbfd00c3940 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfcf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 0x7fbfd00c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfcf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 0x7fbfd00c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60980/0 (socket says 172.21.15.71:60980) 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.995+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/60167519 learned_addr learned my addr 172.21.15.71:0/60167519 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfc7fff640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd00c3940 0x7fbfd014a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:39.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd00c3940 msgr2=0x7fbfd014a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:39.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfceffd640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfd006cc70 0x7fbfd0158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:39.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfcf7fe640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd00c3940 0x7fbfd014a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfd006cc70 msgr2=0x7fbfd0158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:39.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfcf7fe640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfd006cc70 0x7fbfd0158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfcf7fe640 1 -- 172.21.15.71:0/60167519 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8009a70 con 0x7fbfd01594d0 2023-08-25T09:57:39.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfcf7fe640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 0x7fbfd00c94e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009eb0 tx=0x7fbfc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:39.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc803e610 con 0x7fbfd01594d0 2023-08-25T09:57:39.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc803e790 con 0x7fbfd01594d0 2023-08-25T09:57:39.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc803eb40 con 0x7fbfd01594d0 2023-08-25T09:57:39.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.996+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd006fc00 con 0x7fbfd01594d0 2023-08-25T09:57:39.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.997+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd0157360 con 0x7fbfd01594d0 2023-08-25T09:57:39.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.998+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd0157e40 con 0x7fbfd01594d0 2023-08-25T09:57:39.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.999+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbfc803b050 con 0x7fbfd01594d0 2023-08-25T09:57:39.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:38.999+0000 7fbfc77fe640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbfb006a080 0x7fbfb006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:39.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.000+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7fbfc80ab370 con 0x7fbfd01594d0 2023-08-25T09:57:39.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.000+0000 7fbfceffd640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbfb006a080 0x7fbfb006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:39.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.000+0000 7fbfceffd640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbfb006a080 0x7fbfb006c540 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0007fd0 tx=0x7fbfc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:39.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.004+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbfd0157e40 con 0x7fbfd01594d0 2023-08-25T09:57:39.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.230+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfd0157e40 con 0x7fbfd01594d0 2023-08-25T09:57:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.231+0000 7fbfc77fe640 1 -- 172.21.15.71:0/60167519 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fbfd0157e40 con 0x7fbfd01594d0 2023-08-25T09:57:39.232 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:39.233 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:35.725682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2023-08-25T09:57:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbfb006a080 msgr2=0x7fbfb006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbfb006a080 0x7fbfb006c540 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0007fd0 tx=0x7fbfc0009040 comp rx=0 tx=0).stop 2023-08-25T09:57:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 msgr2=0x7fbfd00c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 0x7fbfd00c94e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009eb0 tx=0x7fbfc8004a40 comp rx=0 tx=0).stop 2023-08-25T09:57:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 shutdown_connections 2023-08-25T09:57:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbfb006a080 0x7fbfb006c540 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbfd01594d0 0x7fbfd00c94e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfd00c3940 0x7fbfd014a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.233+0000 7fbfd57e3640 1 --2- 172.21.15.71:0/60167519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfd006cc70 0x7fbfd0158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.234+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 >> 172.21.15.71:0/60167519 conn(0x7fbfd00654f0 msgr2=0x7fbfd014e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.234+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 shutdown_connections 2023-08-25T09:57:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:39.234+0000 7fbfd57e3640 1 -- 172.21.15.71:0/60167519 wait complete. 2023-08-25T09:57:39.253 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:41.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:41.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.476+0000 7f336359e640 1 Processor -- start 2023-08-25T09:57:41.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.476+0000 7f336359e640 1 -- start start 2023-08-25T09:57:41.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f33641532a0 0x7f336406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f33640c6df0 0x7f3364158f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336406e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336406f250 con 0x7f33640c6df0 2023-08-25T09:57:41.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336359e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f336415d350 con 0x7f33641532a0 2023-08-25T09:57:41.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3364159490 con 0x7f33640c7520 2023-08-25T09:57:41.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f33640c6df0 0x7f3364158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f33640c6df0 0x7f3364158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55766/0 (socket says 172.21.15.71:55766) 2023-08-25T09:57:41.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 -- 172.21.15.71:0/4019029250 learned_addr learned my addr 172.21.15.71:0/4019029250 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:41.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f3362d9d640 1 --2- 172.21.15.71:0/4019029250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f336259c640 1 --2- 172.21.15.71:0/4019029250 >> v2:172.21.15.164:3300/0 conn(0x7f33641532a0 0x7f336406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 -- 172.21.15.71:0/4019029250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 msgr2=0x7f336406e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 --2- 172.21.15.71:0/4019029250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336406e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 -- 172.21.15.71:0/4019029250 >> v2:172.21.15.164:3300/0 conn(0x7f33641532a0 msgr2=0x7f336406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.478+0000 7f335bfff640 1 --2- 172.21.15.71:0/4019029250 >> v2:172.21.15.164:3300/0 conn(0x7f33641532a0 0x7f336406ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.479+0000 7f335bfff640 1 -- 172.21.15.71:0/4019029250 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336406ef70 con 0x7f33640c6df0 2023-08-25T09:57:41.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.479+0000 7f336259c640 1 --2- 172.21.15.71:0/4019029250 >> v2:172.21.15.164:3300/0 conn(0x7f33641532a0 0x7f336406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:41.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.480+0000 7f335bfff640 1 --2- 172.21.15.71:0/4019029250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33640c6df0 0x7f3364158f50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3364071860 tx=0x7f334c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c04dba375cd7c56 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:41.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.480+0000 7f3361d9b640 1 -- 172.21.15.71:0/4019029250 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c030860 con 0x7f33640c6df0 2023-08-25T09:57:41.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.480+0000 7f3361d9b640 1 -- 172.21.15.71:0/4019029250 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c0309c0 con 0x7f33640c6df0 2023-08-25T09:57:41.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.480+0000 7f3361d9b640 1 -- 172.21.15.71:0/4019029250 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c030d50 con 0x7f33640c6df0 2023-08-25T09:57:41.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 -- 172.21.15.71:0/4019029250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33640c6df0 msgr2=0x7f3364158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 --2- 172.21.15.71:0/4019029250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33640c6df0 0x7f3364158f50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3364071860 tx=0x7f334c02b9a0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 -- 172.21.15.71:0/4019029250 shutdown_connections 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 --2- 172.21.15.71:0/4019029250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336406e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 --2- 172.21.15.71:0/4019029250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33640c6df0 0x7f3364158f50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 --2- 172.21.15.71:0/4019029250 >> v2:172.21.15.164:3300/0 conn(0x7f33641532a0 0x7f336406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 -- 172.21.15.71:0/4019029250 >> 172.21.15.71:0/4019029250 conn(0x7f33640654f0 msgr2=0x7f3364065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:41.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 -- 172.21.15.71:0/4019029250 shutdown_connections 2023-08-25T09:57:41.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.481+0000 7f336359e640 1 -- 172.21.15.71:0/4019029250 wait complete. 2023-08-25T09:57:41.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.483+0000 7f336359e640 1 Processor -- start 2023-08-25T09:57:41.488 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.483+0000 7f336359e640 1 -- start start 2023-08-25T09:57:41.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33640c6df0 0x7f3364151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336414f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 0x7f33641ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3364152410 con 0x7f33641532a0 2023-08-25T09:57:41.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33640bd400 con 0x7f33640c6df0 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33640bd6e0 con 0x7f33640c7520 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 0x7f33641ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 0x7f33641ef8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55774/0 (socket says 172.21.15.71:55774) 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 -- 172.21.15.71:0/1769270577 learned_addr learned my addr 172.21.15.71:0/1769270577 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f336259c640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33640c6df0 0x7f3364151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f335bfff640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336414f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 -- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 msgr2=0x7f336414f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336414f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 -- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33640c6df0 msgr2=0x7f3364151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33640c6df0 0x7f3364151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.485+0000 7f3362d9d640 1 -- 172.21.15.71:0/1769270577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0094d0 con 0x7f33641532a0 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.486+0000 7f3362d9d640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 0x7f33641ef8b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f335c00cb80 tx=0x7f335c00a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.486+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c00ce70 con 0x7f33641532a0 2023-08-25T09:57:41.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.486+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c00abd0 con 0x7f33641532a0 2023-08-25T09:57:41.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.486+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33640bd9c0 con 0x7f33641532a0 2023-08-25T09:57:41.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.486+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c011410 con 0x7f33641532a0 2023-08-25T09:57:41.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.486+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33640bdf80 con 0x7f33641532a0 2023-08-25T09:57:41.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.488+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336415cbf0 con 0x7f33641532a0 2023-08-25T09:57:41.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.488+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f335c011570 con 0x7f33641532a0 2023-08-25T09:57:41.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.490+0000 7f335affd640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f334406a080 0x7f334406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:41.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.490+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7f335c088b50 con 0x7f33641532a0 2023-08-25T09:57:41.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.490+0000 7f336259c640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f334406a080 0x7f334406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:41.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.491+0000 7f336259c640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f334406a080 0x7f334406c540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3354006fd0 tx=0x7f3354008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:41.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.494+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f335c053ae0 con 0x7f33641532a0 2023-08-25T09:57:41.725 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.723+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f336404e420 con 0x7f33641532a0 2023-08-25T09:57:41.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.725+0000 7f335affd640 1 -- 172.21.15.71:0/1769270577 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f335c01f090 con 0x7f33641532a0 2023-08-25T09:57:41.727 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:41.727 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:35.725682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:41.728 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2023-08-25T09:57:41.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.727+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f334406a080 msgr2=0x7f334406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:41.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.727+0000 7f336359e640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f334406a080 0x7f334406c540 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3354006fd0 tx=0x7f3354008040 comp rx=0 tx=0).stop 2023-08-25T09:57:41.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.727+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 msgr2=0x7f33641ef8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:41.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.727+0000 7f336359e640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 0x7f33641ef8b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f335c00cb80 tx=0x7f335c00a4c0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 shutdown_connections 2023-08-25T09:57:41.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f334406a080 0x7f334406c540 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33641532a0 0x7f33641ef8b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33640c7520 0x7f336414f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 --2- 172.21.15.71:0/1769270577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33640c6df0 0x7f3364151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:41.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 >> 172.21.15.71:0/1769270577 conn(0x7f33640654f0 msgr2=0x7f33640658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 shutdown_connections 2023-08-25T09:57:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:41.728+0000 7f336359e640 1 -- 172.21.15.71:0/1769270577 wait complete. 2023-08-25T09:57:41.748 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:43.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:43.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.971+0000 7fc61ac70640 1 Processor -- start 2023-08-25T09:57:43.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.971+0000 7fc61ac70640 1 -- start start 2023-08-25T09:57:43.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61ac70640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc61406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:43.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61ac70640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc614070cf0 0x7fc6141637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:43.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61ac70640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61406eb60 con 0x7fc614152d80 2023-08-25T09:57:43.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61ac70640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61406ca30 con 0x7fc614070cf0 2023-08-25T09:57:43.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61ac70640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61406cf90 con 0x7fc6140c4560 2023-08-25T09:57:43.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61920d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc614070cf0 0x7fc6141637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:43.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61920d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc614070cf0 0x7fc6141637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55784/0 (socket says 172.21.15.71:55784) 2023-08-25T09:57:43.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.973+0000 7fc61920d640 1 -- 172.21.15.71:0/2709770911 learned_addr learned my addr 172.21.15.71:0/2709770911 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:43.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.974+0000 7fc613fff640 1 --2- 172.21.15.71:0/2709770911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc61406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:43.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.974+0000 7fc61920d640 1 -- 172.21.15.71:0/2709770911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 msgr2=0x7fc61406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:43.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.974+0000 7fc61920d640 1 --2- 172.21.15.71:0/2709770911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc61406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:43.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.974+0000 7fc61920d640 1 -- 172.21.15.71:0/2709770911 >> v1:172.21.15.164:6789/0 conn(0x7fc614152d80 legacy=0x7fc6140c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:43.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.974+0000 7fc61920d640 1 -- 172.21.15.71:0/2709770911 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614158fd0 con 0x7fc614070cf0 2023-08-25T09:57:43.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.975+0000 7fc61920d640 1 --2- 172.21.15.71:0/2709770911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614070cf0 0x7fc6141637b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009e50 tx=0x7fc60c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb790ace61bbc83 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:43.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.975+0000 7fc6137fe640 1 -- 172.21.15.71:0/2709770911 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c02ca80 con 0x7fc614070cf0 2023-08-25T09:57:43.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.975+0000 7fc6137fe640 1 -- 172.21.15.71:0/2709770911 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c02cbe0 con 0x7fc614070cf0 2023-08-25T09:57:43.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.975+0000 7fc6137fe640 1 -- 172.21.15.71:0/2709770911 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0343f0 con 0x7fc614070cf0 2023-08-25T09:57:43.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 -- 172.21.15.71:0/2709770911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614070cf0 msgr2=0x7fc6141637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:43.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2709770911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614070cf0 0x7fc6141637b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009e50 tx=0x7fc60c02bee0 comp rx=0 tx=0).stop 2023-08-25T09:57:43.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 -- 172.21.15.71:0/2709770911 shutdown_connections 2023-08-25T09:57:43.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2709770911 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614070cf0 0x7fc6141637b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:43.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2709770911 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc61406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:43.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 -- 172.21.15.71:0/2709770911 >> 172.21.15.71:0/2709770911 conn(0x7fc6140654f0 msgr2=0x7fc614065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:43.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 -- 172.21.15.71:0/2709770911 shutdown_connections 2023-08-25T09:57:43.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.976+0000 7fc61ac70640 1 -- 172.21.15.71:0/2709770911 wait complete. 2023-08-25T09:57:43.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.978+0000 7fc61ac70640 1 Processor -- start 2023-08-25T09:57:43.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.978+0000 7fc61ac70640 1 -- start start 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61ac70640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc614070cf0 0x7fc61415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61ac70640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc6141f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61ac70640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 0x7fc61415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61ac70640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6141f1da0 con 0x7fc614152d80 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61ac70640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6141f2080 con 0x7fc614070cf0 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61ac70640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6141f2360 con 0x7fc6140c4560 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61920d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 0x7fc61415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:43.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61920d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 0x7fc61415c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55786/0 (socket says 172.21.15.71:55786) 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc61920d640 1 -- 172.21.15.71:0/2752684303 learned_addr learned my addr 172.21.15.71:0/2752684303 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.979+0000 7fc613fff640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc6141f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc618a0c640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc614070cf0 0x7fc61415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61920d640 1 -- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 msgr2=0x7fc6141f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61920d640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc6141f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61920d640 1 -- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc614070cf0 msgr2=0x7fc61415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61920d640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc614070cf0 0x7fc61415b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:43.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61920d640 1 -- 172.21.15.71:0/2752684303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c009b30 con 0x7fc614152d80 2023-08-25T09:57:43.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61920d640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 0x7fc61415c1b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009e50 tx=0x7fc60c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:43.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c009510 con 0x7fc614152d80 2023-08-25T09:57:43.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c03e450 con 0x7fc614152d80 2023-08-25T09:57:43.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6141f2640 con 0x7fc614152d80 2023-08-25T09:57:43.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.980+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c03e800 con 0x7fc614152d80 2023-08-25T09:57:43.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.981+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6140c1990 con 0x7fc614152d80 2023-08-25T09:57:43.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.982+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6140c3a40 con 0x7fc614152d80 2023-08-25T09:57:43.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.983+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc60c02cd50 con 0x7fc614152d80 2023-08-25T09:57:43.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.984+0000 7fc6117fa640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5f806a050 0x7fc5f806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:43.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.984+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7fc60c0ab8b0 con 0x7fc614152d80 2023-08-25T09:57:43.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.984+0000 7fc618a0c640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5f806a050 0x7fc5f806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:43.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.984+0000 7fc618a0c640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5f806a050 0x7fc5f806c510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc608000f60 tx=0x7fc608009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:43.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:43.988+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc60c0767b0 con 0x7fc614152d80 2023-08-25T09:57:44.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.212+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc614069160 con 0x7fc614152d80 2023-08-25T09:57:44.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.213+0000 7fc6117fa640 1 -- 172.21.15.71:0/2752684303 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fc60c079dd0 con 0x7fc614152d80 2023-08-25T09:57:44.215 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:44.215 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:35.725682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:44.216 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2023-08-25T09:57:44.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.215+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5f806a050 msgr2=0x7fc5f806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:44.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.215+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5f806a050 0x7fc5f806c510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc608000f60 tx=0x7fc608009040 comp rx=0 tx=0).stop 2023-08-25T09:57:44.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.215+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 msgr2=0x7fc61415c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:44.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.215+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 0x7fc61415c1b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009e50 tx=0x7fc60c004a40 comp rx=0 tx=0).stop 2023-08-25T09:57:44.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 shutdown_connections 2023-08-25T09:57:44.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5f806a050 0x7fc5f806c510 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:44.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc614152d80 0x7fc61415c1b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:44.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6140c4560 0x7fc6141f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:44.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 --2- 172.21.15.71:0/2752684303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc614070cf0 0x7fc61415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:44.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 >> 172.21.15.71:0/2752684303 conn(0x7fc6140654f0 msgr2=0x7fc61406efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:44.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 shutdown_connections 2023-08-25T09:57:44.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:44.216+0000 7fc61ac70640 1 -- 172.21.15.71:0/2752684303 wait complete. 2023-08-25T09:57:44.239 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:46.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:46.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.453+0000 7f359ffff640 1 Processor -- start 2023-08-25T09:57:46.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.454+0000 7f359ffff640 1 -- start start 2023-08-25T09:57:46.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.455+0000 7f359ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f35a0069850 0x7f35a01576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:46.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.455+0000 7f359ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a00c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:46.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.455+0000 7f359ffff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a0157bf0 con 0x7f35a01542d0 2023-08-25T09:57:46.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.455+0000 7f359ffff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a00c4b60 con 0x7f35a0069850 2023-08-25T09:57:46.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.455+0000 7f359ffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a006a130 con 0x7f35a00c4fb0 2023-08-25T09:57:46.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.455+0000 7f359effd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f35a01542d0 0x7f35a00c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39530/0 (socket says 172.21.15.71:39530) 2023-08-25T09:57:46.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f359effd640 1 -- 172.21.15.71:0/3656992313 learned_addr learned my addr 172.21.15.71:0/3656992313 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:57:46.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f3597fff640 1 --2- 172.21.15.71:0/3656992313 >> v2:172.21.15.71:3300/0 conn(0x7f35a0069850 0x7f35a01576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f359f7fe640 1 --2- 172.21.15.71:0/3656992313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a00c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f359e7fc640 1 -- 172.21.15.71:0/3656992313 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720287456 0 0) 0x7f3590001110 con 0x7f35a01542d0 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f359e7fc640 1 -- 172.21.15.71:0/3656992313 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c003520 con 0x7f35a01542d0 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f3597fff640 1 -- 172.21.15.71:0/3656992313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 msgr2=0x7f35a00c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f3597fff640 1 --2- 172.21.15.71:0/3656992313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a00c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f3597fff640 1 -- 172.21.15.71:0/3656992313 >> v1:172.21.15.71:6789/0 conn(0x7f35a01542d0 legacy=0x7f35a00c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:46.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.456+0000 7f3597fff640 1 -- 172.21.15.71:0/3656992313 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00c5980 con 0x7f35a0069850 2023-08-25T09:57:46.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.457+0000 7f3597fff640 1 --2- 172.21.15.71:0/3656992313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a0069850 0x7f35a01576b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f35880097a0 tx=0x7f358802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef9b5aabc2565db server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359e7fc640 1 -- 172.21.15.71:0/3656992313 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358802d040 con 0x7f35a0069850 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359e7fc640 1 -- 172.21.15.71:0/3656992313 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3588031810 con 0x7f35a0069850 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359e7fc640 1 -- 172.21.15.71:0/3656992313 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588031ba0 con 0x7f35a0069850 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 -- 172.21.15.71:0/3656992313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a0069850 msgr2=0x7f35a01576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 --2- 172.21.15.71:0/3656992313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a0069850 0x7f35a01576b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f35880097a0 tx=0x7f358802b8b0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 -- 172.21.15.71:0/3656992313 shutdown_connections 2023-08-25T09:57:46.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 --2- 172.21.15.71:0/3656992313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a00c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 --2- 172.21.15.71:0/3656992313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a0069850 0x7f35a01576b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 -- 172.21.15.71:0/3656992313 >> 172.21.15.71:0/3656992313 conn(0x7f35a006c280 msgr2=0x7f35a006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:46.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 -- 172.21.15.71:0/3656992313 shutdown_connections 2023-08-25T09:57:46.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.458+0000 7f359ffff640 1 -- 172.21.15.71:0/3656992313 wait complete. 2023-08-25T09:57:46.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.460+0000 7f359ffff640 1 Processor -- start 2023-08-25T09:57:46.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.460+0000 7f359ffff640 1 -- start start 2023-08-25T09:57:46.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.461+0000 7f359ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a0069850 0x7f35a014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:46.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.461+0000 7f359ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a0149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:46.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.461+0000 7f359ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 0x7f35a00c0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:46.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.461+0000 7f359ffff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c9130 con 0x7f35a01542d0 2023-08-25T09:57:46.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.461+0000 7f359ffff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c3a90 con 0x7f35a0069850 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.461+0000 7f359ffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c8e50 con 0x7f35a00c4fb0 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 0x7f35a00c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 0x7f35a00c0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55808/0 (socket says 172.21.15.71:55808) 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 -- 172.21.15.71:0/1374148049 learned_addr learned my addr 172.21.15.71:0/1374148049 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f3597fff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a0149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359effd640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a0069850 0x7f35a014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:46.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 -- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 msgr2=0x7f35a0149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a0149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 -- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a0069850 msgr2=0x7f35a014ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a0069850 0x7f35a014ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 -- 172.21.15.71:0/1374148049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588009400 con 0x7f35a01542d0 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.462+0000 7f359f7fe640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 0x7f35a00c0190 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3598007e40 tx=0x7f359800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.463+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598014620 con 0x7f35a01542d0 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.463+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359800ee90 con 0x7f35a01542d0 2023-08-25T09:57:46.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.463+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35980149e0 con 0x7f35a01542d0 2023-08-25T09:57:46.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.463+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a014c650 con 0x7f35a01542d0 2023-08-25T09:57:46.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.463+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a014b310 con 0x7f35a01542d0 2023-08-25T09:57:46.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.464+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a014c4b0 con 0x7f35a01542d0 2023-08-25T09:57:46.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.465+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3598014b40 con 0x7f35a01542d0 2023-08-25T09:57:46.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.466+0000 7f35977fe640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f358006a080 0x7f358006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:46.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.466+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7f3598089750 con 0x7f35a01542d0 2023-08-25T09:57:46.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.466+0000 7f359effd640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f358006a080 0x7f358006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:46.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.467+0000 7f359effd640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f358006a080 0x7f358006c540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3590000f60 tx=0x7f3590009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:46.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.471+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f35a014c4b0 con 0x7f35a01542d0 2023-08-25T09:57:46.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.700+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a014cb70 con 0x7f35a01542d0 2023-08-25T09:57:46.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.701+0000 7f35977fe640 1 -- 172.21.15.71:0/1374148049 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6538 (secure 0 0 0) 0x7f35a014cb70 con 0x7f35a01542d0 2023-08-25T09:57:46.703 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:46.703 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:35.725682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:46.704 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2023-08-25T09:57:46.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.704+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f358006a080 msgr2=0x7f358006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:46.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.704+0000 7f359ffff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f358006a080 0x7f358006c540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3590000f60 tx=0x7f3590009040 comp rx=0 tx=0).stop 2023-08-25T09:57:46.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.704+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 msgr2=0x7f35a00c0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.704+0000 7f359ffff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 0x7f35a00c0190 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3598007e40 tx=0x7f359800e4b0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 shutdown_connections 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f358006a080 0x7f358006c540 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f35a01542d0 0x7f35a00c0190 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35a00c4fb0 0x7f35a0149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 --2- 172.21.15.71:0/1374148049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35a0069850 0x7f35a014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 >> 172.21.15.71:0/1374148049 conn(0x7f35a006c280 msgr2=0x7f35a014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:46.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 shutdown_connections 2023-08-25T09:57:46.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:46.705+0000 7f359ffff640 1 -- 172.21.15.71:0/1374148049 wait complete. 2023-08-25T09:57:46.725 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:48.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:48.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.943+0000 7fe53eda6640 1 Processor -- start 2023-08-25T09:57:48.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.943+0000 7fe53eda6640 1 -- start start 2023-08-25T09:57:48.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.944+0000 7fe53eda6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe5380c3a40 0x7fe5380c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:48.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.944+0000 7fe53eda6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe5381509d0 0x7fe538150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:48.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53eda6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5380c1dc0 con 0x7fe5380c3660 2023-08-25T09:57:48.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53eda6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe538157a80 con 0x7fe5381509d0 2023-08-25T09:57:48.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53eda6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe538151320 con 0x7fe5380c3a40 2023-08-25T09:57:48.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe5381509d0 0x7fe538150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:48.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe5381509d0 0x7fe538150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55822/0 (socket says 172.21.15.71:55822) 2023-08-25T09:57:48.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 -- 172.21.15.71:0/174202744 learned_addr learned my addr 172.21.15.71:0/174202744 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:48.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe537fff640 1 --2- 172.21.15.71:0/174202744 >> v2:172.21.15.164:3300/0 conn(0x7fe5380c3a40 0x7fe5380c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:48.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 -- 172.21.15.71:0/174202744 >> v2:172.21.15.164:3300/0 conn(0x7fe5380c3a40 msgr2=0x7fe5380c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:48.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 --2- 172.21.15.71:0/174202744 >> v2:172.21.15.164:3300/0 conn(0x7fe5380c3a40 0x7fe5380c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:48.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 -- 172.21.15.71:0/174202744 >> v1:172.21.15.164:6789/0 conn(0x7fe5380c3660 legacy=0x7fe53806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:48.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.945+0000 7fe53d343640 1 -- 172.21.15.71:0/174202744 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538156e00 con 0x7fe5381509d0 2023-08-25T09:57:48.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.946+0000 7fe53d343640 1 --2- 172.21.15.71:0/174202744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe538150de0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe538067520 tx=0x7fe53002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1913b3bb54ab4213 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:48.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.947+0000 7fe5377fe640 1 -- 172.21.15.71:0/174202744 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53002cb20 con 0x7fe5381509d0 2023-08-25T09:57:48.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.947+0000 7fe5377fe640 1 -- 172.21.15.71:0/174202744 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53002cc80 con 0x7fe5381509d0 2023-08-25T09:57:48.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.947+0000 7fe5377fe640 1 -- 172.21.15.71:0/174202744 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5300343f0 con 0x7fe5381509d0 2023-08-25T09:57:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.947+0000 7fe53eda6640 1 -- 172.21.15.71:0/174202744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 msgr2=0x7fe538150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.947+0000 7fe53eda6640 1 --2- 172.21.15.71:0/174202744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe538150de0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe538067520 tx=0x7fe53002bf80 comp rx=0 tx=0).stop 2023-08-25T09:57:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.948+0000 7fe53eda6640 1 -- 172.21.15.71:0/174202744 shutdown_connections 2023-08-25T09:57:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.948+0000 7fe53eda6640 1 --2- 172.21.15.71:0/174202744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe538150de0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:48.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.948+0000 7fe53eda6640 1 --2- 172.21.15.71:0/174202744 >> v2:172.21.15.164:3300/0 conn(0x7fe5380c3a40 0x7fe5380c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.948+0000 7fe53eda6640 1 -- 172.21.15.71:0/174202744 >> 172.21.15.71:0/174202744 conn(0x7fe5380654f0 msgr2=0x7fe538065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.948+0000 7fe53eda6640 1 -- 172.21.15.71:0/174202744 shutdown_connections 2023-08-25T09:57:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.948+0000 7fe53eda6640 1 -- 172.21.15.71:0/174202744 wait complete. 2023-08-25T09:57:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.949+0000 7fe53eda6640 1 Processor -- start 2023-08-25T09:57:48.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 -- start start 2023-08-25T09:57:48.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5380c3660 0x7fe538151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5380c3a40 0x7fe538158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe53814e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:48.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe538158ed0 con 0x7fe5381509d0 2023-08-25T09:57:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53806f8d0 con 0x7fe5380c3a40 2023-08-25T09:57:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.950+0000 7fe53eda6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53806f000 con 0x7fe5380c3660 2023-08-25T09:57:48.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe53814e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe53814e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55828/0 (socket says 172.21.15.71:55828) 2023-08-25T09:57:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 -- 172.21.15.71:0/2091105679 learned_addr learned my addr 172.21.15.71:0/2091105679 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe537fff640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5380c3a40 0x7fe538158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53cb42640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5380c3660 0x7fe538151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 -- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5380c3660 msgr2=0x7fe538151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5380c3660 0x7fe538151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 -- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5380c3a40 msgr2=0x7fe538158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5380c3a40 0x7fe538158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.951+0000 7fe53d343640 1 -- 172.21.15.71:0/2091105679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe530009b30 con 0x7fe5381509d0 2023-08-25T09:57:48.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.952+0000 7fe53d343640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe53814e310 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe530009f40 tx=0x7fe530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:48.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.952+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530009510 con 0x7fe5381509d0 2023-08-25T09:57:48.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.952+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe530007e50 con 0x7fe5381509d0 2023-08-25T09:57:48.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.952+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53003e670 con 0x7fe5381509d0 2023-08-25T09:57:48.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.952+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53815a920 con 0x7fe5381509d0 2023-08-25T09:57:48.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.952+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53815aec0 con 0x7fe5381509d0 2023-08-25T09:57:48.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.953+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53814bb10 con 0x7fe5381509d0 2023-08-25T09:57:48.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.954+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe53003e7d0 con 0x7fe5381509d0 2023-08-25T09:57:48.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.955+0000 7fe5357fa640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe51c06a080 0x7fe51c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:48.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.956+0000 7fe53cb42640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe51c06a080 0x7fe51c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:48.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.956+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7925+0+0 (secure 0 0 0) 0x7fe53007a510 con 0x7fe5381509d0 2023-08-25T09:57:48.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.956+0000 7fe53cb42640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe51c06a080 0x7fe51c06c540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe52c006fd0 tx=0x7fe52c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:48.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:48.961+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe5300761c0 con 0x7fe5381509d0 2023-08-25T09:57:49.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.186+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5380c29c0 con 0x7fe5381509d0 2023-08-25T09:57:49.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.187+0000 7fe5357fa640 1 -- 172.21.15.71:0/2091105679 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+6538 (secure 0 0 0) 0x7fe530043090 con 0x7fe5381509d0 2023-08-25T09:57:49.189 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:49.189 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5547,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:35.725682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":4290,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":2,"incarnation":12,"state":"up:active","state_seq":13,"addr":"172.21.15.71:6837/3404832976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3404832976},{"type":"v1","addr":"172.21.15.71:6837","nonce":3404832976}]},"join_fscid":-1,"export_targets":[1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:49.190 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 62 2023-08-25T09:57:49.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe51c06a080 msgr2=0x7fe51c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:49.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe51c06a080 0x7fe51c06c540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe52c006fd0 tx=0x7fe52c008040 comp rx=0 tx=0).stop 2023-08-25T09:57:49.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 msgr2=0x7fe53814e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:49.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe53814e310 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe530009f40 tx=0x7fe530004a40 comp rx=0 tx=0).stop 2023-08-25T09:57:49.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 shutdown_connections 2023-08-25T09:57:49.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5380c3a40 0x7fe538158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:49.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe51c06a080 0x7fe51c06c540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:49.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe5381509d0 0x7fe53814e310 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:49.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 --2- 172.21.15.71:0/2091105679 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5380c3660 0x7fe538151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:49.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 >> 172.21.15.71:0/2091105679 conn(0x7fe5380654f0 msgr2=0x7fe5380c9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:49.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 shutdown_connections 2023-08-25T09:57:49.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:49.189+0000 7fe53eda6640 1 -- 172.21.15.71:0/2091105679 wait complete. 2023-08-25T09:57:49.210 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-08-25T09:57:51.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:51.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.436+0000 7fde4b014640 1 Processor -- start 2023-08-25T09:57:51.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.436+0000 7fde4b014640 1 -- start start 2023-08-25T09:57:51.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.438+0000 7fde4b014640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 0x7fde44066130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.438+0000 7fde4b014640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde440c27e0 con 0x7fde4406b940 2023-08-25T09:57:51.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.438+0000 7fde4b014640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4406ac10 con 0x7fde441511c0 2023-08-25T09:57:51.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.438+0000 7fde4b014640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4406aef0 con 0x7fde4406bd20 2023-08-25T09:57:51.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.438+0000 7fde48db0640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fde4406b940 0x7fde44150ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41704/0 (socket says 172.21.15.71:41704) 2023-08-25T09:57:51.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.438+0000 7fde48db0640 1 -- 172.21.15.71:0/499589697 learned_addr learned my addr 172.21.15.71:0/499589697 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:57:51.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde495b1640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 0x7fde44066130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836845287 0 0) 0x7fde440c27e0 con 0x7fde4406b940 2023-08-25T09:57:51.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde2c003520 con 0x7fde4406b940 2023-08-25T09:57:51.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900373873 0 0) 0x7fde2c003520 con 0x7fde4406b940 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 msgr2=0x7fde44066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 0x7fde44066130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 >> v1:172.21.15.164:6789/0 conn(0x7fde441511c0 legacy=0x7fde4415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4415aca0 con 0x7fde4406b940 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fde40001670 con 0x7fde4406b940 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 >> v1:172.21.15.71:6789/0 conn(0x7fde4406b940 legacy=0x7fde44150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde2c003a10 0x7fde2c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0047c0 0x7fde2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 0x7fde2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4406aef0 con 0x7fde2c003a10 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c00ba80 con 0x7fde2c0047c0 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.439+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c00bc40 con 0x7fde4406bd20 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde48db0640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde2c003a10 0x7fde2c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde48db0640 1 -- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 msgr2=0x7fde2c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:51.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde495b1640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 0x7fde2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde48db0640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde4406bd20 0x7fde2c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde48db0640 1 -- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0047c0 msgr2=0x7fde2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:51.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde48db0640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c0047c0 0x7fde2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.440+0000 7fde48db0640 1 -- 172.21.15.71:0/499589697 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c003520 con 0x7fde2c003a10 2023-08-25T09:57:51.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.441+0000 7fde48db0640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde2c003a10 0x7fde2c003e30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fde4000c240 tx=0x7fde2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f2f866e55a91f7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:51.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.441+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde40005ad0 con 0x7fde2c003a10 2023-08-25T09:57:51.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.441+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde40005c30 con 0x7fde2c003a10 2023-08-25T09:57:51.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.441+0000 7fde3b7fe640 1 -- 172.21.15.71:0/499589697 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4002c8b0 con 0x7fde2c003a10 2023-08-25T09:57:51.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde4b014640 1 -- 172.21.15.71:0/499589697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde2c003a10 msgr2=0x7fde2c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde4b014640 1 --2- 172.21.15.71:0/499589697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde2c003a10 0x7fde2c003e30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fde4000c240 tx=0x7fde2c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde495b1640 1 -- 172.21.15.71:0/499589697 reap_dead start 2023-08-25T09:57:51.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde4b014640 1 -- 172.21.15.71:0/499589697 shutdown_connections 2023-08-25T09:57:51.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde4b014640 1 -- 172.21.15.71:0/499589697 >> 172.21.15.71:0/499589697 conn(0x7fde440654f0 msgr2=0x7fde44065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:51.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde4b014640 1 -- 172.21.15.71:0/499589697 shutdown_connections 2023-08-25T09:57:51.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.442+0000 7fde4b014640 1 -- 172.21.15.71:0/499589697 wait complete. 2023-08-25T09:57:51.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.444+0000 7fde4b014640 1 Processor -- start 2023-08-25T09:57:51.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.444+0000 7fde4b014640 1 -- start start 2023-08-25T09:57:51.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde4b014640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c003a10 0x7fde441f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde4b014640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde44159de0 0x7fde440c0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde4b014640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 0x7fde44156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde4b014640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde440ba1d0 con 0x7fde44065ea0 2023-08-25T09:57:51.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde4b014640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde440b95d0 con 0x7fde2c003a10 2023-08-25T09:57:51.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde4b014640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde44157650 con 0x7fde44159de0 2023-08-25T09:57:51.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde495b1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 0x7fde44156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde495b1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 0x7fde44156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36350/0 (socket says 172.21.15.71:36350) 2023-08-25T09:57:51.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde495b1640 1 -- 172.21.15.71:0/3029993792 learned_addr learned my addr 172.21.15.71:0/3029993792 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:51.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.445+0000 7fde3bfff640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde44159de0 0x7fde440c0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde48db0640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c003a10 0x7fde441f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde495b1640 1 -- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde44159de0 msgr2=0x7fde440c0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde495b1640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde44159de0 0x7fde440c0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde495b1640 1 -- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c003a10 msgr2=0x7fde441f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde495b1640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c003a10 0x7fde441f2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde495b1640 1 -- 172.21.15.71:0/3029993792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde440c14f0 con 0x7fde44065ea0 2023-08-25T09:57:51.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde3bfff640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde44159de0 0x7fde440c0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:51.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde495b1640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 0x7fde44156a50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fde440c7d90 tx=0x7fde3c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:51.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c017400 con 0x7fde44065ea0 2023-08-25T09:57:51.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3c011040 con 0x7fde44065ea0 2023-08-25T09:57:51.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde440c1a60 con 0x7fde44065ea0 2023-08-25T09:57:51.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3c0177c0 con 0x7fde44065ea0 2023-08-25T09:57:51.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.446+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde441578a0 con 0x7fde44065ea0 2023-08-25T09:57:51.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.447+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde44069160 con 0x7fde44065ea0 2023-08-25T09:57:51.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.448+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fde3c017920 con 0x7fde44065ea0 2023-08-25T09:57:51.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.449+0000 7fde397fa640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde3006a080 0x7fde3006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.449+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fde3c088e80 con 0x7fde44065ea0 2023-08-25T09:57:51.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.450+0000 7fde48db0640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde3006a080 0x7fde3006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.450+0000 7fde48db0640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde3006a080 0x7fde3006c540 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde40034000 tx=0x7fde40009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:51.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.454+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fde3c054050 con 0x7fde44065ea0 2023-08-25T09:57:51.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.682+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde440c6f90 con 0x7fde44065ea0 2023-08-25T09:57:51.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.684+0000 7fde397fa640 1 -- 172.21.15.71:0/3029993792 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5744 (secure 0 0 0) 0x7fde3c057670 con 0x7fde44065ea0 2023-08-25T09:57:51.686 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:51.687 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:50.703611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:51.687 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 63 2023-08-25T09:57:51.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde3006a080 msgr2=0x7fde3006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde3006a080 0x7fde3006c540 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde40034000 tx=0x7fde40009030 comp rx=0 tx=0).stop 2023-08-25T09:57:51.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 msgr2=0x7fde44156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 0x7fde44156a50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fde440c7d90 tx=0x7fde3c00f4a0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 shutdown_connections 2023-08-25T09:57:51.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fde3006a080 0x7fde3006c540 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde44065ea0 0x7fde44156a50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde44159de0 0x7fde440c0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 --2- 172.21.15.71:0/3029993792 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde2c003a10 0x7fde441f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 >> 172.21.15.71:0/3029993792 conn(0x7fde440654f0 msgr2=0x7fde440c92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 shutdown_connections 2023-08-25T09:57:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.687+0000 7fde4b014640 1 -- 172.21.15.71:0/3029993792 wait complete. 2023-08-25T09:57:51.706 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-08-25T09:57:51.707 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-08-25T09:57:51.707 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:51.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.915+0000 7f544dee0640 1 Processor -- start 2023-08-25T09:57:51.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.915+0000 7f544dee0640 1 -- start start 2023-08-25T09:57:51.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f544dee0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f54480c3a40 0x7f54480c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f544dee0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480c1dc0 con 0x7f54481509d0 2023-08-25T09:57:51.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f544dee0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5448156e00 con 0x7f54480c3660 2023-08-25T09:57:51.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f544dee0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5448157a80 con 0x7f54480c3a40 2023-08-25T09:57:51.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f5446ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f54480c3a40 0x7f54480c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f5447fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f54481509d0 0x7f5448150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41718/0 (socket says 172.21.15.71:41718) 2023-08-25T09:57:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.917+0000 7f5447fff640 1 -- 172.21.15.71:0/3426681698 learned_addr learned my addr 172.21.15.71:0/3426681698 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:57:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.918+0000 7f5446ffd640 1 -- 172.21.15.71:0/3426681698 >> v1:172.21.15.164:6789/0 conn(0x7f54480c3660 legacy=0x7f544806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.918+0000 7f5446ffd640 1 -- 172.21.15.71:0/3426681698 >> v1:172.21.15.71:6789/0 conn(0x7f54481509d0 legacy=0x7f5448150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.918+0000 7f5446ffd640 1 -- 172.21.15.71:0/3426681698 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54481570e0 con 0x7f54480c3a40 2023-08-25T09:57:51.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.918+0000 7f54467fc640 1 -- 172.21.15.71:0/3426681698 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120868548 0 0) 0x7f54480c1dc0 con 0x7f54481509d0 2023-08-25T09:57:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.919+0000 7f5446ffd640 1 --2- 172.21.15.71:0/3426681698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3a40 0x7f54480c1710 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f543c0097a0 tx=0x7f543c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a7508f4098caf9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.919+0000 7f54467fc640 1 -- 172.21.15.71:0/3426681698 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c030610 con 0x7f54480c3a40 2023-08-25T09:57:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.919+0000 7f54467fc640 1 -- 172.21.15.71:0/3426681698 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c030770 con 0x7f54480c3a40 2023-08-25T09:57:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.919+0000 7f54467fc640 1 -- 172.21.15.71:0/3426681698 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c030b00 con 0x7f54480c3a40 2023-08-25T09:57:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 -- 172.21.15.71:0/3426681698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3a40 msgr2=0x7f54480c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 --2- 172.21.15.71:0/3426681698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3a40 0x7f54480c1710 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f543c0097a0 tx=0x7f543c02b960 comp rx=0 tx=0).stop 2023-08-25T09:57:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 -- 172.21.15.71:0/3426681698 shutdown_connections 2023-08-25T09:57:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 --2- 172.21.15.71:0/3426681698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3a40 0x7f54480c1710 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 -- 172.21.15.71:0/3426681698 >> 172.21.15.71:0/3426681698 conn(0x7f54480654f0 msgr2=0x7f5448065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 -- 172.21.15.71:0/3426681698 shutdown_connections 2023-08-25T09:57:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.920+0000 7f544dee0640 1 -- 172.21.15.71:0/3426681698 wait complete. 2023-08-25T09:57:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.922+0000 7f544dee0640 1 Processor -- start 2023-08-25T09:57:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.922+0000 7f544dee0640 1 -- start start 2023-08-25T09:57:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f544dee0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 0x7f5448159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f544dee0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54480c3a40 0x7f5448159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f544dee0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54481509d0 0x7f54481f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f544dee0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544815afa0 con 0x7f54480c3660 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f544dee0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5448152f70 con 0x7f54481509d0 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f544dee0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54480bc3a0 con 0x7f54480c3a40 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 0x7f5448159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 0x7f5448159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36380/0 (socket says 172.21.15.71:36380) 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 -- 172.21.15.71:0/1929812636 learned_addr learned my addr 172.21.15.71:0/1929812636 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f5446ffd640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54480c3a40 0x7f5448159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 -- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54480c3a40 msgr2=0x7f5448159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54480c3a40 0x7f5448159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 -- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54481509d0 msgr2=0x7f54481f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54481509d0 0x7f54481f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f54477fe640 1 -- 172.21.15.71:0/1929812636 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c009400 con 0x7f54480c3660 2023-08-25T09:57:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.923+0000 7f5446ffd640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54480c3a40 0x7f5448159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.924+0000 7f54477fe640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 0x7f5448159050 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f543800b3f0 tx=0x7f543800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.924+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438011830 con 0x7f54480c3660 2023-08-25T09:57:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.924+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5438011990 con 0x7f54480c3660 2023-08-25T09:57:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.924+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5448153460 con 0x7f54480c3660 2023-08-25T09:57:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.924+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438011d40 con 0x7f54480c3660 2023-08-25T09:57:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.924+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54480c6710 con 0x7f54480c3660 2023-08-25T09:57:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.925+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54480c4e80 con 0x7f54480c3660 2023-08-25T09:57:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.926+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f543801a400 con 0x7f54480c3660 2023-08-25T09:57:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.927+0000 7f542ffff640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a050 0x7f543406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.927+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f5438088850 con 0x7f54480c3660 2023-08-25T09:57:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.927+0000 7f5446ffd640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a050 0x7f543406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.928+0000 7f5446ffd640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a050 0x7f543406c510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f543c036000 tx=0x7f543c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:51.932+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5438053ab0 con 0x7f54480c3660 2023-08-25T09:57:52.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.161+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5448069160 con 0x7f54480c3660 2023-08-25T09:57:52.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.162+0000 7f542ffff640 1 -- 172.21.15.71:0/1929812636 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f54380570d0 con 0x7f54480c3660 2023-08-25T09:57:52.165 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:52.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:50.703611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:52.165 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 63 2023-08-25T09:57:52.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a050 msgr2=0x7f543406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:52.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a050 0x7f543406c510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f543c036000 tx=0x7f543c03b040 comp rx=0 tx=0).stop 2023-08-25T09:57:52.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 msgr2=0x7f5448159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:52.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 0x7f5448159050 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f543800b3f0 tx=0x7f543800b8b0 comp rx=0 tx=0).stop 2023-08-25T09:57:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 shutdown_connections 2023-08-25T09:57:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54481509d0 0x7f54481f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a050 0x7f543406c510 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f54480c3a40 0x7f5448159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 --2- 172.21.15.71:0/1929812636 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480c3660 0x7f5448159050 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 >> 172.21.15.71:0/1929812636 conn(0x7f54480654f0 msgr2=0x7f54480c9460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:52.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 shutdown_connections 2023-08-25T09:57:52.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:52.165+0000 7f544dee0640 1 -- 172.21.15.71:0/1929812636 wait complete. 2023-08-25T09:57:52.184 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:57:50.703611+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [] } 2023-08-25T09:57:54.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:54.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.408+0000 7f5ef3fff640 1 Processor -- start 2023-08-25T09:57:54.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.408+0000 7f5ef3fff640 1 -- start start 2023-08-25T09:57:54.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.409+0000 7f5ef3fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5ef40af290 0x7f5ef406b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:54.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.409+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef415c6d0 0x7f5ef415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:54.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.409+0000 7f5ef3fff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef40c0480 con 0x7f5ef4071760 2023-08-25T09:57:54.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef3fff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef40bf500 con 0x7f5ef40af290 2023-08-25T09:57:54.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef3fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4069850 con 0x7f5ef415c6d0 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5ef40af290 0x7f5ef406b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5ef40af290 0x7f5ef406b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36386/0 (socket says 172.21.15.71:36386) 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/3528998933 learned_addr learned my addr 172.21.15.71:0/3528998933 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef37fe640 1 --2- 172.21.15.71:0/3528998933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef415c6d0 0x7f5ef415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/3528998933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef415c6d0 msgr2=0x7f5ef415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 --2- 172.21.15.71:0/3528998933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef415c6d0 0x7f5ef415cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/3528998933 >> v1:172.21.15.164:6789/0 conn(0x7f5ef4071760 legacy=0x7f5ef406a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.410+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/3528998933 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef415d060 con 0x7f5ef40af290 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.411+0000 7f5ef2ffd640 1 --2- 172.21.15.71:0/3528998933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef40af290 0x7f5ef406b300 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009e50 tx=0x7f5ee402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc9c4bae7ff3f45 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef1ffb640 1 -- 172.21.15.71:0/3528998933 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee402ca70 con 0x7f5ef40af290 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef1ffb640 1 -- 172.21.15.71:0/3528998933 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee402cbd0 con 0x7f5ef40af290 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef1ffb640 1 -- 172.21.15.71:0/3528998933 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40343f0 con 0x7f5ef40af290 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 -- 172.21.15.71:0/3528998933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef40af290 msgr2=0x7f5ef406b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:54.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/3528998933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef40af290 0x7f5ef406b300 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009e50 tx=0x7f5ee402bed0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 -- 172.21.15.71:0/3528998933 shutdown_connections 2023-08-25T09:57:54.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/3528998933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef415c6d0 0x7f5ef415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/3528998933 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef40af290 0x7f5ef406b300 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 -- 172.21.15.71:0/3528998933 >> 172.21.15.71:0/3528998933 conn(0x7f5ef406c280 msgr2=0x7f5ef406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:54.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 -- 172.21.15.71:0/3528998933 shutdown_connections 2023-08-25T09:57:54.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.412+0000 7f5ef3fff640 1 -- 172.21.15.71:0/3528998933 wait complete. 2023-08-25T09:57:54.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.414+0000 7f5ef3fff640 1 Processor -- start 2023-08-25T09:57:54.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.414+0000 7f5ef3fff640 1 -- start start 2023-08-25T09:57:54.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 0x7f5ef40c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef40af290 0x7f5ef406ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ef415c6d0 0x7f5ef406f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef3fff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40c3b80 con 0x7f5ef4071760 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef3fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40c1fb0 con 0x7f5ef415c6d0 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef3fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40c45a0 con 0x7f5ef40af290 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 0x7f5ef40c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 0x7f5ef40c1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36394/0 (socket says 172.21.15.71:36394) 2023-08-25T09:57:54.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.415+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/2755612775 learned_addr learned my addr 172.21.15.71:0/2755612775 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef27fc640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef40af290 0x7f5ef406ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef40af290 msgr2=0x7f5ef406ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef2ffd640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef40af290 0x7f5ef406ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ef415c6d0 msgr2=0x7f5ef406f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef2ffd640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ef415c6d0 0x7f5ef406f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef2ffd640 1 -- 172.21.15.71:0/2755612775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4009b30 con 0x7f5ef4071760 2023-08-25T09:57:54.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5ef2ffd640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 0x7f5ef40c1a70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009e50 tx=0x7f5ee4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.416+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee403e450 con 0x7f5ef4071760 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.417+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee403e5f0 con 0x7f5ef4071760 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.417+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee403e9a0 con 0x7f5ef4071760 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.417+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef40c2f00 con 0x7f5ef4071760 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.417+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef40c34c0 con 0x7f5ef4071760 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.418+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef40c3cc0 con 0x7f5ef4071760 2023-08-25T09:57:54.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.419+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5ee403eb00 con 0x7f5ef4071760 2023-08-25T09:57:54.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.419+0000 7f5edb7fe640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5ee006a050 0x7f5ee006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:54.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.420+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f5ee40abb00 con 0x7f5ef4071760 2023-08-25T09:57:54.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.420+0000 7f5ef27fc640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5ee006a050 0x7f5ee006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:54.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.420+0000 7f5ef27fc640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5ee006a050 0x7f5ee006c510 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5ef406b5e0 tx=0x7f5ee8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:54.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.424+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5ef40c3cc0 con 0x7f5ef4071760 2023-08-25T09:57:54.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:57:54.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:57:54.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.654+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ef40c3cc0 con 0x7f5ef4071760 2023-08-25T09:57:54.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.655+0000 7f5edb7fe640 1 -- 172.21.15.71:0/2755612775 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5744 (secure 0 0 0) 0x7f5ef40c3cc0 con 0x7f5ef4071760 2023-08-25T09:57:54.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:54.657 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:50.703611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:54.657 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 63 2023-08-25T09:57:54.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5ee006a050 msgr2=0x7f5ee006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:54.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5ee006a050 0x7f5ee006c510 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5ef406b5e0 tx=0x7f5ee8009040 comp rx=0 tx=0).stop 2023-08-25T09:57:54.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 msgr2=0x7f5ef40c1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:54.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 0x7f5ef40c1a70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009e50 tx=0x7f5ee4004a40 comp rx=0 tx=0).stop 2023-08-25T09:57:54.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 shutdown_connections 2023-08-25T09:57:54.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5ef415c6d0 0x7f5ef406f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5ee006a050 0x7f5ee006c510 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5ef40af290 0x7f5ef406ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 --2- 172.21.15.71:0/2755612775 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5ef4071760 0x7f5ef40c1a70 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:54.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 >> 172.21.15.71:0/2755612775 conn(0x7f5ef406c280 msgr2=0x7f5ef406a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:54.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 shutdown_connections 2023-08-25T09:57:54.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:54.658+0000 7f5ef3fff640 1 -- 172.21.15.71:0/2755612775 wait complete. 2023-08-25T09:57:54.677 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:57:56.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:56.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.901+0000 7f641dd14640 1 Processor -- start 2023-08-25T09:57:56.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.901+0000 7f641dd14640 1 -- start start 2023-08-25T09:57:56.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641dd14640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f64180c72f0 0x7f64180acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:56.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641dd14640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f64180c76d0 0x7f6418152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:56.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641dd14640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64180c1060 con 0x7f64180ad2f0 2023-08-25T09:57:56.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641dd14640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64180adf70 con 0x7f64180c72f0 2023-08-25T09:57:56.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641dd14640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64180af730 con 0x7f64180c76d0 2023-08-25T09:57:56.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641cd12640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f64180c72f0 0x7f64180acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:56.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641cd12640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f64180c72f0 0x7f64180acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36402/0 (socket says 172.21.15.71:36402) 2023-08-25T09:57:56.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.903+0000 7f641cd12640 1 -- 172.21.15.71:0/3026891336 learned_addr learned my addr 172.21.15.71:0/3026891336 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:56.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.904+0000 7f640ffff640 1 --2- 172.21.15.71:0/3026891336 >> v2:172.21.15.164:3300/0 conn(0x7f64180c76d0 0x7f6418152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.904+0000 7f641cd12640 1 -- 172.21.15.71:0/3026891336 >> v2:172.21.15.164:3300/0 conn(0x7f64180c76d0 msgr2=0x7f6418152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.904+0000 7f641cd12640 1 --2- 172.21.15.71:0/3026891336 >> v2:172.21.15.164:3300/0 conn(0x7f64180c76d0 0x7f6418152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.904+0000 7f641cd12640 1 -- 172.21.15.71:0/3026891336 >> v1:172.21.15.164:6789/0 conn(0x7f64180ad2f0 legacy=0x7f6418155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.904+0000 7f641cd12640 1 -- 172.21.15.71:0/3026891336 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641806b4a0 con 0x7f64180c72f0 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.905+0000 7f641cd12640 1 --2- 172.21.15.71:0/3026891336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c72f0 0x7f64180acdb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f64180c7b80 tx=0x7f641402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d9349560b42a67 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.905+0000 7f640f7fe640 1 -- 172.21.15.71:0/3026891336 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414030860 con 0x7f64180c72f0 2023-08-25T09:57:56.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.905+0000 7f640f7fe640 1 -- 172.21.15.71:0/3026891336 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64140309c0 con 0x7f64180c72f0 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.905+0000 7f640f7fe640 1 -- 172.21.15.71:0/3026891336 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414030d50 con 0x7f64180c72f0 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 -- 172.21.15.71:0/3026891336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c72f0 msgr2=0x7f64180acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 --2- 172.21.15.71:0/3026891336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c72f0 0x7f64180acdb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f64180c7b80 tx=0x7f641402b9a0 comp rx=0 tx=0).stop 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 -- 172.21.15.71:0/3026891336 shutdown_connections 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 --2- 172.21.15.71:0/3026891336 >> v2:172.21.15.164:3300/0 conn(0x7f64180c76d0 0x7f6418152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 --2- 172.21.15.71:0/3026891336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c72f0 0x7f64180acdb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:56.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 -- 172.21.15.71:0/3026891336 >> 172.21.15.71:0/3026891336 conn(0x7f641806c280 msgr2=0x7f641806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 -- 172.21.15.71:0/3026891336 shutdown_connections 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.906+0000 7f641dd14640 1 -- 172.21.15.71:0/3026891336 wait complete. 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.908+0000 7f641dd14640 1 Processor -- start 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.908+0000 7f641dd14640 1 -- start start 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641dd14640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64180ad2f0 0x7f64180c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641dd14640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64180c72f0 0x7f64181df480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641dd14640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 0x7f641814c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:56.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641dd14640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641814baf0 con 0x7f64180c76d0 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641dd14640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641814e2b0 con 0x7f64180ad2f0 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641dd14640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641814dfd0 con 0x7f64180c72f0 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641d513640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 0x7f641814c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641d513640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 0x7f641814c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36408/0 (socket says 172.21.15.71:36408) 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.909+0000 7f641d513640 1 -- 172.21.15.71:0/2224882214 learned_addr learned my addr 172.21.15.71:0/2224882214 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641cd12640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64180ad2f0 0x7f64180c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:56.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f640ffff640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64180c72f0 0x7f64181df480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641d513640 1 -- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64180c72f0 msgr2=0x7f64181df480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641d513640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64180c72f0 0x7f64181df480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641d513640 1 -- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64180ad2f0 msgr2=0x7f64180c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641d513640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64180ad2f0 0x7f64180c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641d513640 1 -- 172.21.15.71:0/2224882214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140094d0 con 0x7f64180c76d0 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641cd12640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64180ad2f0 0x7f64180c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641d513640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 0x7f641814c3f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f641000c780 tx=0x7f641000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:56.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410004b60 con 0x7f64180c76d0 2023-08-25T09:57:56.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641000ce90 con 0x7f64180c76d0 2023-08-25T09:57:56.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641000a850 con 0x7f64180c76d0 2023-08-25T09:57:56.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641814b530 con 0x7f64180c76d0 2023-08-25T09:57:56.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.910+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64180c4dd0 con 0x7f64180c76d0 2023-08-25T09:57:56.920 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.912+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641804e420 con 0x7f64180c76d0 2023-08-25T09:57:56.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.912+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f641000a9b0 con 0x7f64180c76d0 2023-08-25T09:57:56.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.914+0000 7f640d7fa640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f640406a050 0x7f640406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:56.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.914+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f6410088bf0 con 0x7f64180c76d0 2023-08-25T09:57:56.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.914+0000 7f641cd12640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f640406a050 0x7f640406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:56.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.915+0000 7f641cd12640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f640406a050 0x7f640406c510 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6414036000 tx=0x7f641403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:56.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:56.919+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6410053ef0 con 0x7f64180c76d0 2023-08-25T09:57:57.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.147+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f641806a970 con 0x7f64180c76d0 2023-08-25T09:57:57.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.149+0000 7f640d7fa640 1 -- 172.21.15.71:0/2224882214 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f6410057510 con 0x7f64180c76d0 2023-08-25T09:57:57.151 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:57.152 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:56.720005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:resolve","state_seq":17,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:57.152 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 64 2023-08-25T09:57:57.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f640406a050 msgr2=0x7f640406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:57.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f640406a050 0x7f640406c510 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6414036000 tx=0x7f641403b040 comp rx=0 tx=0).stop 2023-08-25T09:57:57.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 msgr2=0x7f641814c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:57.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 0x7f641814c3f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f641000c780 tx=0x7f641000cc40 comp rx=0 tx=0).stop 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 shutdown_connections 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f640406a050 0x7f640406c510 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f64180c76d0 0x7f641814c3f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64180c72f0 0x7f64181df480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 --2- 172.21.15.71:0/2224882214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64180ad2f0 0x7f64180c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.151+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 >> 172.21.15.71:0/2224882214 conn(0x7f641806c280 msgr2=0x7f641814fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:57.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.152+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 shutdown_connections 2023-08-25T09:57:57.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:57.152+0000 7f641dd14640 1 -- 172.21.15.71:0/2224882214 wait complete. 2023-08-25T09:57:57.172 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:57:59.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:57:59.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.395+0000 7fac795d6640 1 Processor -- start 2023-08-25T09:57:59.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.395+0000 7fac795d6640 1 -- start start 2023-08-25T09:57:59.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.396+0000 7fac795d6640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fac740c3a40 0x7fac740c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:59.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.396+0000 7fac795d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac741509d0 0x7fac74150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:59.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.396+0000 7fac795d6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac740c1dc0 con 0x7fac740c3660 2023-08-25T09:57:59.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.396+0000 7fac795d6640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac74157a80 con 0x7fac740c3a40 2023-08-25T09:57:59.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.396+0000 7fac795d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac74151360 con 0x7fac741509d0 2023-08-25T09:57:59.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fac740c3a40 0x7fac740c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:59.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fac740c3a40 0x7fac740c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36420/0 (socket says 172.21.15.71:36420) 2023-08-25T09:57:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 -- 172.21.15.71:0/2126522049 learned_addr learned my addr 172.21.15.71:0/2126522049 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac737fe640 1 --2- 172.21.15.71:0/2126522049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac741509d0 0x7fac74150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 -- 172.21.15.71:0/2126522049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac741509d0 msgr2=0x7fac74150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 --2- 172.21.15.71:0/2126522049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac741509d0 0x7fac74150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 -- 172.21.15.71:0/2126522049 >> v1:172.21.15.164:6789/0 conn(0x7fac740c3660 legacy=0x7fac7406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:57:59.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.397+0000 7fac6bfff640 1 -- 172.21.15.71:0/2126522049 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74156e00 con 0x7fac740c3a40 2023-08-25T09:57:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.399+0000 7fac6bfff640 1 --2- 172.21.15.71:0/2126522049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac740c3a40 0x7fac740c1710 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009870 tx=0x7fac5c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8059188eb15453e3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.399+0000 7fac727fc640 1 -- 172.21.15.71:0/2126522049 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c02f830 con 0x7fac740c3a40 2023-08-25T09:57:59.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.399+0000 7fac727fc640 1 -- 172.21.15.71:0/2126522049 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c02f990 con 0x7fac740c3a40 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.399+0000 7fac727fc640 1 -- 172.21.15.71:0/2126522049 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c02fd20 con 0x7fac740c3a40 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 -- 172.21.15.71:0/2126522049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac740c3a40 msgr2=0x7fac740c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 --2- 172.21.15.71:0/2126522049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac740c3a40 0x7fac740c1710 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009870 tx=0x7fac5c02ba10 comp rx=0 tx=0).stop 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 -- 172.21.15.71:0/2126522049 shutdown_connections 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 --2- 172.21.15.71:0/2126522049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac741509d0 0x7fac74150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 --2- 172.21.15.71:0/2126522049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac740c3a40 0x7fac740c1710 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 -- 172.21.15.71:0/2126522049 >> 172.21.15.71:0/2126522049 conn(0x7fac740654f0 msgr2=0x7fac74065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:59.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 -- 172.21.15.71:0/2126522049 shutdown_connections 2023-08-25T09:57:59.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.400+0000 7fac795d6640 1 -- 172.21.15.71:0/2126522049 wait complete. 2023-08-25T09:57:59.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.401+0000 7fac795d6640 1 Processor -- start 2023-08-25T09:57:59.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.402+0000 7fac795d6640 1 -- start start 2023-08-25T09:57:59.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac795d6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac740c3660 0x7fac740c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:59.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac795d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac740c3a40 0x7fac74145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac795d6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 0x7fac74146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac795d6640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74159580 con 0x7fac741509d0 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac795d6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74157e20 con 0x7fac740c3a40 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac795d6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac74156240 con 0x7fac740c3660 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 0x7fac74146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 0x7fac74146460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36434/0 (socket says 172.21.15.71:36434) 2023-08-25T09:57:59.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 -- 172.21.15.71:0/320741475 learned_addr learned my addr 172.21.15.71:0/320741475 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:57:59.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac6bfff640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac740c3a40 0x7fac74145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:59.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac72ffd640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac740c3660 0x7fac740c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:59.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 -- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac740c3660 msgr2=0x7fac740c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac740c3660 0x7fac740c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 -- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac740c3a40 msgr2=0x7fac74145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac740c3a40 0x7fac74145f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.403+0000 7fac737fe640 1 -- 172.21.15.71:0/320741475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c0094d0 con 0x7fac741509d0 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.404+0000 7fac737fe640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 0x7fac74146460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fac6c007e40 tx=0x7fac6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.404+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c014620 con 0x7fac741509d0 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.404+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6c00ee90 con 0x7fac741509d0 2023-08-25T09:57:59.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.404+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac741580c0 con 0x7fac741509d0 2023-08-25T09:57:59.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.404+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c0149e0 con 0x7fac741509d0 2023-08-25T09:57:59.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.404+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7406fd70 con 0x7fac741509d0 2023-08-25T09:57:59.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.405+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac740c29c0 con 0x7fac741509d0 2023-08-25T09:57:59.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.406+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fac6c014b40 con 0x7fac741509d0 2023-08-25T09:57:59.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.407+0000 7fac6b7fe640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fac5406a080 0x7fac5406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:57:59.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.407+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fac6c089510 con 0x7fac741509d0 2023-08-25T09:57:59.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.408+0000 7fac72ffd640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fac5406a080 0x7fac5406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:57:59.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.408+0000 7fac72ffd640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fac5406a080 0x7fac5406c540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fac64000f60 tx=0x7fac64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:57:59.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.412+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fac6c054650 con 0x7fac741509d0 2023-08-25T09:57:59.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.636+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac74069160 con 0x7fac741509d0 2023-08-25T09:57:59.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.637+0000 7fac6b7fe640 1 -- 172.21.15.71:0/320741475 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fac6c057c70 con 0x7fac741509d0 2023-08-25T09:57:59.640 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:57:59.640 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:56.720005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:resolve","state_seq":17,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:57:59.641 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 64 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fac5406a080 msgr2=0x7fac5406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fac5406a080 0x7fac5406c540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fac64000f60 tx=0x7fac64009040 comp rx=0 tx=0).stop 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 msgr2=0x7fac74146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 0x7fac74146460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fac6c007e40 tx=0x7fac6c00e4b0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 shutdown_connections 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac740c3a40 0x7fac74145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fac5406a080 0x7fac5406c540 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac741509d0 0x7fac74146460 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 --2- 172.21.15.71:0/320741475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac740c3660 0x7fac740c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:57:59.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 >> 172.21.15.71:0/320741475 conn(0x7fac740654f0 msgr2=0x7fac740c92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:57:59.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 shutdown_connections 2023-08-25T09:57:59.644 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:57:59.640+0000 7fac795d6640 1 -- 172.21.15.71:0/320741475 wait complete. 2023-08-25T09:57:59.664 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:58:01.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:01.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.886+0000 7fd890b1f640 1 Processor -- start 2023-08-25T09:58:01.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.886+0000 7fd890b1f640 1 -- start start 2023-08-25T09:58:01.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd890b1f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd88c0c3660 0x7fd88c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd890b1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 0x7fd88c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd890b1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88c0c1dc0 con 0x7fd88c0c3a40 2023-08-25T09:58:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd890b1f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88c157a80 con 0x7fd88c0c3660 2023-08-25T09:58:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd890b1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c151360 con 0x7fd88c1509d0 2023-08-25T09:58:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd889d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd88c0c3a40 0x7fd88c06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42414/0 (socket says 172.21.15.71:42414) 2023-08-25T09:58:01.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd889d9b640 1 -- 172.21.15.71:0/443168804 learned_addr learned my addr 172.21.15.71:0/443168804 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:01.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.888+0000 7fd88ad9d640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 0x7fd88c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921731673 0 0) 0x7fd88c0c1dc0 con 0x7fd88c0c3a40 2023-08-25T09:58:01.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88a59c640 1 --2- 172.21.15.71:0/443168804 >> v2:172.21.15.164:3300/0 conn(0x7fd88c0c3660 0x7fd88c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003520 con 0x7fd88c0c3a40 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1637000221 0 0) 0x7fd874003520 con 0x7fd88c0c3a40 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 msgr2=0x7fd88c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 0x7fd88c150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 >> v2:172.21.15.164:3300/0 conn(0x7fd88c0c3660 msgr2=0x7fd88c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 --2- 172.21.15.71:0/443168804 >> v2:172.21.15.164:3300/0 conn(0x7fd88c0c3660 0x7fd88c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c156e00 con 0x7fd88c0c3a40 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fd880001670 con 0x7fd88c0c3a40 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 >> v1:172.21.15.71:6789/0 conn(0x7fd88c0c3a40 legacy=0x7fd88c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8740039e0 0x7fd874003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 0x7fd8740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd874005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c157a80 con 0x7fd874004d00 2023-08-25T09:58:01.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c151360 con 0x7fd8740039e0 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.889+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87400ba00 con 0x7fd88c1509d0 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88ad9d640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd874005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd889d9b640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 0x7fd8740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88a59c640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8740039e0 0x7fd874003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88ad9d640 1 -- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 msgr2=0x7fd8740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88ad9d640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c1509d0 0x7fd8740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88ad9d640 1 -- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8740039e0 msgr2=0x7fd874003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:01.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88ad9d640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8740039e0 0x7fd874003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88ad9d640 1 -- 172.21.15.71:0/443168804 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd874003520 con 0x7fd874004d00 2023-08-25T09:58:01.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.890+0000 7fd88a59c640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8740039e0 0x7fd874003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:01.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.891+0000 7fd88ad9d640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd874005180 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fc0 tx=0x7fd8740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4c9d5058a87815 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:01.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.891+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88402cc20 con 0x7fd874004d00 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.891+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88400f040 con 0x7fd874004d00 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.891+0000 7fd88959a640 1 -- 172.21.15.71:0/443168804 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884034440 con 0x7fd874004d00 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd890b1f640 1 -- 172.21.15.71:0/443168804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 msgr2=0x7fd874005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd890b1f640 1 --2- 172.21.15.71:0/443168804 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd874005180 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fc0 tx=0x7fd8740057f0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd88ad9d640 1 -- 172.21.15.71:0/443168804 reap_dead start 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd890b1f640 1 -- 172.21.15.71:0/443168804 shutdown_connections 2023-08-25T09:58:01.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd890b1f640 1 -- 172.21.15.71:0/443168804 >> 172.21.15.71:0/443168804 conn(0x7fd88c0654f0 msgr2=0x7fd88c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd890b1f640 1 -- 172.21.15.71:0/443168804 shutdown_connections 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.892+0000 7fd890b1f640 1 -- 172.21.15.71:0/443168804 wait complete. 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.894+0000 7fd890b1f640 1 Processor -- start 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.894+0000 7fd890b1f640 1 -- start start 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd890b1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd88c1ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd890b1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88c0c9380 0x7fd88c159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd890b1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c14da40 0x7fd88c1e58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd890b1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1eeca0 con 0x7fd874004d00 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd890b1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c153e90 con 0x7fd88c0c9380 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd890b1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c154130 con 0x7fd88c14da40 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd88a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd88c1ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd88a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd88c1ef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48840/0 (socket says 172.21.15.71:48840) 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.895+0000 7fd88a59c640 1 -- 172.21.15.71:0/285099141 learned_addr learned my addr 172.21.15.71:0/285099141 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:01.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd88a59c640 1 -- 172.21.15.71:0/285099141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c14da40 msgr2=0x7fd88c1e58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:01.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd88a59c640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c14da40 0x7fd88c1e58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd88a59c640 1 -- 172.21.15.71:0/285099141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88c0c9380 msgr2=0x7fd88c159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:01.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd88a59c640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88c0c9380 0x7fd88c159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:01.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd88a59c640 1 -- 172.21.15.71:0/285099141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c154410 con 0x7fd874004d00 2023-08-25T09:58:01.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd88a59c640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd88c1ef0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0109d0 tx=0x7fd87c010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:01.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c00eba0 con 0x7fd874004d00 2023-08-25T09:58:01.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c00ed00 con 0x7fd874004d00 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c01c620 con 0x7fd874004d00 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.896+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c1ee5e0 con 0x7fd874004d00 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.897+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c1e5e30 con 0x7fd874004d00 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.898+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c069160 con 0x7fd874004d00 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.898+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd87c01c780 con 0x7fd874004d00 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.899+0000 7fd87affd640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd86c06a050 0x7fd86c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:01.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.900+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fd87c088ff0 con 0x7fd874004d00 2023-08-25T09:58:01.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.900+0000 7fd889d9b640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd86c06a050 0x7fd86c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:01.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.900+0000 7fd889d9b640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd86c06a050 0x7fd86c06c510 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd880000f60 tx=0x7fd880000ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:01.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:01.904+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd87c0bb990 con 0x7fd874004d00 2023-08-25T09:58:02.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.128+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd88c1eef10 con 0x7fd874004d00 2023-08-25T09:58:02.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.129+0000 7fd87affd640 1 -- 172.21.15.71:0/285099141 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fd87c054320 con 0x7fd874004d00 2023-08-25T09:58:02.132 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:02.132 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:57:56.720005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:resolve","state_seq":17,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:02.132 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 64 2023-08-25T09:58:02.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.131+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd86c06a050 msgr2=0x7fd86c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:02.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.131+0000 7fd890b1f640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd86c06a050 0x7fd86c06c510 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd880000f60 tx=0x7fd880000ef0 comp rx=0 tx=0).stop 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.131+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 msgr2=0x7fd88c1ef0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.131+0000 7fd890b1f640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd88c1ef0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0109d0 tx=0x7fd87c010e90 comp rx=0 tx=0).stop 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 shutdown_connections 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd88c0c9380 0x7fd88c159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd86c06a050 0x7fd86c06c510 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd88c14da40 0x7fd88c1e58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:02.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 --2- 172.21.15.71:0/285099141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd874004d00 0x7fd88c1ef0c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:02.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 >> 172.21.15.71:0/285099141 conn(0x7fd88c0654f0 msgr2=0x7fd88c0c5350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:02.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 shutdown_connections 2023-08-25T09:58:02.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:02.132+0000 7fd890b1f640 1 -- 172.21.15.71:0/285099141 wait complete. 2023-08-25T09:58:02.153 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:58:04.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:04.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.376+0000 7f451521f640 1 Processor -- start 2023-08-25T09:58:04.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.376+0000 7f451521f640 1 -- start start 2023-08-25T09:58:04.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f451521f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f45100c4560 0x7f451006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f451521f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451006eb60 con 0x7f4510070cf0 2023-08-25T09:58:04.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f451521f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451006ca30 con 0x7f4510152d80 2023-08-25T09:58:04.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f451521f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f451006cf90 con 0x7f45100c4560 2023-08-25T09:58:04.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f450f59e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4510070cf0 0x7f45101637b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42428/0 (socket says 172.21.15.71:42428) 2023-08-25T09:58:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f450f59e640 1 -- 172.21.15.71:0/3532359535 learned_addr learned my addr 172.21.15.71:0/3532359535 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f450e59c640 1 --2- 172.21.15.71:0/3532359535 >> v2:172.21.15.164:3300/0 conn(0x7f45100c4560 0x7f451006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.378+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309847647 0 0) 0x7f451006eb60 con 0x7f4510070cf0 2023-08-25T09:58:04.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc003520 con 0x7f4510070cf0 2023-08-25T09:58:04.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544513973 0 0) 0x7f44fc003520 con 0x7f4510070cf0 2023-08-25T09:58:04.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 >> v2:172.21.15.164:3300/0 conn(0x7f45100c4560 msgr2=0x7f451006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:04.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 --2- 172.21.15.71:0/3532359535 >> v2:172.21.15.164:3300/0 conn(0x7f45100c4560 0x7f451006e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 >> v1:172.21.15.164:6789/0 conn(0x7f4510152d80 legacy=0x7f45100c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4510158fd0 con 0x7f4510070cf0 2023-08-25T09:58:04.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4508001670 con 0x7f4510070cf0 2023-08-25T09:58:04.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 >> v1:172.21.15.71:6789/0 conn(0x7f4510070cf0 legacy=0x7f45101637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f44fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44fc0047c0 0x7f44fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44fc0051f0 0x7f44fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451006cf90 con 0x7f44fc0039e0 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc00c150 con 0x7f44fc0051f0 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc00c310 con 0x7f44fc0047c0 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.379+0000 7f450ed9d640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f44fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450e59c640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44fc0047c0 0x7f44fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450ed9d640 1 -- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44fc0047c0 msgr2=0x7f44fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:04.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450ed9d640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44fc0047c0 0x7f44fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450ed9d640 1 -- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44fc0051f0 msgr2=0x7f44fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:04.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450ed9d640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44fc0051f0 0x7f44fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450ed9d640 1 -- 172.21.15.71:0/3532359535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc003520 con 0x7f44fc0039e0 2023-08-25T09:58:04.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450f59e640 1 -- 172.21.15.71:0/3532359535 reap_dead start 2023-08-25T09:58:04.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.380+0000 7f450e59c640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44fc0047c0 0x7f44fc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.381+0000 7f450ed9d640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f44fc003e00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4500001e70 tx=0x7f44fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc5c7044606aa38 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.381+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500003b70 con 0x7f44fc0039e0 2023-08-25T09:58:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.381+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4500003cd0 con 0x7f44fc0039e0 2023-08-25T09:58:04.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.381+0000 7f450dd9b640 1 -- 172.21.15.71:0/3532359535 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500005310 con 0x7f44fc0039e0 2023-08-25T09:58:04.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 -- 172.21.15.71:0/3532359535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 msgr2=0x7f44fc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f44fc003e00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4500001e70 tx=0x7f44fc0057f0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 -- 172.21.15.71:0/3532359535 shutdown_connections 2023-08-25T09:58:04.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 --2- 172.21.15.71:0/3532359535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f44fc003e00 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 -- 172.21.15.71:0/3532359535 >> 172.21.15.71:0/3532359535 conn(0x7f45100654f0 msgr2=0x7f4510065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:04.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 -- 172.21.15.71:0/3532359535 shutdown_connections 2023-08-25T09:58:04.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.382+0000 7f451521f640 1 -- 172.21.15.71:0/3532359535 wait complete. 2023-08-25T09:58:04.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.384+0000 7f451521f640 1 Processor -- start 2023-08-25T09:58:04.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.384+0000 7f451521f640 1 -- start start 2023-08-25T09:58:04.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f451521f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f45101481b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f451521f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f451014ea60 0x7f4510155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f451521f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4510156290 0x7f45100c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f451521f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4510153f60 con 0x7f44fc0039e0 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f451521f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45101560a0 con 0x7f451014ea60 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f451521f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45101ee6f0 con 0x7f4510156290 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f450ed9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f45101481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f450ed9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f45101481b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48860/0 (socket says 172.21.15.71:48860) 2023-08-25T09:58:04.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.385+0000 7f450ed9d640 1 -- 172.21.15.71:0/468780471 learned_addr learned my addr 172.21.15.71:0/468780471 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450e59c640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f451014ea60 0x7f4510155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450f59e640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4510156290 0x7f45100c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450ed9d640 1 -- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4510156290 msgr2=0x7f45100c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450ed9d640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4510156290 0x7f45100c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450ed9d640 1 -- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f451014ea60 msgr2=0x7f4510155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450ed9d640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f451014ea60 0x7f4510155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450ed9d640 1 -- 172.21.15.71:0/468780471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4510066110 con 0x7f44fc0039e0 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450f59e640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4510156290 0x7f45100c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f450ed9d640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f45101481b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4500005040 tx=0x7f45000014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.386+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500006d50 con 0x7f44fc0039e0 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.387+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4500003e40 con 0x7f44fc0039e0 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.387+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45101ee990 con 0x7f44fc0039e0 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.387+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4500039880 con 0x7f44fc0039e0 2023-08-25T09:58:04.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.387+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4510155080 con 0x7f44fc0039e0 2023-08-25T09:58:04.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.388+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4510069160 con 0x7f44fc0039e0 2023-08-25T09:58:04.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.389+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f45000399e0 con 0x7f44fc0039e0 2023-08-25T09:58:04.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.390+0000 7f45077fe640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44f006a080 0x7f44f006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:04.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.390+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f45000ab120 con 0x7f44fc0039e0 2023-08-25T09:58:04.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.390+0000 7f450e59c640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44f006a080 0x7f44f006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:04.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.391+0000 7f450e59c640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44f006a080 0x7f44f006c540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f45100c2c80 tx=0x7f44f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:04.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.394+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4500076380 con 0x7f44fc0039e0 2023-08-25T09:58:04.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.630+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f451015d2c0 con 0x7f44fc0039e0 2023-08-25T09:58:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.631+0000 7f45077fe640 1 -- 172.21.15.71:0/468780471 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f45000799a0 con 0x7f44fc0039e0 2023-08-25T09:58:04.634 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:04.635 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:02.726727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 65 2023-08-25T09:58:04.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44f006a080 msgr2=0x7f44f006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44f006a080 0x7f44f006c540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f45100c2c80 tx=0x7f44f8008040 comp rx=0 tx=0).stop 2023-08-25T09:58:04.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 msgr2=0x7f45101481b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:04.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f45101481b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4500005040 tx=0x7f45000014a0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 shutdown_connections 2023-08-25T09:58:04.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f451014ea60 0x7f4510155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44f006a080 0x7f44f006c540 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4510156290 0x7f45100c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 --2- 172.21.15.71:0/468780471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44fc0039e0 0x7f45101481b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:04.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 >> 172.21.15.71:0/468780471 conn(0x7f45100654f0 msgr2=0x7f451006aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:04.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 shutdown_connections 2023-08-25T09:58:04.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:04.634+0000 7f451521f640 1 -- 172.21.15.71:0/468780471 wait complete. 2023-08-25T09:58:04.656 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 65, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:reconnect", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:58:02.726727+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [] } 2023-08-25T09:58:06.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:06.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.877+0000 7f2c50c6a640 1 Processor -- start 2023-08-25T09:58:06.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.877+0000 7f2c50c6a640 1 -- start start 2023-08-25T09:58:06.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.878+0000 7f2c50c6a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2c4c1594d0 0x7f2c4c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:06.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.878+0000 7f2c50c6a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2c4c06cc70 0x7f2c4c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:06.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.878+0000 7f2c50c6a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4c06a960 con 0x7f2c4c0c3940 2023-08-25T09:58:06.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.878+0000 7f2c50c6a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4c06ac20 con 0x7f2c4c1594d0 2023-08-25T09:58:06.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.878+0000 7f2c50c6a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4c06af00 con 0x7f2c4c06cc70 2023-08-25T09:58:06.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c4a59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2c4c1594d0 0x7f2c4c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c4ad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f2c4c0c3940 0x7f2c4c1598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42430/0 (socket says 172.21.15.71:42430) 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c4ad9d640 1 -- 172.21.15.71:0/2757301853 learned_addr learned my addr 172.21.15.71:0/2757301853 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c49d9b640 1 --2- 172.21.15.71:0/2757301853 >> v2:172.21.15.164:3300/0 conn(0x7f2c4c06cc70 0x7f2c4c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c4959a640 1 -- 172.21.15.71:0/2757301853 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121164465 0 0) 0x7f2c4c06a960 con 0x7f2c4c0c3940 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c4959a640 1 -- 172.21.15.71:0/2757301853 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c34005730 con 0x7f2c4c0c3940 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.879+0000 7f2c4a59c640 1 -- 172.21.15.71:0/2757301853 >> v2:172.21.15.164:3300/0 conn(0x7f2c4c06cc70 msgr2=0x7f2c4c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:06.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.880+0000 7f2c4a59c640 1 --2- 172.21.15.71:0/2757301853 >> v2:172.21.15.164:3300/0 conn(0x7f2c4c06cc70 0x7f2c4c14f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:06.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.880+0000 7f2c4a59c640 1 -- 172.21.15.71:0/2757301853 >> v1:172.21.15.71:6789/0 conn(0x7f2c4c0c3940 legacy=0x7f2c4c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:06.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.880+0000 7f2c4a59c640 1 -- 172.21.15.71:0/2757301853 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c0c2e60 con 0x7f2c4c1594d0 2023-08-25T09:58:06.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.881+0000 7f2c4a59c640 1 --2- 172.21.15.71:0/2757301853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c1594d0 0x7f2c4c0c4c90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c156da0 tx=0x7f2c3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d2e311612b077b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:06.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c4959a640 1 -- 172.21.15.71:0/2757301853 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c031830 con 0x7f2c4c1594d0 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c4959a640 1 -- 172.21.15.71:0/2757301853 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c031990 con 0x7f2c4c1594d0 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c4959a640 1 -- 172.21.15.71:0/2757301853 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c031d20 con 0x7f2c4c1594d0 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 -- 172.21.15.71:0/2757301853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c1594d0 msgr2=0x7f2c4c0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/2757301853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c1594d0 0x7f2c4c0c4c90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c156da0 tx=0x7f2c3c02b8d0 comp rx=0 tx=0).stop 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 -- 172.21.15.71:0/2757301853 shutdown_connections 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/2757301853 >> v2:172.21.15.164:3300/0 conn(0x7f2c4c06cc70 0x7f2c4c14f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:06.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/2757301853 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c1594d0 0x7f2c4c0c4c90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:06.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 -- 172.21.15.71:0/2757301853 >> 172.21.15.71:0/2757301853 conn(0x7f2c4c0654f0 msgr2=0x7f2c4c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:06.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 -- 172.21.15.71:0/2757301853 shutdown_connections 2023-08-25T09:58:06.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.882+0000 7f2c50c6a640 1 -- 172.21.15.71:0/2757301853 wait complete. 2023-08-25T09:58:06.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.884+0000 7f2c50c6a640 1 Processor -- start 2023-08-25T09:58:06.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.884+0000 7f2c50c6a640 1 -- start start 2023-08-25T09:58:06.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c50c6a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c4c06cc70 0x7f2c4c1e36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:06.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c50c6a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 0x7f2c4c1f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:06.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c50c6a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c4c1594d0 0x7f2c4c0c9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:06.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c50c6a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c149fe0 con 0x7f2c4c0c3940 2023-08-25T09:58:06.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c50c6a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1e3c30 con 0x7f2c4c06cc70 2023-08-25T09:58:06.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c50c6a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1edfc0 con 0x7f2c4c1594d0 2023-08-25T09:58:06.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c49d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 0x7f2c4c1f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:06.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c49d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 0x7f2c4c1f1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48870/0 (socket says 172.21.15.71:48870) 2023-08-25T09:58:06.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.885+0000 7f2c49d9b640 1 -- 172.21.15.71:0/1580096757 learned_addr learned my addr 172.21.15.71:0/1580096757 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:06.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c4a59c640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c4c06cc70 0x7f2c4c1e36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c49d9b640 1 -- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c4c1594d0 msgr2=0x7f2c4c0c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c4ad9d640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c4c1594d0 0x7f2c4c0c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c49d9b640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c4c1594d0 0x7f2c4c0c9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c49d9b640 1 -- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c4c06cc70 msgr2=0x7f2c4c1e36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c49d9b640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c4c06cc70 0x7f2c4c1e36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c49d9b640 1 -- 172.21.15.71:0/1580096757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c009400 con 0x7f2c4c0c3940 2023-08-25T09:58:06.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c49d9b640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 0x7f2c4c1f1f40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000efc0 tx=0x7f2c4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:06.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40009b80 con 0x7f2c4c0c3940 2023-08-25T09:58:06.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4000f040 con 0x7f2c4c0c3940 2023-08-25T09:58:06.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.886+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c1f2480 con 0x7f2c4c0c3940 2023-08-25T09:58:06.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.887+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c400146a0 con 0x7f2c4c0c3940 2023-08-25T09:58:06.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.887+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c1e40f0 con 0x7f2c4c0c3940 2023-08-25T09:58:06.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.888+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c069160 con 0x7f2c4c0c3940 2023-08-25T09:58:06.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.889+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2c40014800 con 0x7f2c4c0c3940 2023-08-25T09:58:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.889+0000 7f2c3affd640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2c2c06a050 0x7f2c2c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.890+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f2c400893e0 con 0x7f2c4c0c3940 2023-08-25T09:58:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.890+0000 7f2c4a59c640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2c2c06a050 0x7f2c2c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:06.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.891+0000 7f2c4a59c640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2c2c06a050 0x7f2c2c06c510 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c036000 tx=0x7f2c3c03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:06.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:06.894+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2c400580e0 con 0x7f2c4c0c3940 2023-08-25T09:58:07.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.121+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4c1f2480 con 0x7f2c4c0c3940 2023-08-25T09:58:07.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.122+0000 7f2c3affd640 1 -- 172.21.15.71:0/1580096757 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f2c4c1f2480 con 0x7f2c4c0c3940 2023-08-25T09:58:07.126 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:07.126 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:02.726727+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:reconnect","state_seq":19,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:07.127 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 65 2023-08-25T09:58:07.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2c2c06a050 msgr2=0x7f2c2c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:07.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2c2c06a050 0x7f2c2c06c510 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c036000 tx=0x7f2c3c03c040 comp rx=0 tx=0).stop 2023-08-25T09:58:07.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 msgr2=0x7f2c4c1f1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:07.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 0x7f2c4c1f1f40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000efc0 tx=0x7f2c4000c490 comp rx=0 tx=0).stop 2023-08-25T09:58:07.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 shutdown_connections 2023-08-25T09:58:07.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2c2c06a050 0x7f2c2c06c510 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:07.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c4c1594d0 0x7f2c4c0c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:07.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2c4c0c3940 0x7f2c4c1f1f40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:07.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 --2- 172.21.15.71:0/1580096757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c4c06cc70 0x7f2c4c1e36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:07.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 >> 172.21.15.71:0/1580096757 conn(0x7f2c4c0654f0 msgr2=0x7f2c4c0c5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:07.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 shutdown_connections 2023-08-25T09:58:07.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:07.124+0000 7f2c50c6a640 1 -- 172.21.15.71:0/1580096757 wait complete. 2023-08-25T09:58:07.146 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:58:09.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:09.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.367+0000 7fbc7674e640 1 Processor -- start 2023-08-25T09:58:09.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.368+0000 7fbc7674e640 1 -- start start 2023-08-25T09:58:09.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.369+0000 7fbc7674e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbc701530c0 0x7fbc700c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:09.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.369+0000 7fbc7674e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7006a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:09.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.369+0000 7fbc7674e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc70158e40 con 0x7fbc7006aea0 2023-08-25T09:58:09.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.369+0000 7fbc7674e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7006bc80 con 0x7fbc701530c0 2023-08-25T09:58:09.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.369+0000 7fbc7674e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc701534a0 con 0x7fbc700c7740 2023-08-25T09:58:09.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbc701530c0 0x7fbc700c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:09.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbc701530c0 0x7fbc700c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48882/0 (socket says 172.21.15.71:48882) 2023-08-25T09:58:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 -- 172.21.15.71:0/1486426327 learned_addr learned my addr 172.21.15.71:0/1486426327 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6f7fe640 1 --2- 172.21.15.71:0/1486426327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 -- 172.21.15.71:0/1486426327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 msgr2=0x7fbc7006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:09.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 --2- 172.21.15.71:0/1486426327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7006a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 -- 172.21.15.71:0/1486426327 >> v1:172.21.15.164:6789/0 conn(0x7fbc7006aea0 legacy=0x7fbc701592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:09.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.370+0000 7fbc6ffff640 1 -- 172.21.15.71:0/1486426327 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc700c7380 con 0x7fbc701530c0 2023-08-25T09:58:09.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.371+0000 7fbc6ffff640 1 --2- 172.21.15.71:0/1486426327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc700c6e40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009e50 tx=0x7fbc6002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa68fc86fba8d0a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:09.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc6effd640 1 -- 172.21.15.71:0/1486426327 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6002cb00 con 0x7fbc701530c0 2023-08-25T09:58:09.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc6effd640 1 -- 172.21.15.71:0/1486426327 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6002cc60 con 0x7fbc701530c0 2023-08-25T09:58:09.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc6effd640 1 -- 172.21.15.71:0/1486426327 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600343f0 con 0x7fbc701530c0 2023-08-25T09:58:09.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc7674e640 1 -- 172.21.15.71:0/1486426327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 msgr2=0x7fbc700c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:09.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc7674e640 1 --2- 172.21.15.71:0/1486426327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc700c6e40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009e50 tx=0x7fbc6002bf60 comp rx=0 tx=0).stop 2023-08-25T09:58:09.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc7674e640 1 -- 172.21.15.71:0/1486426327 shutdown_connections 2023-08-25T09:58:09.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc7674e640 1 --2- 172.21.15.71:0/1486426327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc7674e640 1 --2- 172.21.15.71:0/1486426327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc700c6e40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.372+0000 7fbc7674e640 1 -- 172.21.15.71:0/1486426327 >> 172.21.15.71:0/1486426327 conn(0x7fbc700654f0 msgr2=0x7fbc70065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:09.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.373+0000 7fbc7674e640 1 -- 172.21.15.71:0/1486426327 shutdown_connections 2023-08-25T09:58:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.373+0000 7fbc7674e640 1 -- 172.21.15.71:0/1486426327 wait complete. 2023-08-25T09:58:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.374+0000 7fbc7674e640 1 Processor -- start 2023-08-25T09:58:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.375+0000 7fbc7674e640 1 -- start start 2023-08-25T09:58:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.375+0000 7fbc7674e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc7006aea0 0x7fbc700cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.375+0000 7fbc7674e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7014a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:09.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc7674e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc7014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc7674e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700cb570 con 0x7fbc701530c0 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc7674e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc700caab0 con 0x7fbc7006aea0 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc7674e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc70146280 con 0x7fbc700c7740 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc7014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc7014aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48888/0 (socket says 172.21.15.71:48888) 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 -- 172.21.15.71:0/2477208960 learned_addr learned my addr 172.21.15.71:0/2477208960 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:09.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc6ffff640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc7006aea0 0x7fbc700cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc6f7fe640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7014a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 -- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 msgr2=0x7fbc7014a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7014a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 -- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc7006aea0 msgr2=0x7fbc700cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc7006aea0 0x7fbc700cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.376+0000 7fbc74ceb640 1 -- 172.21.15.71:0/2477208960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60009b30 con 0x7fbc701530c0 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.377+0000 7fbc74ceb640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc7014aa80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800b440 tx=0x7fbc6800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:09.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.377+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68011830 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.377+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc68011990 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.377+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc70146560 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.377+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68011d40 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.377+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc700ca740 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.378+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc70069160 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.379+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbc6801a400 con 0x7fbc701530c0 2023-08-25T09:58:09.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.380+0000 7fbc6cff9640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbc5006a080 0x7fbc5006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:09.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.380+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fbc680888b0 con 0x7fbc701530c0 2023-08-25T09:58:09.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.380+0000 7fbc6ffff640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbc5006a080 0x7fbc5006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:09.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.381+0000 7fbc6ffff640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbc5006a080 0x7fbc5006c540 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbc60035000 tx=0x7fbc60030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:09.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.384+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbc68053b30 con 0x7fbc701530c0 2023-08-25T09:58:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.625+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc7006f250 con 0x7fbc701530c0 2023-08-25T09:58:09.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.626+0000 7fbc6cff9640 1 -- 172.21.15.71:0/2477208960 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fbc68057150 con 0x7fbc701530c0 2023-08-25T09:58:09.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:09.629 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:07.958477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:09.630 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 66 2023-08-25T09:58:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.628+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbc5006a080 msgr2=0x7fbc5006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.628+0000 7fbc7674e640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbc5006a080 0x7fbc5006c540 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbc60035000 tx=0x7fbc60030000 comp rx=0 tx=0).stop 2023-08-25T09:58:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.628+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 msgr2=0x7fbc7014aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.628+0000 7fbc7674e640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc7014aa80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800b440 tx=0x7fbc6800b900 comp rx=0 tx=0).stop 2023-08-25T09:58:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 shutdown_connections 2023-08-25T09:58:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbc5006a080 0x7fbc5006c540 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbc701530c0 0x7fbc7014aa80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc700c7740 0x7fbc7014a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 --2- 172.21.15.71:0/2477208960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc7006aea0 0x7fbc700cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:09.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 >> 172.21.15.71:0/2477208960 conn(0x7fbc700654f0 msgr2=0x7fbc700658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:09.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 shutdown_connections 2023-08-25T09:58:09.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:09.629+0000 7fbc7674e640 1 -- 172.21.15.71:0/2477208960 wait complete. 2023-08-25T09:58:09.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:58:11.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:11.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.872+0000 7fee99912640 1 Processor -- start 2023-08-25T09:58:11.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.872+0000 7fee99912640 1 -- start start 2023-08-25T09:58:11.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee99912640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9415ce20 0x7fee940ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:11.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee99912640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee940ceb80 0x7fee940c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:11.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee99912640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee94159cc0 con 0x7fee94153120 2023-08-25T09:58:11.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee99912640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee940ca040 con 0x7fee940ceb80 2023-08-25T09:58:11.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee99912640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee94158ea0 con 0x7fee9415ce20 2023-08-25T09:58:11.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee92ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fee94153120 0x7fee940c9670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49484/0 (socket says 172.21.15.71:49484) 2023-08-25T09:58:11.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee92ffd640 1 -- 172.21.15.71:0/590832604 learned_addr learned my addr 172.21.15.71:0/590832604 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:11.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee937fe640 1 --2- 172.21.15.71:0/590832604 >> v2:172.21.15.164:3300/0 conn(0x7fee940ceb80 0x7fee940c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:11.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.874+0000 7fee8bfff640 1 --2- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9415ce20 0x7fee940ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:11.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee927fc640 1 -- 172.21.15.71:0/590832604 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751040356 0 0) 0x7fee94159cc0 con 0x7fee94153120 2023-08-25T09:58:11.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee927fc640 1 -- 172.21.15.71:0/590832604 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee80003520 con 0x7fee94153120 2023-08-25T09:58:11.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee937fe640 1 -- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9415ce20 msgr2=0x7fee940ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:11.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee937fe640 1 --2- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9415ce20 0x7fee940ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:11.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee937fe640 1 -- 172.21.15.71:0/590832604 >> v1:172.21.15.71:6789/0 conn(0x7fee94153120 legacy=0x7fee940c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:11.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee927fc640 1 -- 172.21.15.71:0/590832604 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158954395 0 0) 0x7fee80003520 con 0x7fee94153120 2023-08-25T09:58:11.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.875+0000 7fee937fe640 1 -- 172.21.15.71:0/590832604 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940c9d80 con 0x7fee940ceb80 2023-08-25T09:58:11.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.876+0000 7fee937fe640 1 --2- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee940ceb80 0x7fee940c37c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009d90 tx=0x7fee8c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5304c4366fc9f702 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:11.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.877+0000 7fee927fc640 1 -- 172.21.15.71:0/590832604 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84001670 con 0x7fee940ceb80 2023-08-25T09:58:11.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.877+0000 7fee927fc640 1 -- 172.21.15.71:0/590832604 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c02c8a0 con 0x7fee940ceb80 2023-08-25T09:58:11.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.877+0000 7fee927fc640 1 -- 172.21.15.71:0/590832604 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8c02cc30 con 0x7fee940ceb80 2023-08-25T09:58:11.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 -- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee940ceb80 msgr2=0x7fee940c37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:11.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 --2- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee940ceb80 0x7fee940c37c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fee8c009d90 tx=0x7fee8c02bd70 comp rx=0 tx=0).stop 2023-08-25T09:58:11.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 -- 172.21.15.71:0/590832604 shutdown_connections 2023-08-25T09:58:11.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 --2- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9415ce20 0x7fee940ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:11.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 --2- 172.21.15.71:0/590832604 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee940ceb80 0x7fee940c37c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:11.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 -- 172.21.15.71:0/590832604 >> 172.21.15.71:0/590832604 conn(0x7fee9406a5c0 msgr2=0x7fee9406b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:11.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 -- 172.21.15.71:0/590832604 shutdown_connections 2023-08-25T09:58:11.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.878+0000 7fee99912640 1 -- 172.21.15.71:0/590832604 wait complete. 2023-08-25T09:58:11.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.880+0000 7fee99912640 1 Processor -- start 2023-08-25T09:58:11.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.880+0000 7fee99912640 1 -- start start 2023-08-25T09:58:11.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee99912640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 0x7fee94150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:11.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee99912640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee94153120 0x7fee941ef1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:11.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee99912640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9415ce20 0x7fee9415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:11.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee99912640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9414b210 con 0x7fee940ceb80 2023-08-25T09:58:11.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee99912640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9414eb30 con 0x7fee9415ce20 2023-08-25T09:58:11.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee99912640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9414ed60 con 0x7fee94153120 2023-08-25T09:58:11.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee92ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 0x7fee94150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:11.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee92ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 0x7fee94150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48978/0 (socket says 172.21.15.71:48978) 2023-08-25T09:58:11.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.881+0000 7fee92ffd640 1 -- 172.21.15.71:0/20432744 learned_addr learned my addr 172.21.15.71:0/20432744 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee92ffd640 1 -- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee94153120 msgr2=0x7fee941ef1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee8bfff640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee94153120 0x7fee941ef1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee92ffd640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee94153120 0x7fee941ef1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee92ffd640 1 -- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9415ce20 msgr2=0x7fee9415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee937fe640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9415ce20 0x7fee9415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee92ffd640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9415ce20 0x7fee9415b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:11.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee92ffd640 1 -- 172.21.15.71:0/20432744 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c009a70 con 0x7fee940ceb80 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee92ffd640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 0x7fee94150420 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c790 tx=0x7fee8400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84005ad0 con 0x7fee940ceb80 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee84005c30 con 0x7fee940ceb80 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee84007070 con 0x7fee940ceb80 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9415c260 con 0x7fee940ceb80 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.882+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9414f300 con 0x7fee940ceb80 2023-08-25T09:58:11.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.884+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee941eeff0 con 0x7fee940ceb80 2023-08-25T09:58:11.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.884+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fee840071d0 con 0x7fee940ceb80 2023-08-25T09:58:11.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.886+0000 7fee90ff9640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fee7406a050 0x7fee7406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:11.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.886+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7fee840888c0 con 0x7fee940ceb80 2023-08-25T09:58:11.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.886+0000 7fee8bfff640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fee7406a050 0x7fee7406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:11.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.887+0000 7fee8bfff640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fee7406a050 0x7fee7406c510 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee9415cba0 tx=0x7fee7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:11.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:11.891+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fee84053a00 con 0x7fee940ceb80 2023-08-25T09:58:12.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.118+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee94069160 con 0x7fee940ceb80 2023-08-25T09:58:12.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.119+0000 7fee90ff9640 1 -- 172.21.15.71:0/20432744 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fee84057020 con 0x7fee940ceb80 2023-08-25T09:58:12.122 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:12.122 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:07.958477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:12.122 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 66 2023-08-25T09:58:12.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fee7406a050 msgr2=0x7fee7406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:12.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fee7406a050 0x7fee7406c510 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee9415cba0 tx=0x7fee7c008040 comp rx=0 tx=0).stop 2023-08-25T09:58:12.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 msgr2=0x7fee94150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:12.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 0x7fee94150420 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fee8400c790 tx=0x7fee8400cc50 comp rx=0 tx=0).stop 2023-08-25T09:58:12.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 shutdown_connections 2023-08-25T09:58:12.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9415ce20 0x7fee9415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:12.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fee7406a050 0x7fee7406c510 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:12.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee94153120 0x7fee941ef1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:12.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 --2- 172.21.15.71:0/20432744 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fee940ceb80 0x7fee94150420 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:12.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.121+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 >> 172.21.15.71:0/20432744 conn(0x7fee9406a5c0 msgr2=0x7fee940b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:12.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.123+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 shutdown_connections 2023-08-25T09:58:12.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:12.123+0000 7fee99912640 1 -- 172.21.15.71:0/20432744 wait complete. 2023-08-25T09:58:12.143 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:58:14.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:14.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.368+0000 7f2451c2d640 1 Processor -- start 2023-08-25T09:58:14.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.368+0000 7f2451c2d640 1 -- start start 2023-08-25T09:58:14.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.369+0000 7f2451c2d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 0x7f244c0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.369+0000 7f2451c2d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f244c070a90 0x7f244c0c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.369+0000 7f2451c2d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0c2b80 con 0x7f244c150c50 2023-08-25T09:58:14.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.369+0000 7f2451c2d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c0c1770 con 0x7f244c070a90 2023-08-25T09:58:14.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.369+0000 7f2451c2d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0c1f30 con 0x7f244c0c76b0 2023-08-25T09:58:14.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.370+0000 7f244b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f244c150c50 0x7f244c0c5380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49488/0 (socket says 172.21.15.71:49488) 2023-08-25T09:58:14.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.370+0000 7f244b7fe640 1 -- 172.21.15.71:0/522107748 learned_addr learned my addr 172.21.15.71:0/522107748 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:14.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.370+0000 7f2443fff640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 0x7f244c0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.370+0000 7f244bfff640 1 --2- 172.21.15.71:0/522107748 >> v2:172.21.15.164:3300/0 conn(0x7f244c070a90 0x7f244c0c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.370+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800974115 0 0) 0x7f244c0c2b80 con 0x7f244c150c50 2023-08-25T09:58:14.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.370+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2438003520 con 0x7f244c150c50 2023-08-25T09:58:14.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162640457 0 0) 0x7f2438003520 con 0x7f244c150c50 2023-08-25T09:58:14.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 msgr2=0x7f244c0c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:14.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 0x7f244c0c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 >> v2:172.21.15.164:3300/0 conn(0x7f244c070a90 msgr2=0x7f244c0c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:14.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 --2- 172.21.15.71:0/522107748 >> v2:172.21.15.164:3300/0 conn(0x7f244c070a90 0x7f244c0c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c0c6400 con 0x7f244c150c50 2023-08-25T09:58:14.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f243c0017f0 con 0x7f244c150c50 2023-08-25T09:58:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 >> v1:172.21.15.71:6789/0 conn(0x7f244c150c50 legacy=0x7f244c0c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 0x7f2438003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2438003ef0 0x7f24380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2438004d00 0x7f2438005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0c1770 con 0x7f2438004d00 2023-08-25T09:58:14.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0c1f30 con 0x7f2438003ef0 2023-08-25T09:58:14.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243800b9b0 con 0x7f244c0c76b0 2023-08-25T09:58:14.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244bfff640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2438004d00 0x7f2438005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244b7fe640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 0x7f2438003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.371+0000 7f244bfff640 1 -- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 msgr2=0x7f2438003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:14.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.372+0000 7f244bfff640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f244c0c76b0 0x7f2438003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.372+0000 7f244bfff640 1 -- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2438003ef0 msgr2=0x7f24380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:14.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.372+0000 7f244bfff640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2438003ef0 0x7f24380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.372+0000 7f244bfff640 1 -- 172.21.15.71:0/522107748 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2438003520 con 0x7f2438004d00 2023-08-25T09:58:14.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f244bfff640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2438004d00 0x7f2438005180 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2444009f90 tx=0x7f24380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5222ee699e4ed3d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:14.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244402cc20 con 0x7f2438004d00 2023-08-25T09:58:14.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244400f040 con 0x7f2438004d00 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f244affd640 1 -- 172.21.15.71:0/522107748 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444034440 con 0x7f2438004d00 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f2451c2d640 1 -- 172.21.15.71:0/522107748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2438004d00 msgr2=0x7f2438005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f2451c2d640 1 --2- 172.21.15.71:0/522107748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2438004d00 0x7f2438005180 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2444009f90 tx=0x7f24380057f0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.373+0000 7f244bfff640 1 -- 172.21.15.71:0/522107748 reap_dead start 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.374+0000 7f2451c2d640 1 -- 172.21.15.71:0/522107748 shutdown_connections 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.374+0000 7f2451c2d640 1 -- 172.21.15.71:0/522107748 >> 172.21.15.71:0/522107748 conn(0x7f244c06bff0 msgr2=0x7f244c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:14.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.374+0000 7f2451c2d640 1 -- 172.21.15.71:0/522107748 shutdown_connections 2023-08-25T09:58:14.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.374+0000 7f2451c2d640 1 -- 172.21.15.71:0/522107748 wait complete. 2023-08-25T09:58:14.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.375+0000 7f2451c2d640 1 Processor -- start 2023-08-25T09:58:14.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.376+0000 7f2451c2d640 1 -- start start 2023-08-25T09:58:14.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.376+0000 7f2451c2d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2438004d00 0x7f244c13e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2451c2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f244c13c700 0x7f244c13eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2451c2d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 0x7f244c14a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2451c2d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c13f3e0 con 0x7f244c149e70 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2451c2d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c13df10 con 0x7f244c13c700 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2451c2d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c06aab0 con 0x7f2438004d00 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 0x7f244c14a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 0x7f244c14a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48986/0 (socket says 172.21.15.71:48986) 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 -- 172.21.15.71:0/3943649012 learned_addr learned my addr 172.21.15.71:0/3943649012 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244b7fe640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2438004d00 0x7f244c13e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2443fff640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f244c13c700 0x7f244c13eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 -- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2438004d00 msgr2=0x7f244c13e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2438004d00 0x7f244c13e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 -- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f244c13c700 msgr2=0x7f244c13eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:14.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f244c13c700 0x7f244c13eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244bfff640 1 -- 172.21.15.71:0/3943649012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c147a10 con 0x7f244c149e70 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f2443fff640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f244c13c700 0x7f244c13eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.377+0000 7f244b7fe640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2438004d00 0x7f244c13e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.378+0000 7f244bfff640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 0x7f244c14a320 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2444002230 tx=0x7f2444009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.378+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2444034d50 con 0x7f244c149e70 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.378+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244403e440 con 0x7f244c149e70 2023-08-25T09:58:14.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.378+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c147cf0 con 0x7f244c149e70 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.378+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244403e7f0 con 0x7f244c149e70 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.378+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c13e460 con 0x7f244c149e70 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.379+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c147eb0 con 0x7f244c149e70 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.380+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f244403e950 con 0x7f244c149e70 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.381+0000 7f2448ff9640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f242c06a080 0x7f242c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.381+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f24440aae20 con 0x7f244c149e70 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.381+0000 7f244b7fe640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f242c06a080 0x7f242c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:14.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.382+0000 7f244b7fe640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f242c06a080 0x7f242c06c540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f243c000ef0 tx=0x7f243c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:14.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.385+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2444075f60 con 0x7f244c149e70 2023-08-25T09:58:14.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.613+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f244c04e420 con 0x7f244c149e70 2023-08-25T09:58:14.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.613+0000 7f2448ff9640 1 -- 172.21.15.71:0/3943649012 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f2444039030 con 0x7f244c149e70 2023-08-25T09:58:14.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:14.616 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:07.958477+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:14.616 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 66 2023-08-25T09:58:14.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f242c06a080 msgr2=0x7f242c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:14.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f242c06a080 0x7f242c06c540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f243c000ef0 tx=0x7f243c009040 comp rx=0 tx=0).stop 2023-08-25T09:58:14.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 msgr2=0x7f244c14a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:14.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 0x7f244c14a320 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2444002230 tx=0x7f2444009750 comp rx=0 tx=0).stop 2023-08-25T09:58:14.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 shutdown_connections 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f244c13c700 0x7f244c13eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f242c06a080 0x7f242c06c540 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f244c149e70 0x7f244c14a320 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 --2- 172.21.15.71:0/3943649012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2438004d00 0x7f244c13e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 >> 172.21.15.71:0/3943649012 conn(0x7f244c06bff0 msgr2=0x7f244c14e370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 shutdown_connections 2023-08-25T09:58:14.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:14.616+0000 7f2451c2d640 1 -- 172.21.15.71:0/3943649012 wait complete. 2023-08-25T09:58:14.638 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:58:16.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:16.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.862+0000 7f2db89f3640 1 Processor -- start 2023-08-25T09:58:16.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.862+0000 7f2db89f3640 1 -- start start 2023-08-25T09:58:16.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db89f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2db41510e0 0x7f2db40c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:16.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db89f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40c1dc0 con 0x7f2db40c3a40 2023-08-25T09:58:16.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db89f3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4156e00 con 0x7f2db40c3660 2023-08-25T09:58:16.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db89f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db4157a80 con 0x7f2db41510e0 2023-08-25T09:58:16.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db1d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f2db40c3a40 0x7f2db41509d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49490/0 (socket says 172.21.15.71:49490) 2023-08-25T09:58:16.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db1d9b640 1 -- 172.21.15.71:0/109432203 learned_addr learned my addr 172.21.15.71:0/109432203 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:16.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.864+0000 7f2db2d9d640 1 --2- 172.21.15.71:0/109432203 >> v2:172.21.15.71:3300/0 conn(0x7f2db41510e0 0x7f2db40c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:16.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.865+0000 7f2db2d9d640 1 -- 172.21.15.71:0/109432203 >> v1:172.21.15.164:6789/0 conn(0x7f2db40c3660 legacy=0x7f2db406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:16.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.865+0000 7f2db2d9d640 1 -- 172.21.15.71:0/109432203 >> v1:172.21.15.71:6789/0 conn(0x7f2db40c3a40 legacy=0x7f2db41509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:16.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.865+0000 7f2db2d9d640 1 -- 172.21.15.71:0/109432203 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db41570e0 con 0x7f2db41510e0 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.865+0000 7f2db159a640 1 -- 172.21.15.71:0/109432203 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644096793 0 0) 0x7f2db40c1dc0 con 0x7f2db40c3a40 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.866+0000 7f2db2d9d640 1 --2- 172.21.15.71:0/109432203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db41510e0 0x7f2db40c1710 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2dac0097a0 tx=0x7f2dac02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c79dfc318da18c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.866+0000 7f2db159a640 1 -- 172.21.15.71:0/109432203 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac02f810 con 0x7f2db41510e0 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.866+0000 7f2db159a640 1 -- 172.21.15.71:0/109432203 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dac02f970 con 0x7f2db41510e0 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.866+0000 7f2db159a640 1 -- 172.21.15.71:0/109432203 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dac02fd00 con 0x7f2db41510e0 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 -- 172.21.15.71:0/109432203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db41510e0 msgr2=0x7f2db40c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:16.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 --2- 172.21.15.71:0/109432203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db41510e0 0x7f2db40c1710 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2dac0097a0 tx=0x7f2dac02b9f0 comp rx=0 tx=0).stop 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 -- 172.21.15.71:0/109432203 shutdown_connections 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 --2- 172.21.15.71:0/109432203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db41510e0 0x7f2db40c1710 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 -- 172.21.15.71:0/109432203 >> 172.21.15.71:0/109432203 conn(0x7f2db40654f0 msgr2=0x7f2db4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 -- 172.21.15.71:0/109432203 shutdown_connections 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.867+0000 7f2db89f3640 1 -- 172.21.15.71:0/109432203 wait complete. 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.869+0000 7f2db89f3640 1 Processor -- start 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.869+0000 7f2db89f3640 1 -- start start 2023-08-25T09:58:16.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.870+0000 7f2db89f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 0x7f2db4154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db89f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db40c3a40 0x7f2db4148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db89f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2db41510e0 0x7f2db41f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db89f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db41599e0 con 0x7f2db40c3660 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db89f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4070630 con 0x7f2db41510e0 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db89f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db415ad20 con 0x7f2db40c3a40 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 0x7f2db4154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:16.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 0x7f2db4154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49000/0 (socket says 172.21.15.71:49000) 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 -- 172.21.15.71:0/442122607 learned_addr learned my addr 172.21.15.71:0/442122607 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db2d9d640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2db41510e0 0x7f2db41f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db1d9b640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db40c3a40 0x7f2db4148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 -- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db40c3a40 msgr2=0x7f2db4148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db40c3a40 0x7f2db4148a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 -- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2db41510e0 msgr2=0x7f2db41f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:16.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2db41510e0 0x7f2db41f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.871+0000 7f2db259c640 1 -- 172.21.15.71:0/442122607 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac009400 con 0x7f2db40c3660 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.872+0000 7f2db259c640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 0x7f2db4154ab0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c400 tx=0x7f2da400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.872+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40052d0 con 0x7f2db40c3660 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.872+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da4005430 con 0x7f2db40c3660 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.872+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4003830 con 0x7f2db40c3660 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.872+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db4159fb0 con 0x7f2db40c3660 2023-08-25T09:58:16.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.872+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db415a3f0 con 0x7f2db40c3660 2023-08-25T09:58:16.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.874+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db4069160 con 0x7f2db40c3660 2023-08-25T09:58:16.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.874+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2da4003990 con 0x7f2db40c3660 2023-08-25T09:58:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.875+0000 7f2daaffd640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d9806a050 0x7f2d9806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.875+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f2da4087fe0 con 0x7f2db40c3660 2023-08-25T09:58:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.876+0000 7f2db1d9b640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d9806a050 0x7f2d9806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.876+0000 7f2db1d9b640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d9806a050 0x7f2d9806c510 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2db4156d30 tx=0x7f2d9c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:16.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:16.880+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2da40531f0 con 0x7f2db40c3660 2023-08-25T09:58:17.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.110+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2db4154680 con 0x7f2db40c3660 2023-08-25T09:58:17.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.111+0000 7f2daaffd640 1 -- 172.21.15.71:0/442122607 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f2da4056810 con 0x7f2db40c3660 2023-08-25T09:58:17.113 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:17.113 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:14.744210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:17.114 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 67 2023-08-25T09:58:17.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d9806a050 msgr2=0x7f2d9806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:17.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d9806a050 0x7f2d9806c510 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2db4156d30 tx=0x7f2d9c009040 comp rx=0 tx=0).stop 2023-08-25T09:58:17.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 msgr2=0x7f2db4154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:17.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 0x7f2db4154ab0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2da400c400 tx=0x7f2da400c8c0 comp rx=0 tx=0).stop 2023-08-25T09:58:17.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 shutdown_connections 2023-08-25T09:58:17.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2db41510e0 0x7f2db41f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:17.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2d9806a050 0x7f2d9806c510 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:17.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2db40c3a40 0x7f2db4148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:17.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 --2- 172.21.15.71:0/442122607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2db40c3660 0x7f2db4154ab0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:17.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 >> 172.21.15.71:0/442122607 conn(0x7f2db40654f0 msgr2=0x7f2db40c97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:17.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 shutdown_connections 2023-08-25T09:58:17.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:17.114+0000 7f2db89f3640 1 -- 172.21.15.71:0/442122607 wait complete. 2023-08-25T09:58:17.135 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=2, replacing gid=4290 2023-08-25T09:58:17.135 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.c 2023-08-25T09:58:21.136 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-08-25T09:58:21.137 INFO:tasks.ceph.mds.c:Restarting daemon 2023-08-25T09:58:21.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-08-25T09:58:21.146 INFO:tasks.ceph.mds.c:Started 2023-08-25T09:58:21.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:21.315 INFO:tasks.ceph.mds.c.smithi071.stdout:starting mds.c at 2023-08-25T09:58:21.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/34' 2023-08-25T09:58:21.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/38' 2023-08-25T09:58:21.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/11' 2023-08-25T09:58:21.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/16' 2023-08-25T09:58:21.316 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/32' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/30' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/14' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/18' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/13' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/33' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/17' 2023-08-25T09:58:21.317 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/35' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './196/37' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './196' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/19' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/15' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/31' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/36' 2023-08-25T09:58:21.318 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/20' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/12' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/39' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/10' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/34' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/38' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/11' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/16' 2023-08-25T09:58:21.319 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/32' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/30' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/14' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/18' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/13' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/33' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/17' 2023-08-25T09:58:21.320 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/35' 2023-08-25T09:58:21.321 INFO:tasks.workunit.client.0.smithi071.stdout:removed './197/37' 2023-08-25T09:58:21.321 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './197' 2023-08-25T09:58:21.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/19' 2023-08-25T09:58:21.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/15' 2023-08-25T09:58:21.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/31' 2023-08-25T09:58:21.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/36' 2023-08-25T09:58:21.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/20' 2023-08-25T09:58:21.322 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/12' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/39' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/10' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/34' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/38' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/11' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/16' 2023-08-25T09:58:21.323 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/32' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/30' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/14' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/18' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/13' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/33' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/17' 2023-08-25T09:58:21.324 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/35' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './198/37' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './198' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/19' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/15' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/31' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/36' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/20' 2023-08-25T09:58:21.325 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/12' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/39' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/10' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/34' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/38' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/11' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/16' 2023-08-25T09:58:21.326 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/32' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/30' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/14' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/18' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/13' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/33' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/17' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/35' 2023-08-25T09:58:21.327 INFO:tasks.workunit.client.0.smithi071.stdout:removed './199/37' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './199' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/21' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/27' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/31' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/36' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/20' 2023-08-25T09:58:21.328 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/39' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/34' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/38' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/28' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/24' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/29' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/32' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/22' 2023-08-25T09:58:21.329 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/25' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/30' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/33' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/23' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/35' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/26' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './200/37' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './200' 2023-08-25T09:58:21.330 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/21' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/27' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/31' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/36' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/20' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/39' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/34' 2023-08-25T09:58:21.331 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/38' 2023-08-25T09:58:21.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/28' 2023-08-25T09:58:21.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/24' 2023-08-25T09:58:21.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/29' 2023-08-25T09:58:21.332 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/32' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/22' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/25' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/30' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/33' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/23' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/35' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/26' 2023-08-25T09:58:21.333 INFO:tasks.workunit.client.0.smithi071.stdout:removed './201/37' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './201' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/21' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/27' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/31' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/36' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/20' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/39' 2023-08-25T09:58:21.334 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/34' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/38' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/28' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/24' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/29' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/32' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/22' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/25' 2023-08-25T09:58:21.335 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/30' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/33' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/23' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/35' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/26' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './202/37' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './202' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/21' 2023-08-25T09:58:21.336 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/27' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/31' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/36' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/20' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/39' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/34' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/38' 2023-08-25T09:58:21.337 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/28' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/24' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/29' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/32' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/22' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/25' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/30' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/33' 2023-08-25T09:58:21.338 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/23' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/35' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/26' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './203/37' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './203' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/21' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/27' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/31' 2023-08-25T09:58:21.339 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/36' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/20' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/39' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/34' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/38' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/28' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/24' 2023-08-25T09:58:21.340 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/29' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/32' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/22' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/25' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/30' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/33' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/23' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/35' 2023-08-25T09:58:21.341 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/26' 2023-08-25T09:58:21.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './204/37' 2023-08-25T09:58:21.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './204' 2023-08-25T09:58:21.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/21' 2023-08-25T09:58:21.342 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/27' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/31' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/36' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/20' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/39' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/34' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/38' 2023-08-25T09:58:21.343 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/28' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/24' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/29' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/32' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/22' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/25' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/30' 2023-08-25T09:58:21.344 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/33' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/23' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/35' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/26' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './205/37' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './205' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/21' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/27' 2023-08-25T09:58:21.345 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/31' 2023-08-25T09:58:21.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/36' 2023-08-25T09:58:21.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/20' 2023-08-25T09:58:21.346 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/39' 2023-08-25T09:58:21.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.351+0000 7f180f942640 1 Processor -- start 2023-08-25T09:58:21.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.351+0000 7f180f942640 1 -- start start 2023-08-25T09:58:21.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180f942640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f18080c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:21.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180f942640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18080c3a40 0x7f180806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:21.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180f942640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080c1dc0 con 0x7f18081509d0 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180f942640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1808157a80 con 0x7f18080c3a40 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180f942640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180806cf70 con 0x7f18080c3660 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18080c3a40 0x7f180806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f18080c3a40 0x7f180806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56046/0 (socket says 172.21.15.71:56046) 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 -- 172.21.15.71:0/3917370452 learned_addr learned my addr 172.21.15.71:0/3917370452 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180d6de640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f18080c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 -- 172.21.15.71:0/3917370452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 msgr2=0x7f18080c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:21.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f18080c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 -- 172.21.15.71:0/3917370452 >> v1:172.21.15.164:6789/0 conn(0x7f18081509d0 legacy=0x7f1808150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:21.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180cedd640 1 -- 172.21.15.71:0/3917370452 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1808156e00 con 0x7f18080c3a40 2023-08-25T09:58:21.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.353+0000 7f180d6de640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f18080c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:21.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.355+0000 7f180cedd640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18080c3a40 0x7f180806ca30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009870 tx=0x7f17f802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96aa8b1ad2aa4d6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:21.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.355+0000 7f17fffff640 1 -- 172.21.15.71:0/3917370452 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8030840 con 0x7f18080c3a40 2023-08-25T09:58:21.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.355+0000 7f17fffff640 1 -- 172.21.15.71:0/3917370452 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f80309a0 con 0x7f18080c3a40 2023-08-25T09:58:21.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.355+0000 7f17fffff640 1 -- 172.21.15.71:0/3917370452 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8030d30 con 0x7f18080c3a40 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 -- 172.21.15.71:0/3917370452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18080c3a40 msgr2=0x7f180806ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18080c3a40 0x7f180806ca30 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009870 tx=0x7f17f802b980 comp rx=0 tx=0).stop 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 -- 172.21.15.71:0/3917370452 shutdown_connections 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18080c3a40 0x7f180806ca30 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 --2- 172.21.15.71:0/3917370452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f18080c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 -- 172.21.15.71:0/3917370452 >> 172.21.15.71:0/3917370452 conn(0x7f18080654f0 msgr2=0x7f1808065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:21.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 -- 172.21.15.71:0/3917370452 shutdown_connections 2023-08-25T09:58:21.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.356+0000 7f180f942640 1 -- 172.21.15.71:0/3917370452 wait complete. 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.358+0000 7f180f942640 1 Processor -- start 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.358+0000 7f180f942640 1 -- start start 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180f942640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f180814f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180f942640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18080c3a40 0x7f1808151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180f942640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 0x7f1808152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180f942640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081528b0 con 0x7f18081509d0 2023-08-25T09:58:21.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180f942640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081515f0 con 0x7f18080c3a40 2023-08-25T09:58:21.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180f942640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180814c190 con 0x7f18080c3660 2023-08-25T09:58:21.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 0x7f1808152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:21.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 0x7f1808152010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56050/0 (socket says 172.21.15.71:56050) 2023-08-25T09:58:21.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 -- 172.21.15.71:0/2591332254 learned_addr learned my addr 172.21.15.71:0/2591332254 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:21.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180d6de640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f180814f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:21.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180cedd640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18080c3a40 0x7f1808151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:21.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 -- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 msgr2=0x7f180814f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:21.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f180814f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 -- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18080c3a40 msgr2=0x7f1808151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:21.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18080c3a40 0x7f1808151ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.359+0000 7f180dedf640 1 -- 172.21.15.71:0/2591332254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f80094d0 con 0x7f18081509d0 2023-08-25T09:58:21.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.360+0000 7f180dedf640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 0x7f1808152010 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f180000c780 tx=0x7f180000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.360+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800004b60 con 0x7f18081509d0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.360+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000ce90 con 0x7f18081509d0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.360+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000a850 con 0x7f18081509d0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.360+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180814c470 con 0x7f18081509d0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.360+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180814ca30 con 0x7f18081509d0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.362+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18080c29c0 con 0x7f18081509d0 2023-08-25T09:58:21.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.362+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f180000a9b0 con 0x7f18081509d0 2023-08-25T09:58:21.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.363+0000 7f17fdffb640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f17ec06a080 0x7f17ec06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:21.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.363+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f1800088b50 con 0x7f18081509d0 2023-08-25T09:58:21.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.364+0000 7f180d6de640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f17ec06a080 0x7f17ec06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:21.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.364+0000 7f180d6de640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f17ec06a080 0x7f17ec06c540 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f180400b840 tx=0x7f1804008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:21.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.369+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1800053dc0 con 0x7f18081509d0 2023-08-25T09:58:21.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.617+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18081517b0 con 0x7f18081509d0 2023-08-25T09:58:21.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.619+0000 7f17fdffb640 1 -- 172.21.15.71:0/2591332254 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18000573e0 con 0x7f18081509d0 2023-08-25T09:58:21.621 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:21.621 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:17.960837+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:21.621 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 68 2023-08-25T09:58:21.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.621+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f17ec06a080 msgr2=0x7f17ec06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:21.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.621+0000 7f180f942640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f17ec06a080 0x7f17ec06c540 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f180400b840 tx=0x7f1804008040 comp rx=0 tx=0).stop 2023-08-25T09:58:21.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.621+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 msgr2=0x7f1808152010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:21.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.621+0000 7f180f942640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 0x7f1808152010 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f180000c780 tx=0x7f180000cc40 comp rx=0 tx=0).stop 2023-08-25T09:58:21.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 shutdown_connections 2023-08-25T09:58:21.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f18080c3a40 0x7f1808151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f17ec06a080 0x7f17ec06c540 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f18081509d0 0x7f1808152010 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 --2- 172.21.15.71:0/2591332254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f18080c3660 0x7f180814f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:21.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 >> 172.21.15.71:0/2591332254 conn(0x7f18080654f0 msgr2=0x7f18080c92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:21.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 shutdown_connections 2023-08-25T09:58:21.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:21.622+0000 7f180f942640 1 -- 172.21.15.71:0/2591332254 wait complete. 2023-08-25T09:58:21.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-08-25T09:58:23.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:58:23.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.840+0000 7f1c74399640 1 Processor -- start 2023-08-25T09:58:23.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.840+0000 7f1c74399640 1 -- start start 2023-08-25T09:58:23.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c74399640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 0x7f1c6c06a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c74399640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c0b1830 con 0x7f1c6c0c9e00 2023-08-25T09:58:23.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c74399640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c06aaa0 con 0x7f1c6c0cb5b0 2023-08-25T09:58:23.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c74399640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c159100 con 0x7f1c6c1531e0 2023-08-25T09:58:23.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c71934640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1c6c0c9e00 0x7f1c6c0ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41820/0 (socket says 172.21.15.71:41820) 2023-08-25T09:58:23.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c71934640 1 -- 172.21.15.71:0/916989481 learned_addr learned my addr 172.21.15.71:0/916989481 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:58:23.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.843+0000 7f1c72135640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 0x7f1c6c06a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506841230 0 0) 0x7f1c6c0b1830 con 0x7f1c6c0c9e00 2023-08-25T09:58:23.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c58003520 con 0x7f1c6c0c9e00 2023-08-25T09:58:23.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937746107 0 0) 0x7f1c58003520 con 0x7f1c6c0c9e00 2023-08-25T09:58:23.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 msgr2=0x7f1c6c06a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:23.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 0x7f1c6c06a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:23.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 >> v1:172.21.15.164:6789/0 conn(0x7f1c6c0cb5b0 legacy=0x7f1c6c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c1593e0 con 0x7f1c6c0c9e00 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f1c5c001670 con 0x7f1c6c0c9e00 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 >> v1:172.21.15.71:6789/0 conn(0x7f1c6c0c9e00 legacy=0x7f1c6c0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 0x7f1c58003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58003f90 0x7f1c580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c58004d00 0x7f1c58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c159100 con 0x7f1c58004d00 2023-08-25T09:58:23.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5800bae0 con 0x7f1c58003f90 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.844+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5800bca0 con 0x7f1c6c1531e0 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72936640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c58004d00 0x7f1c58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72135640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 0x7f1c58003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c71934640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58003f90 0x7f1c580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72936640 1 -- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 msgr2=0x7f1c58003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72936640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1531e0 0x7f1c58003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:23.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72936640 1 -- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58003f90 msgr2=0x7f1c580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:58:23.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72936640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58003f90 0x7f1c580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:23.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c72936640 1 -- 172.21.15.71:0/916989481 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c58003520 con 0x7f1c58004d00 2023-08-25T09:58:23.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.845+0000 7f1c71934640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58003f90 0x7f1c580047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:58:23.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.846+0000 7f1c72936640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c58004d00 0x7f1c58005180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1c64001e70 tx=0x7f1c580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679c5859746908e5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:23.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64003bc0 con 0x7f1c58004d00 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64003d20 con 0x7f1c58004d00 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c71133640 1 -- 172.21.15.71:0/916989481 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64005310 con 0x7f1c58004d00 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c74399640 1 -- 172.21.15.71:0/916989481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c58004d00 msgr2=0x7f1c58005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c74399640 1 --2- 172.21.15.71:0/916989481 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c58004d00 0x7f1c58005180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1c64001e70 tx=0x7f1c580057f0 comp rx=0 tx=0).stop 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c72936640 1 -- 172.21.15.71:0/916989481 reap_dead start 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.847+0000 7f1c74399640 1 -- 172.21.15.71:0/916989481 shutdown_connections 2023-08-25T09:58:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.848+0000 7f1c74399640 1 -- 172.21.15.71:0/916989481 >> 172.21.15.71:0/916989481 conn(0x7f1c6c06b240 msgr2=0x7f1c6c06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.848+0000 7f1c74399640 1 -- 172.21.15.71:0/916989481 shutdown_connections 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.848+0000 7f1c74399640 1 -- 172.21.15.71:0/916989481 wait complete. 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.849+0000 7f1c74399640 1 Processor -- start 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.850+0000 7f1c74399640 1 -- start start 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c74399640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58004d00 0x7f1c6c156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c74399640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1f23b0 0x7f1c6c0cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c74399640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 0x7f1c6c1551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c74399640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c1538a0 con 0x7f1c6c154d40 2023-08-25T09:58:23.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c74399640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c157510 con 0x7f1c58004d00 2023-08-25T09:58:23.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c74399640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c1ee430 con 0x7f1c6c1f23b0 2023-08-25T09:58:23.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c72936640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 0x7f1c6c1551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c72936640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 0x7f1c6c1551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:56076/0 (socket says 172.21.15.71:56076) 2023-08-25T09:58:23.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c72936640 1 -- 172.21.15.71:0/4142072383 learned_addr learned my addr 172.21.15.71:0/4142072383 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:58:23.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c71934640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1f23b0 0x7f1c6c0cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.851+0000 7f1c72135640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58004d00 0x7f1c6c156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c72936640 1 -- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1f23b0 msgr2=0x7f1c6c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:23.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c72936640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1f23b0 0x7f1c6c0cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:23.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c72936640 1 -- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58004d00 msgr2=0x7f1c6c156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c72936640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58004d00 0x7f1c6c156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c72936640 1 -- 172.21.15.71:0/4142072383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c1ef400 con 0x7f1c6c154d40 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c72936640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 0x7f1c6c1551c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1c64001f90 tx=0x7f1c640014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64006da0 con 0x7f1c6c154d40 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c64007e00 con 0x7f1c6c154d40 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6402f660 con 0x7f1c6c154d40 2023-08-25T09:58:23.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.852+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c157230 con 0x7f1c6c154d40 2023-08-25T09:58:23.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.853+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c1ee920 con 0x7f1c6c154d40 2023-08-25T09:58:23.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.854+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c069160 con 0x7f1c6c154d40 2023-08-25T09:58:23.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.855+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1c6402f7c0 con 0x7f1c6c154d40 2023-08-25T09:58:23.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.856+0000 7f1c62ffd640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c5006a080 0x7f1c5006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:58:23.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.857+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f1c640aa110 con 0x7f1c6c154d40 2023-08-25T09:58:23.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.857+0000 7f1c72135640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c5006a080 0x7f1c5006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:58:23.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.857+0000 7f1c72135640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c5006a080 0x7f1c5006c540 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009a10 tx=0x7f1c68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:58:23.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:23.861+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1c64075400 con 0x7f1c6c154d40 2023-08-25T09:58:24.070 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/34' 2023-08-25T09:58:24.070 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/38' 2023-08-25T09:58:24.070 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/28' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/24' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/29' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/32' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/22' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/25' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/30' 2023-08-25T09:58:24.071 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/33' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/23' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/35' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/26' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './206/37' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './206' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/21' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/27' 2023-08-25T09:58:24.072 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/31' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/36' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/20' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/39' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/34' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/38' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/28' 2023-08-25T09:58:24.073 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/24' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/29' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/32' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/22' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/25' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/30' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/33' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/23' 2023-08-25T09:58:24.074 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/35' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/26' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './207/37' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './207' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/21' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/27' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/31' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/36' 2023-08-25T09:58:24.075 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/20' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/39' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/34' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/38' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/28' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/24' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/29' 2023-08-25T09:58:24.076 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/32' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/22' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/25' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/30' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/33' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/23' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/35' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/26' 2023-08-25T09:58:24.077 INFO:tasks.workunit.client.0.smithi071.stdout:removed './208/37' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './208' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/21' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/27' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/31' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/36' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/20' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/39' 2023-08-25T09:58:24.078 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/34' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/38' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/28' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/24' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/29' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/32' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/22' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/25' 2023-08-25T09:58:24.079 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/30' 2023-08-25T09:58:24.080 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/33' 2023-08-25T09:58:24.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/23' 2023-08-25T09:58:24.083 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/35' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/26' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './209/37' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './209' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/21' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/27' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/31' 2023-08-25T09:58:24.084 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/36' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/20' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/39' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/34' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/38' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/28' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/24' 2023-08-25T09:58:24.085 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/29' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/32' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/22' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/25' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/30' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/33' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/23' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/35' 2023-08-25T09:58:24.086 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/26' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './210/37' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './210' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/21' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/27' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/31' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/36' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/20' 2023-08-25T09:58:24.087 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/39' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/34' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/38' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/28' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/24' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/29' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/32' 2023-08-25T09:58:24.088 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/22' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/25' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/30' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/33' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/23' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/35' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/26' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed './211/37' 2023-08-25T09:58:24.089 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './211' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/21' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/27' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/31' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/36' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/20' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/39' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/34' 2023-08-25T09:58:24.090 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/38' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/28' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/24' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/29' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/32' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/22' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/25' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/30' 2023-08-25T09:58:24.091 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/33' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/23' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/35' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/26' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './212/37' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './212' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/21' 2023-08-25T09:58:24.092 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/27' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/31' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/36' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/20' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/39' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/34' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/38' 2023-08-25T09:58:24.093 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/28' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/24' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/29' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/32' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/22' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/25' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/30' 2023-08-25T09:58:24.094 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/33' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/23' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/35' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/26' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './213/37' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './213' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/21' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/27' 2023-08-25T09:58:24.095 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/31' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/36' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/20' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/39' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/34' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/38' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/28' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/24' 2023-08-25T09:58:24.096 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/29' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/32' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/22' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/25' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/30' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/33' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/23' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/35' 2023-08-25T09:58:24.097 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/26' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './214/37' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './214' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/21' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/27' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/31' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/36' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/20' 2023-08-25T09:58:24.098 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/39' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/34' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/38' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/28' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/24' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/29' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/32' 2023-08-25T09:58:24.099 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/22' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/25' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/30' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/33' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/23' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/35' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/26' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed './215/37' 2023-08-25T09:58:24.100 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './215' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/21' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/27' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/31' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/36' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/20' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/39' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/34' 2023-08-25T09:58:24.101 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/38' 2023-08-25T09:58:24.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.099+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c6c06b620 con 0x7f1c6c154d40 2023-08-25T09:58:24.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.101+0000 7f1c62ffd640 1 -- 172.21.15.71:0/4142072383 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f1c64047090 con 0x7f1c6c154d40 2023-08-25T09:58:24.103 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:58:24.103 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:58:23.738771+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:58:24.103 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 70 2023-08-25T09:58:24.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c5006a080 msgr2=0x7f1c5006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:24.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c5006a080 0x7f1c5006c540 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009a10 tx=0x7f1c68008040 comp rx=0 tx=0).stop 2023-08-25T09:58:24.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 msgr2=0x7f1c6c1551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:58:24.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 0x7f1c6c1551c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1c64001f90 tx=0x7f1c640014f0 comp rx=0 tx=0).stop 2023-08-25T09:58:24.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 shutdown_connections 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1c5006a080 0x7f1c5006c540 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c6c154d40 0x7f1c6c1551c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c6c1f23b0 0x7f1c6c0cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 --2- 172.21.15.71:0/4142072383 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c58004d00 0x7f1c6c156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 >> 172.21.15.71:0/4142072383 conn(0x7f1c6c06b240 msgr2=0x7f1c6c0ca510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 shutdown_connections 2023-08-25T09:58:24.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:58:24.103+0000 7f1c74399640 1 -- 172.21.15.71:0/4142072383 wait complete. 2023-08-25T09:58:24.123 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-08-25T09:58:24.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2023-08-25T09:58:24.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:58:24.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:58:27.576 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/28' 2023-08-25T09:58:27.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/24' 2023-08-25T09:58:27.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/29' 2023-08-25T09:58:27.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/32' 2023-08-25T09:58:27.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/22' 2023-08-25T09:58:27.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/25' 2023-08-25T09:58:27.577 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/30' 2023-08-25T09:58:27.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/33' 2023-08-25T09:58:27.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/23' 2023-08-25T09:58:27.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/35' 2023-08-25T09:58:27.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/26' 2023-08-25T09:58:27.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed './216/37' 2023-08-25T09:58:27.578 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './216' 2023-08-25T09:58:27.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/21' 2023-08-25T09:58:27.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/27' 2023-08-25T09:58:27.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/31' 2023-08-25T09:58:27.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/36' 2023-08-25T09:58:27.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/20' 2023-08-25T09:58:27.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/39' 2023-08-25T09:58:27.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/34' 2023-08-25T09:58:27.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/38' 2023-08-25T09:58:27.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/28' 2023-08-25T09:58:27.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/24' 2023-08-25T09:58:27.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/29' 2023-08-25T09:58:27.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/32' 2023-08-25T09:58:27.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/22' 2023-08-25T09:58:27.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/25' 2023-08-25T09:58:27.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/30' 2023-08-25T09:58:27.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/33' 2023-08-25T09:58:27.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/23' 2023-08-25T09:58:27.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/35' 2023-08-25T09:58:27.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/26' 2023-08-25T09:58:27.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './217/37' 2023-08-25T09:58:27.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './217' 2023-08-25T09:58:27.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/21' 2023-08-25T09:58:27.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/27' 2023-08-25T09:58:27.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/31' 2023-08-25T09:58:27.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/36' 2023-08-25T09:58:27.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/20' 2023-08-25T09:58:27.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/39' 2023-08-25T09:58:27.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/34' 2023-08-25T09:58:27.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/38' 2023-08-25T09:58:27.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/28' 2023-08-25T09:58:27.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/24' 2023-08-25T09:58:27.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/29' 2023-08-25T09:58:27.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/32' 2023-08-25T09:58:27.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/22' 2023-08-25T09:58:27.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/25' 2023-08-25T09:58:27.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/30' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/33' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/23' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/35' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/26' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './218/37' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './218' 2023-08-25T09:58:27.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/21' 2023-08-25T09:58:27.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/27' 2023-08-25T09:58:27.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/31' 2023-08-25T09:58:27.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/36' 2023-08-25T09:58:27.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/20' 2023-08-25T09:58:27.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/39' 2023-08-25T09:58:27.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/34' 2023-08-25T09:58:27.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/38' 2023-08-25T09:58:27.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/28' 2023-08-25T09:58:27.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/24' 2023-08-25T09:58:27.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/29' 2023-08-25T09:58:27.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/32' 2023-08-25T09:58:27.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/22' 2023-08-25T09:58:27.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/25' 2023-08-25T09:58:27.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/30' 2023-08-25T09:58:27.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/33' 2023-08-25T09:58:27.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/23' 2023-08-25T09:58:27.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/35' 2023-08-25T09:58:27.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/26' 2023-08-25T09:58:27.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './219/37' 2023-08-25T09:58:27.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './219' 2023-08-25T09:58:27.594 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/21' 2023-08-25T09:58:27.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/27' 2023-08-25T09:58:27.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/31' 2023-08-25T09:58:27.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/36' 2023-08-25T09:58:27.595 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/20' 2023-08-25T09:58:27.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/39' 2023-08-25T09:58:27.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/34' 2023-08-25T09:58:27.596 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/38' 2023-08-25T09:58:27.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/28' 2023-08-25T09:58:27.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/24' 2023-08-25T09:58:27.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/29' 2023-08-25T09:58:27.597 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/32' 2023-08-25T09:58:27.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/22' 2023-08-25T09:58:27.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/25' 2023-08-25T09:58:27.598 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/30' 2023-08-25T09:58:27.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/33' 2023-08-25T09:58:27.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/23' 2023-08-25T09:58:27.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/35' 2023-08-25T09:58:27.599 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/26' 2023-08-25T09:58:27.600 INFO:tasks.workunit.client.0.smithi071.stdout:removed './220/37' 2023-08-25T09:58:27.600 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './220' 2023-08-25T09:58:27.600 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/21' 2023-08-25T09:58:27.600 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/27' 2023-08-25T09:58:27.601 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/31' 2023-08-25T09:58:27.601 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/36' 2023-08-25T09:58:27.601 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/20' 2023-08-25T09:58:27.602 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/39' 2023-08-25T09:58:27.602 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/34' 2023-08-25T09:58:27.602 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/38' 2023-08-25T09:58:27.603 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/28' 2023-08-25T09:58:27.603 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/24' 2023-08-25T09:58:27.603 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/29' 2023-08-25T09:58:27.603 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/32' 2023-08-25T09:58:27.604 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/22' 2023-08-25T09:58:27.604 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/25' 2023-08-25T09:58:27.604 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/30' 2023-08-25T09:58:27.604 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/33' 2023-08-25T09:58:27.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/23' 2023-08-25T09:58:27.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/35' 2023-08-25T09:58:27.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/26' 2023-08-25T09:58:27.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './221/37' 2023-08-25T09:58:27.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './221' 2023-08-25T09:58:27.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/21' 2023-08-25T09:58:27.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/27' 2023-08-25T09:58:27.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/31' 2023-08-25T09:58:27.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/36' 2023-08-25T09:58:27.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/20' 2023-08-25T09:58:27.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/39' 2023-08-25T09:58:27.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/34' 2023-08-25T09:58:27.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/38' 2023-08-25T09:58:27.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/28' 2023-08-25T09:58:27.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/24' 2023-08-25T09:58:27.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/29' 2023-08-25T09:58:27.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/32' 2023-08-25T09:58:27.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/22' 2023-08-25T09:58:27.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/25' 2023-08-25T09:58:27.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/30' 2023-08-25T09:58:27.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/33' 2023-08-25T09:58:27.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/23' 2023-08-25T09:58:27.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/35' 2023-08-25T09:58:27.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/26' 2023-08-25T09:58:27.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './222/37' 2023-08-25T09:58:27.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './222' 2023-08-25T09:58:27.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/21' 2023-08-25T09:58:27.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/27' 2023-08-25T09:58:27.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/31' 2023-08-25T09:58:27.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/36' 2023-08-25T09:58:27.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/20' 2023-08-25T09:58:27.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/39' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/34' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/38' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/28' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/24' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/29' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/32' 2023-08-25T09:58:27.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/22' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/25' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/30' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/33' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/23' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/35' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/26' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './223/37' 2023-08-25T09:58:27.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './223' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/21' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/27' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/31' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/36' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/20' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/39' 2023-08-25T09:58:27.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/34' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/38' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/28' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/24' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/29' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/32' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/22' 2023-08-25T09:58:27.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/25' 2023-08-25T09:58:27.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/30' 2023-08-25T09:58:27.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/33' 2023-08-25T09:58:27.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/23' 2023-08-25T09:58:27.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/35' 2023-08-25T09:58:27.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/26' 2023-08-25T09:58:27.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './224/37' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './224' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/21' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/27' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/31' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/36' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/20' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/39' 2023-08-25T09:58:27.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/34' 2023-08-25T09:58:27.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/38' 2023-08-25T09:58:27.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/28' 2023-08-25T09:58:27.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/24' 2023-08-25T09:58:27.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/29' 2023-08-25T09:58:27.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/32' 2023-08-25T09:58:27.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/22' 2023-08-25T09:58:27.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/25' 2023-08-25T09:58:27.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/30' 2023-08-25T09:58:27.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/33' 2023-08-25T09:58:27.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/23' 2023-08-25T09:58:27.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/35' 2023-08-25T09:58:27.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/26' 2023-08-25T09:58:27.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './225/37' 2023-08-25T09:58:27.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './225' 2023-08-25T09:58:27.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/21' 2023-08-25T09:58:27.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/27' 2023-08-25T09:58:27.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/31' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/36' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/20' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/39' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/34' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/38' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/28' 2023-08-25T09:58:27.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/24' 2023-08-25T09:58:29.743 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/29' 2023-08-25T09:58:29.744 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/32' 2023-08-25T09:58:29.744 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/22' 2023-08-25T09:58:29.744 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/25' 2023-08-25T09:58:29.744 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/30' 2023-08-25T09:58:29.744 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/33' 2023-08-25T09:58:29.745 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/23' 2023-08-25T09:58:29.745 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/35' 2023-08-25T09:58:29.745 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/26' 2023-08-25T09:58:29.745 INFO:tasks.workunit.client.0.smithi071.stdout:removed './226/37' 2023-08-25T09:58:29.745 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './226' 2023-08-25T09:58:29.745 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/21' 2023-08-25T09:58:29.746 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/27' 2023-08-25T09:58:29.746 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/31' 2023-08-25T09:58:29.746 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/36' 2023-08-25T09:58:29.746 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/20' 2023-08-25T09:58:29.746 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/39' 2023-08-25T09:58:29.746 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/34' 2023-08-25T09:58:29.747 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/38' 2023-08-25T09:58:29.747 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/28' 2023-08-25T09:58:29.747 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/24' 2023-08-25T09:58:29.748 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/29' 2023-08-25T09:58:29.748 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/32' 2023-08-25T09:58:29.749 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/22' 2023-08-25T09:58:29.749 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/25' 2023-08-25T09:58:29.749 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/30' 2023-08-25T09:58:29.749 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/33' 2023-08-25T09:58:29.750 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/23' 2023-08-25T09:58:29.750 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/35' 2023-08-25T09:58:29.750 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/26' 2023-08-25T09:58:29.751 INFO:tasks.workunit.client.0.smithi071.stdout:removed './227/37' 2023-08-25T09:58:29.751 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './227' 2023-08-25T09:58:29.751 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/21' 2023-08-25T09:58:29.751 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/27' 2023-08-25T09:58:29.752 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/31' 2023-08-25T09:58:29.752 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/36' 2023-08-25T09:58:29.752 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/20' 2023-08-25T09:58:29.752 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/39' 2023-08-25T09:58:29.753 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/34' 2023-08-25T09:58:29.753 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/38' 2023-08-25T09:58:29.753 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/28' 2023-08-25T09:58:29.754 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/24' 2023-08-25T09:58:29.754 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/29' 2023-08-25T09:58:29.754 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/32' 2023-08-25T09:58:29.754 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/22' 2023-08-25T09:58:29.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/25' 2023-08-25T09:58:29.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/30' 2023-08-25T09:58:29.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/33' 2023-08-25T09:58:29.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/23' 2023-08-25T09:58:29.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/35' 2023-08-25T09:58:29.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/26' 2023-08-25T09:58:29.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './228/37' 2023-08-25T09:58:29.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './228' 2023-08-25T09:58:29.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/21' 2023-08-25T09:58:29.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/27' 2023-08-25T09:58:29.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/31' 2023-08-25T09:58:29.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/36' 2023-08-25T09:58:29.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/20' 2023-08-25T09:58:29.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/39' 2023-08-25T09:58:29.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/34' 2023-08-25T09:58:29.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/38' 2023-08-25T09:58:29.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/28' 2023-08-25T09:58:29.760 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/24' 2023-08-25T09:58:29.760 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/29' 2023-08-25T09:58:29.760 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/32' 2023-08-25T09:58:29.760 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/22' 2023-08-25T09:58:29.761 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/25' 2023-08-25T09:58:29.761 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/30' 2023-08-25T09:58:29.761 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/33' 2023-08-25T09:58:29.762 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/23' 2023-08-25T09:58:29.762 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/35' 2023-08-25T09:58:29.763 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/26' 2023-08-25T09:58:29.763 INFO:tasks.workunit.client.0.smithi071.stdout:removed './229/37' 2023-08-25T09:58:29.763 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './229' 2023-08-25T09:58:29.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/21' 2023-08-25T09:58:29.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/27' 2023-08-25T09:58:29.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/31' 2023-08-25T09:58:29.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/36' 2023-08-25T09:58:29.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/20' 2023-08-25T09:58:29.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/39' 2023-08-25T09:58:29.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/34' 2023-08-25T09:58:29.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/38' 2023-08-25T09:58:29.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/28' 2023-08-25T09:58:29.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/24' 2023-08-25T09:58:29.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/29' 2023-08-25T09:58:29.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/32' 2023-08-25T09:58:29.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/22' 2023-08-25T09:58:29.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/25' 2023-08-25T09:58:29.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/30' 2023-08-25T09:58:29.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/33' 2023-08-25T09:58:29.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/23' 2023-08-25T09:58:29.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/35' 2023-08-25T09:58:29.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/26' 2023-08-25T09:58:29.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './230/37' 2023-08-25T09:58:29.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './230' 2023-08-25T09:58:29.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/21' 2023-08-25T09:58:29.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/27' 2023-08-25T09:58:29.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/31' 2023-08-25T09:58:29.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/36' 2023-08-25T09:58:29.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/20' 2023-08-25T09:58:29.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/39' 2023-08-25T09:58:29.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/34' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/38' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/28' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/24' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/29' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/32' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/22' 2023-08-25T09:58:29.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/25' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/30' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/33' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/23' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/35' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/26' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './231/37' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './231' 2023-08-25T09:58:29.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/21' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/27' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/31' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/36' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/20' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/39' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/34' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/38' 2023-08-25T09:58:29.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/28' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/24' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/29' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/32' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/22' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/25' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/30' 2023-08-25T09:58:29.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/33' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/23' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/35' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/26' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './232/37' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './232' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/21' 2023-08-25T09:58:29.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/27' 2023-08-25T09:58:29.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/31' 2023-08-25T09:58:29.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/36' 2023-08-25T09:58:29.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/20' 2023-08-25T09:58:29.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/39' 2023-08-25T09:58:29.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/34' 2023-08-25T09:58:29.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/38' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/28' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/24' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/29' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/32' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/22' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/25' 2023-08-25T09:58:29.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/30' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/33' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/23' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/35' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/26' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './233/37' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './233' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/21' 2023-08-25T09:58:29.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/27' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/31' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/36' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/20' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/39' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/34' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/38' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/28' 2023-08-25T09:58:29.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/24' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/29' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/32' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/22' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/25' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/30' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/33' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/23' 2023-08-25T09:58:29.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/35' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/26' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './234/37' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './234' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/21' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/27' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/31' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/36' 2023-08-25T09:58:29.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/20' 2023-08-25T09:58:29.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/39' 2023-08-25T09:58:29.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/34' 2023-08-25T09:58:29.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/38' 2023-08-25T09:58:29.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/28' 2023-08-25T09:58:29.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/24' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/29' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/32' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/22' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/25' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/30' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/33' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/23' 2023-08-25T09:58:29.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/35' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/26' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './235/37' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './235' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/21' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/27' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/31' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/36' 2023-08-25T09:58:29.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/20' 2023-08-25T09:58:29.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/39' 2023-08-25T09:58:29.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/34' 2023-08-25T09:58:29.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/38' 2023-08-25T09:58:29.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/28' 2023-08-25T09:58:29.788 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/24' 2023-08-25T09:58:29.789 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/29' 2023-08-25T09:58:29.789 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/32' 2023-08-25T09:58:31.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/22' 2023-08-25T09:58:31.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/25' 2023-08-25T09:58:31.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/30' 2023-08-25T09:58:31.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/33' 2023-08-25T09:58:31.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/23' 2023-08-25T09:58:31.755 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/35' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/26' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './236/37' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './236' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/21' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/27' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/31' 2023-08-25T09:58:31.756 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/36' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/20' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/39' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/34' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/38' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/28' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/24' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/29' 2023-08-25T09:58:31.757 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/32' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/22' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/25' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/30' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/33' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/23' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/35' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/26' 2023-08-25T09:58:31.758 INFO:tasks.workunit.client.0.smithi071.stdout:removed './237/37' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './237' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/21' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/27' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/31' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/36' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/20' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/39' 2023-08-25T09:58:31.759 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/34' 2023-08-25T09:58:31.760 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/38' 2023-08-25T09:58:31.760 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/28' 2023-08-25T09:58:31.761 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/24' 2023-08-25T09:58:31.761 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/29' 2023-08-25T09:58:31.762 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/32' 2023-08-25T09:58:31.762 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/22' 2023-08-25T09:58:31.763 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/25' 2023-08-25T09:58:31.763 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/30' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/33' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/23' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/35' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/26' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './238/37' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './238' 2023-08-25T09:58:31.764 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/21' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/27' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/31' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/36' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/20' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/39' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/34' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/38' 2023-08-25T09:58:31.765 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/28' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/24' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/29' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/32' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/22' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/25' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/30' 2023-08-25T09:58:31.766 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/33' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/23' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/35' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/26' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './239/37' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './239' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/21' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/27' 2023-08-25T09:58:31.767 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/31' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/36' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/20' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/39' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/34' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/38' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/28' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/24' 2023-08-25T09:58:31.768 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/29' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/32' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/22' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/25' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/30' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/33' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/23' 2023-08-25T09:58:31.769 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/35' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/26' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './240/37' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './240' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/21' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/27' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/31' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/36' 2023-08-25T09:58:31.770 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/20' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/39' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/34' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/38' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/28' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/24' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/29' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/32' 2023-08-25T09:58:31.771 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/22' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/25' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/30' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/33' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/23' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/35' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/26' 2023-08-25T09:58:31.772 INFO:tasks.workunit.client.0.smithi071.stdout:removed './241/37' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './241' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/21' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/27' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/31' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/36' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/20' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/39' 2023-08-25T09:58:31.773 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/34' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/38' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/28' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/24' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/29' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/32' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/22' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/25' 2023-08-25T09:58:31.774 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/30' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/33' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/23' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/35' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/26' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './242/37' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './242' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/21' 2023-08-25T09:58:31.775 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/27' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/31' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/36' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/20' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/39' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/34' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/38' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/28' 2023-08-25T09:58:31.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/24' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/29' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/32' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/22' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/25' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/30' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/33' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/23' 2023-08-25T09:58:31.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/35' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/26' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './243/37' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './243' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/21' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/27' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/31' 2023-08-25T09:58:31.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/36' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/20' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/39' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/34' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/38' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/28' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/24' 2023-08-25T09:58:31.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/29' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/32' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/22' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/25' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/30' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/33' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/23' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/35' 2023-08-25T09:58:31.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/26' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './244/37' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './244' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/21' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/27' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/31' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/36' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/20' 2023-08-25T09:58:31.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/39' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/34' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/38' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/28' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/24' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/29' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/32' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/22' 2023-08-25T09:58:31.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/25' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/30' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/33' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/23' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/35' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/26' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './245/37' 2023-08-25T09:58:31.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './245' 2023-08-25T09:58:31.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/21' 2023-08-25T09:58:31.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/27' 2023-08-25T09:58:31.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/31' 2023-08-25T09:58:31.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/36' 2023-08-25T09:58:31.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/20' 2023-08-25T09:58:31.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/39' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/34' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/38' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/28' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/24' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/29' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/32' 2023-08-25T09:58:31.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/22' 2023-08-25T09:58:33.697 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/25' 2023-08-25T09:58:33.697 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/30' 2023-08-25T09:58:33.698 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/33' 2023-08-25T09:58:33.698 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/23' 2023-08-25T09:58:33.698 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/35' 2023-08-25T09:58:33.698 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/26' 2023-08-25T09:58:33.698 INFO:tasks.workunit.client.0.smithi071.stdout:removed './246/37' 2023-08-25T09:58:33.698 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './246' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/21' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/27' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/31' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/36' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/20' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/39' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/34' 2023-08-25T09:58:33.699 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/38' 2023-08-25T09:58:33.700 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/28' 2023-08-25T09:58:33.700 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/24' 2023-08-25T09:58:33.700 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/29' 2023-08-25T09:58:33.700 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/32' 2023-08-25T09:58:33.700 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/22' 2023-08-25T09:58:33.700 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/25' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/30' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/33' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/23' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/35' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/26' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed './247/37' 2023-08-25T09:58:33.701 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './247' 2023-08-25T09:58:33.703 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/21' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/27' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/31' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/36' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/20' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/39' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/34' 2023-08-25T09:58:33.704 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/38' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/28' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/24' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/29' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/32' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/22' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/25' 2023-08-25T09:58:33.705 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/30' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/33' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/23' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/35' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/26' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './248/37' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './248' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/21' 2023-08-25T09:58:33.706 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/27' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/31' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/36' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/20' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/39' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/34' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/38' 2023-08-25T09:58:33.707 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/28' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/24' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/29' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/32' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/22' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/25' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/30' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/33' 2023-08-25T09:58:33.708 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/23' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/35' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/26' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './249/37' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './249' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/21' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/27' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/31' 2023-08-25T09:58:33.709 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/36' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/20' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/39' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/34' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/38' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/28' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/24' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/29' 2023-08-25T09:58:33.710 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/32' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/22' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/25' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/30' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/33' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/23' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/35' 2023-08-25T09:58:33.711 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/26' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed './250/37' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './250' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/21' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/27' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/31' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/36' 2023-08-25T09:58:33.712 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/20' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/39' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/34' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/38' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/28' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/24' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/29' 2023-08-25T09:58:33.713 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/32' 2023-08-25T09:58:33.714 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/22' 2023-08-25T09:58:33.714 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/25' 2023-08-25T09:58:33.714 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/30' 2023-08-25T09:58:33.714 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/33' 2023-08-25T09:58:33.714 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/23' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/35' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/26' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './251/37' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './251' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/21' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/27' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/31' 2023-08-25T09:58:33.715 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/36' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/20' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/39' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/34' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/38' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/28' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/24' 2023-08-25T09:58:33.716 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/29' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/32' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/22' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/25' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/30' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/33' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/23' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/35' 2023-08-25T09:58:33.717 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/26' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './252/37' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './252' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/21' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/27' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/31' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/36' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/20' 2023-08-25T09:58:33.718 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/39' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/34' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/38' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/28' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/24' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/29' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/32' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/22' 2023-08-25T09:58:33.719 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/25' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/30' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/33' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/23' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/35' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/26' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed './253/37' 2023-08-25T09:58:33.720 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './253' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/21' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/27' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/31' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/36' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/20' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/39' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/34' 2023-08-25T09:58:33.721 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/38' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/28' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/24' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/29' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/32' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/22' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/25' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/30' 2023-08-25T09:58:33.722 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/33' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/23' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/35' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/26' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './254/37' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './254' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/21' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/27' 2023-08-25T09:58:33.723 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/31' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/36' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/20' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/39' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/34' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/38' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/28' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/24' 2023-08-25T09:58:33.724 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/29' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/32' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/22' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/25' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/30' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/33' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/23' 2023-08-25T09:58:33.725 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/35' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/26' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './255/37' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './255' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/21' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/27' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/31' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/36' 2023-08-25T09:58:33.726 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/20' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/39' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/34' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/38' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/28' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/24' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/29' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/32' 2023-08-25T09:58:33.727 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/22' 2023-08-25T09:58:33.728 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/25' 2023-08-25T09:58:33.728 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/30' 2023-08-25T09:58:35.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/33' 2023-08-25T09:58:35.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/23' 2023-08-25T09:58:35.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/35' 2023-08-25T09:58:35.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/26' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './256/37' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './256' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/21' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/27' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/31' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/36' 2023-08-25T09:58:35.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/20' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/39' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/34' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/38' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/28' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/24' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/29' 2023-08-25T09:58:35.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/32' 2023-08-25T09:58:35.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/22' 2023-08-25T09:58:35.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/25' 2023-08-25T09:58:35.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/30' 2023-08-25T09:58:35.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/33' 2023-08-25T09:58:35.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/23' 2023-08-25T09:58:35.231 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/35' 2023-08-25T09:58:35.231 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/26' 2023-08-25T09:58:35.231 INFO:tasks.workunit.client.0.smithi071.stdout:removed './257/37' 2023-08-25T09:58:35.231 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './257' 2023-08-25T09:58:35.231 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/21' 2023-08-25T09:58:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/27' 2023-08-25T09:58:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/31' 2023-08-25T09:58:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/36' 2023-08-25T09:58:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/20' 2023-08-25T09:58:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/39' 2023-08-25T09:58:35.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/34' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/38' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/28' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/24' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/29' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/32' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/22' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/25' 2023-08-25T09:58:35.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/30' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/33' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/23' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/35' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/26' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './258/37' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './258' 2023-08-25T09:58:35.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/21' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/27' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/31' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/36' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/20' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/39' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/34' 2023-08-25T09:58:35.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/38' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/28' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/24' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/29' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/32' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/22' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/25' 2023-08-25T09:58:35.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/30' 2023-08-25T09:58:35.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/33' 2023-08-25T09:58:35.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/23' 2023-08-25T09:58:35.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/35' 2023-08-25T09:58:35.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/26' 2023-08-25T09:58:35.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './259/37' 2023-08-25T09:58:35.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './259' 2023-08-25T09:58:35.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/21' 2023-08-25T09:58:35.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/27' 2023-08-25T09:58:35.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/31' 2023-08-25T09:58:35.239 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/36' 2023-08-25T09:58:35.239 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/20' 2023-08-25T09:58:35.239 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/39' 2023-08-25T09:58:35.239 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/34' 2023-08-25T09:58:35.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/38' 2023-08-25T09:58:35.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/28' 2023-08-25T09:58:35.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/24' 2023-08-25T09:58:35.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/29' 2023-08-25T09:58:35.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/32' 2023-08-25T09:58:35.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/22' 2023-08-25T09:58:35.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/25' 2023-08-25T09:58:35.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/30' 2023-08-25T09:58:35.243 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/33' 2023-08-25T09:58:35.243 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/23' 2023-08-25T09:58:35.243 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/35' 2023-08-25T09:58:35.243 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/26' 2023-08-25T09:58:35.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './260/37' 2023-08-25T09:58:35.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './260' 2023-08-25T09:58:35.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/21' 2023-08-25T09:58:35.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/27' 2023-08-25T09:58:35.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/31' 2023-08-25T09:58:35.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/36' 2023-08-25T09:58:35.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/20' 2023-08-25T09:58:35.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/39' 2023-08-25T09:58:35.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/34' 2023-08-25T09:58:35.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/38' 2023-08-25T09:58:35.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/28' 2023-08-25T09:58:35.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/24' 2023-08-25T09:58:35.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/29' 2023-08-25T09:58:35.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/32' 2023-08-25T09:58:35.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/22' 2023-08-25T09:58:35.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/25' 2023-08-25T09:58:35.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/30' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/33' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/23' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/35' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/26' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './261/37' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './261' 2023-08-25T09:58:35.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/21' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/27' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/31' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/36' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/20' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/39' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/34' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/38' 2023-08-25T09:58:35.249 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/28' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/24' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/29' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/32' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/22' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/25' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/30' 2023-08-25T09:58:35.250 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/33' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/23' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/35' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/26' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed './262/37' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './262' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/21' 2023-08-25T09:58:35.251 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/27' 2023-08-25T09:58:35.252 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/31' 2023-08-25T09:58:35.252 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/36' 2023-08-25T09:58:35.252 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/20' 2023-08-25T09:58:35.253 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/39' 2023-08-25T09:58:35.253 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/34' 2023-08-25T09:58:35.253 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/38' 2023-08-25T09:58:35.254 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/28' 2023-08-25T09:58:35.254 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/24' 2023-08-25T09:58:35.254 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/29' 2023-08-25T09:58:35.255 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/32' 2023-08-25T09:58:35.255 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/22' 2023-08-25T09:58:35.255 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/25' 2023-08-25T09:58:35.255 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/30' 2023-08-25T09:58:35.256 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/33' 2023-08-25T09:58:35.256 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/23' 2023-08-25T09:58:35.256 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/35' 2023-08-25T09:58:35.256 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/26' 2023-08-25T09:58:35.256 INFO:tasks.workunit.client.0.smithi071.stdout:removed './263/37' 2023-08-25T09:58:35.257 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './263' 2023-08-25T09:58:35.257 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/21' 2023-08-25T09:58:35.257 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/27' 2023-08-25T09:58:35.257 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/31' 2023-08-25T09:58:35.258 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/36' 2023-08-25T09:58:35.258 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/20' 2023-08-25T09:58:35.258 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/39' 2023-08-25T09:58:35.259 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/34' 2023-08-25T09:58:35.259 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/38' 2023-08-25T09:58:35.259 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/28' 2023-08-25T09:58:35.259 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/24' 2023-08-25T09:58:35.259 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/29' 2023-08-25T09:58:35.260 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/32' 2023-08-25T09:58:35.260 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/22' 2023-08-25T09:58:35.260 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/25' 2023-08-25T09:58:35.260 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/30' 2023-08-25T09:58:35.261 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/33' 2023-08-25T09:58:35.261 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/23' 2023-08-25T09:58:35.261 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/35' 2023-08-25T09:58:35.261 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/26' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed './264/37' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './264' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/21' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/27' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/31' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/36' 2023-08-25T09:58:35.262 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/20' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/39' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/34' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/38' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/28' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/24' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/29' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/32' 2023-08-25T09:58:35.263 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/22' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/25' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/30' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/33' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/23' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/35' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/26' 2023-08-25T09:58:35.264 INFO:tasks.workunit.client.0.smithi071.stdout:removed './265/37' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './265' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/21' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/27' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/31' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/36' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/20' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/39' 2023-08-25T09:58:35.265 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/34' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/38' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/28' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/24' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/29' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/32' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/22' 2023-08-25T09:58:35.266 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/25' 2023-08-25T09:58:35.267 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/30' 2023-08-25T09:58:35.267 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/33' 2023-08-25T09:58:35.267 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/23' 2023-08-25T09:58:36.396 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/35' 2023-08-25T09:58:36.396 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/26' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './266/37' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './266' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/21' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/27' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/31' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/36' 2023-08-25T09:58:36.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/20' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/39' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/34' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/38' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/28' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/24' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/29' 2023-08-25T09:58:36.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/32' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/22' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/25' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/30' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/33' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/23' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/35' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/26' 2023-08-25T09:58:36.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './267/37' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './267' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/21' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/27' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/31' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/36' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/20' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/39' 2023-08-25T09:58:36.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/34' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/38' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/28' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/24' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/29' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/32' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/22' 2023-08-25T09:58:36.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/25' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/30' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/33' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/23' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/35' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/26' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './268/37' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './268' 2023-08-25T09:58:36.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/21' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/27' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/31' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/36' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/20' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/39' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/34' 2023-08-25T09:58:36.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/38' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/28' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/24' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/29' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/32' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/22' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/25' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/30' 2023-08-25T09:58:36.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/33' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/23' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/35' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/26' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './269/37' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './269' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/21' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/27' 2023-08-25T09:58:36.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/31' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/36' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/20' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/39' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/34' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/38' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/28' 2023-08-25T09:58:36.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/24' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/29' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/32' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/22' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/25' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/30' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/33' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/23' 2023-08-25T09:58:36.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/35' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/26' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './270/37' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './270' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/21' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/27' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/31' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/36' 2023-08-25T09:58:36.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/20' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/39' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/34' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/38' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/28' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/24' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/29' 2023-08-25T09:58:36.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/32' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/22' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/25' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/30' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/33' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/23' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/35' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/26' 2023-08-25T09:58:36.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './271/37' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './271' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/21' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/27' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/31' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/36' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/20' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/39' 2023-08-25T09:58:36.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/34' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/38' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/28' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/24' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/29' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/32' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/22' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/25' 2023-08-25T09:58:36.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/30' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/33' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/23' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/35' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/26' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './272/37' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './272' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/21' 2023-08-25T09:58:36.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/27' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/31' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/36' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/20' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/39' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/34' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/38' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/28' 2023-08-25T09:58:36.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/24' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/29' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/32' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/22' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/25' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/30' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/33' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/23' 2023-08-25T09:58:36.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/35' 2023-08-25T09:58:36.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/26' 2023-08-25T09:58:36.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed './273/37' 2023-08-25T09:58:36.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './273' 2023-08-25T09:58:36.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/21' 2023-08-25T09:58:36.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/27' 2023-08-25T09:58:36.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/31' 2023-08-25T09:58:36.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/36' 2023-08-25T09:58:36.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/20' 2023-08-25T09:58:36.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/39' 2023-08-25T09:58:36.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/34' 2023-08-25T09:58:36.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/38' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/28' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/24' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/29' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/32' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/22' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/25' 2023-08-25T09:58:36.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/30' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/33' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/23' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/35' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/26' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './274/37' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './274' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/21' 2023-08-25T09:58:36.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/27' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/31' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/36' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/20' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/39' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/34' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/38' 2023-08-25T09:58:36.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/28' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/24' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/29' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/32' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/22' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/25' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/30' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/33' 2023-08-25T09:58:36.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/23' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/35' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/26' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './275/37' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './275' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/21' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/27' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/31' 2023-08-25T09:58:36.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/36' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/20' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/39' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/34' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/38' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/28' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/24' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/29' 2023-08-25T09:58:36.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/32' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/22' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/25' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/30' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/33' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/23' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/35' 2023-08-25T09:58:36.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/26' 2023-08-25T09:58:39.381 INFO:tasks.workunit.client.0.smithi071.stdout:removed './276/37' 2023-08-25T09:58:39.382 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './276' 2023-08-25T09:58:39.382 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/21' 2023-08-25T09:58:39.382 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/27' 2023-08-25T09:58:39.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/31' 2023-08-25T09:58:39.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/36' 2023-08-25T09:58:39.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/20' 2023-08-25T09:58:39.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/39' 2023-08-25T09:58:39.383 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/34' 2023-08-25T09:58:39.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/38' 2023-08-25T09:58:39.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/28' 2023-08-25T09:58:39.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/24' 2023-08-25T09:58:39.384 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/29' 2023-08-25T09:58:39.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/32' 2023-08-25T09:58:39.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/22' 2023-08-25T09:58:39.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/25' 2023-08-25T09:58:39.385 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/30' 2023-08-25T09:58:39.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/33' 2023-08-25T09:58:39.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/23' 2023-08-25T09:58:39.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/35' 2023-08-25T09:58:39.386 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/26' 2023-08-25T09:58:39.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed './277/37' 2023-08-25T09:58:39.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './277' 2023-08-25T09:58:39.387 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/21' 2023-08-25T09:58:39.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/27' 2023-08-25T09:58:39.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/31' 2023-08-25T09:58:39.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/36' 2023-08-25T09:58:39.388 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/20' 2023-08-25T09:58:39.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/39' 2023-08-25T09:58:39.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/34' 2023-08-25T09:58:39.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/38' 2023-08-25T09:58:39.389 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/28' 2023-08-25T09:58:39.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/24' 2023-08-25T09:58:39.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/29' 2023-08-25T09:58:39.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/32' 2023-08-25T09:58:39.390 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/22' 2023-08-25T09:58:39.391 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/25' 2023-08-25T09:58:39.391 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/30' 2023-08-25T09:58:39.391 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/33' 2023-08-25T09:58:39.392 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/23' 2023-08-25T09:58:39.392 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/35' 2023-08-25T09:58:39.392 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/26' 2023-08-25T09:58:39.392 INFO:tasks.workunit.client.0.smithi071.stdout:removed './278/37' 2023-08-25T09:58:39.393 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './278' 2023-08-25T09:58:39.393 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/21' 2023-08-25T09:58:39.393 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/27' 2023-08-25T09:58:39.393 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/31' 2023-08-25T09:58:39.394 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/36' 2023-08-25T09:58:39.394 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/20' 2023-08-25T09:58:39.394 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/39' 2023-08-25T09:58:39.394 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/34' 2023-08-25T09:58:39.394 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/38' 2023-08-25T09:58:39.395 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/28' 2023-08-25T09:58:39.395 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/24' 2023-08-25T09:58:39.395 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/29' 2023-08-25T09:58:39.395 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/32' 2023-08-25T09:58:39.396 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/22' 2023-08-25T09:58:39.396 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/25' 2023-08-25T09:58:39.396 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/30' 2023-08-25T09:58:39.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/33' 2023-08-25T09:58:39.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/23' 2023-08-25T09:58:39.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/35' 2023-08-25T09:58:39.397 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/26' 2023-08-25T09:58:39.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './279/37' 2023-08-25T09:58:39.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './279' 2023-08-25T09:58:39.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/21' 2023-08-25T09:58:39.398 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/27' 2023-08-25T09:58:39.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/31' 2023-08-25T09:58:39.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/36' 2023-08-25T09:58:39.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/20' 2023-08-25T09:58:39.399 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/39' 2023-08-25T09:58:39.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/34' 2023-08-25T09:58:39.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/38' 2023-08-25T09:58:39.400 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/28' 2023-08-25T09:58:39.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/24' 2023-08-25T09:58:39.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/29' 2023-08-25T09:58:39.401 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/32' 2023-08-25T09:58:39.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/22' 2023-08-25T09:58:39.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/25' 2023-08-25T09:58:39.402 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/30' 2023-08-25T09:58:39.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/33' 2023-08-25T09:58:39.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/23' 2023-08-25T09:58:39.403 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/35' 2023-08-25T09:58:39.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/26' 2023-08-25T09:58:39.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './280/37' 2023-08-25T09:58:39.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './280' 2023-08-25T09:58:39.404 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/21' 2023-08-25T09:58:39.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/27' 2023-08-25T09:58:39.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/31' 2023-08-25T09:58:39.405 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/36' 2023-08-25T09:58:39.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/20' 2023-08-25T09:58:39.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/39' 2023-08-25T09:58:39.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/34' 2023-08-25T09:58:39.406 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/38' 2023-08-25T09:58:39.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/28' 2023-08-25T09:58:39.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/24' 2023-08-25T09:58:39.407 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/29' 2023-08-25T09:58:39.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/32' 2023-08-25T09:58:39.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/22' 2023-08-25T09:58:39.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/25' 2023-08-25T09:58:39.408 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/30' 2023-08-25T09:58:39.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/33' 2023-08-25T09:58:39.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/23' 2023-08-25T09:58:39.409 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/35' 2023-08-25T09:58:39.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/26' 2023-08-25T09:58:39.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './281/37' 2023-08-25T09:58:39.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './281' 2023-08-25T09:58:39.410 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/21' 2023-08-25T09:58:39.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/27' 2023-08-25T09:58:39.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/31' 2023-08-25T09:58:39.411 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/36' 2023-08-25T09:58:39.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/20' 2023-08-25T09:58:39.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/39' 2023-08-25T09:58:39.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/34' 2023-08-25T09:58:39.412 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/38' 2023-08-25T09:58:39.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/28' 2023-08-25T09:58:39.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/24' 2023-08-25T09:58:39.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/29' 2023-08-25T09:58:39.413 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/32' 2023-08-25T09:58:39.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/22' 2023-08-25T09:58:39.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/25' 2023-08-25T09:58:39.414 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/30' 2023-08-25T09:58:39.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/33' 2023-08-25T09:58:39.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/23' 2023-08-25T09:58:39.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/35' 2023-08-25T09:58:39.415 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/26' 2023-08-25T09:58:39.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed './282/37' 2023-08-25T09:58:39.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './282' 2023-08-25T09:58:39.416 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/21' 2023-08-25T09:58:39.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/27' 2023-08-25T09:58:39.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/31' 2023-08-25T09:58:39.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/36' 2023-08-25T09:58:39.417 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/20' 2023-08-25T09:58:39.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/39' 2023-08-25T09:58:39.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/34' 2023-08-25T09:58:39.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/38' 2023-08-25T09:58:39.418 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/28' 2023-08-25T09:58:39.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/24' 2023-08-25T09:58:39.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/29' 2023-08-25T09:58:39.419 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/32' 2023-08-25T09:58:39.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/22' 2023-08-25T09:58:39.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/25' 2023-08-25T09:58:39.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/30' 2023-08-25T09:58:39.420 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/33' 2023-08-25T09:58:39.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/23' 2023-08-25T09:58:39.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/35' 2023-08-25T09:58:39.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/26' 2023-08-25T09:58:39.421 INFO:tasks.workunit.client.0.smithi071.stdout:removed './283/37' 2023-08-25T09:58:39.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './283' 2023-08-25T09:58:39.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/21' 2023-08-25T09:58:39.422 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/27' 2023-08-25T09:58:39.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/31' 2023-08-25T09:58:39.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/36' 2023-08-25T09:58:39.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/20' 2023-08-25T09:58:39.423 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/39' 2023-08-25T09:58:39.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/34' 2023-08-25T09:58:39.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/38' 2023-08-25T09:58:39.424 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/28' 2023-08-25T09:58:39.425 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/24' 2023-08-25T09:58:39.425 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/29' 2023-08-25T09:58:39.425 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/32' 2023-08-25T09:58:39.425 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/22' 2023-08-25T09:58:39.426 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/25' 2023-08-25T09:58:39.426 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/30' 2023-08-25T09:58:39.426 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/33' 2023-08-25T09:58:39.426 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/23' 2023-08-25T09:58:39.427 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/35' 2023-08-25T09:58:39.427 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/26' 2023-08-25T09:58:39.427 INFO:tasks.workunit.client.0.smithi071.stdout:removed './284/37' 2023-08-25T09:58:39.428 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './284' 2023-08-25T09:58:39.428 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/21' 2023-08-25T09:58:39.428 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/27' 2023-08-25T09:58:39.428 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/31' 2023-08-25T09:58:39.429 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/36' 2023-08-25T09:58:39.429 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/20' 2023-08-25T09:58:39.429 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/39' 2023-08-25T09:58:39.430 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/34' 2023-08-25T09:58:39.430 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/38' 2023-08-25T09:58:39.430 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/28' 2023-08-25T09:58:39.430 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/24' 2023-08-25T09:58:39.431 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/29' 2023-08-25T09:58:39.431 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/32' 2023-08-25T09:58:39.431 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/22' 2023-08-25T09:58:39.432 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/25' 2023-08-25T09:58:39.432 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/30' 2023-08-25T09:58:39.432 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/33' 2023-08-25T09:58:39.432 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/23' 2023-08-25T09:58:39.433 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/35' 2023-08-25T09:58:39.433 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/26' 2023-08-25T09:58:39.433 INFO:tasks.workunit.client.0.smithi071.stdout:removed './285/37' 2023-08-25T09:58:39.434 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './285' 2023-08-25T09:58:39.434 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/21' 2023-08-25T09:58:39.434 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/27' 2023-08-25T09:58:39.434 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/31' 2023-08-25T09:58:39.435 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/36' 2023-08-25T09:58:39.435 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/20' 2023-08-25T09:58:39.435 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/39' 2023-08-25T09:58:39.435 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/34' 2023-08-25T09:58:39.436 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/38' 2023-08-25T09:58:39.436 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/28' 2023-08-25T09:58:39.436 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/24' 2023-08-25T09:58:39.437 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/29' 2023-08-25T09:58:39.437 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/32' 2023-08-25T09:58:39.437 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/22' 2023-08-25T09:58:39.437 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/25' 2023-08-25T09:58:39.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/30' 2023-08-25T09:58:39.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/33' 2023-08-25T09:58:39.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/23' 2023-08-25T09:58:39.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/35' 2023-08-25T09:58:39.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/26' 2023-08-25T09:58:39.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './286/37' 2023-08-25T09:58:39.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './286' 2023-08-25T09:58:43.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/21' 2023-08-25T09:58:43.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/27' 2023-08-25T09:58:43.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/31' 2023-08-25T09:58:43.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/36' 2023-08-25T09:58:43.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/20' 2023-08-25T09:58:43.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/39' 2023-08-25T09:58:43.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/34' 2023-08-25T09:58:43.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/38' 2023-08-25T09:58:43.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/28' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/24' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/29' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/32' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/22' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/25' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/30' 2023-08-25T09:58:43.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/33' 2023-08-25T09:58:43.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/23' 2023-08-25T09:58:43.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/35' 2023-08-25T09:58:43.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/26' 2023-08-25T09:58:43.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './287/37' 2023-08-25T09:58:43.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './287' 2023-08-25T09:58:43.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/21' 2023-08-25T09:58:43.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/27' 2023-08-25T09:58:43.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/31' 2023-08-25T09:58:43.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/36' 2023-08-25T09:58:43.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/20' 2023-08-25T09:58:43.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/39' 2023-08-25T09:58:43.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/34' 2023-08-25T09:58:43.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/38' 2023-08-25T09:58:43.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/28' 2023-08-25T09:58:43.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/24' 2023-08-25T09:58:43.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/29' 2023-08-25T09:58:43.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/32' 2023-08-25T09:58:43.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/22' 2023-08-25T09:58:43.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/25' 2023-08-25T09:58:43.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/30' 2023-08-25T09:58:43.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/33' 2023-08-25T09:58:43.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/23' 2023-08-25T09:58:43.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/35' 2023-08-25T09:58:43.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/26' 2023-08-25T09:58:43.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './288/37' 2023-08-25T09:58:43.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './288' 2023-08-25T09:58:43.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/21' 2023-08-25T09:58:43.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/27' 2023-08-25T09:58:43.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/31' 2023-08-25T09:58:43.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/36' 2023-08-25T09:58:43.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/20' 2023-08-25T09:58:43.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/39' 2023-08-25T09:58:43.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/34' 2023-08-25T09:58:43.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/38' 2023-08-25T09:58:43.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/28' 2023-08-25T09:58:43.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/24' 2023-08-25T09:58:43.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/29' 2023-08-25T09:58:43.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/32' 2023-08-25T09:58:43.620 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/22' 2023-08-25T09:58:43.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/25' 2023-08-25T09:58:43.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/30' 2023-08-25T09:58:43.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/33' 2023-08-25T09:58:43.621 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/23' 2023-08-25T09:58:43.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/35' 2023-08-25T09:58:43.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/26' 2023-08-25T09:58:43.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed './289/37' 2023-08-25T09:58:43.622 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './289' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/21' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/27' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/31' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/36' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/20' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/39' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/34' 2023-08-25T09:58:43.623 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/38' 2023-08-25T09:58:43.624 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/28' 2023-08-25T09:58:43.624 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/24' 2023-08-25T09:58:43.624 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/29' 2023-08-25T09:58:43.624 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/32' 2023-08-25T09:58:43.624 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/22' 2023-08-25T09:58:43.624 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/25' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/30' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/33' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/23' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/35' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/26' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './290/37' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './290' 2023-08-25T09:58:43.625 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/21' 2023-08-25T09:58:43.626 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/27' 2023-08-25T09:58:43.626 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/31' 2023-08-25T09:58:43.626 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/36' 2023-08-25T09:58:43.626 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/20' 2023-08-25T09:58:43.626 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/39' 2023-08-25T09:58:43.626 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/34' 2023-08-25T09:58:43.627 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/38' 2023-08-25T09:58:43.627 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/28' 2023-08-25T09:58:43.627 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/24' 2023-08-25T09:58:43.627 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/29' 2023-08-25T09:58:43.627 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/32' 2023-08-25T09:58:43.627 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/22' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/25' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/30' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/33' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/23' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/35' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/26' 2023-08-25T09:58:43.628 INFO:tasks.workunit.client.0.smithi071.stdout:removed './291/37' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './291' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/21' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/27' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/31' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/36' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/20' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/39' 2023-08-25T09:58:43.629 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/34' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/38' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/28' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/24' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/29' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/32' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/22' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/25' 2023-08-25T09:58:43.630 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/30' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/33' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/23' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/35' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/26' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './292/37' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './292' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/21' 2023-08-25T09:58:43.631 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/27' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/31' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/36' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/20' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/39' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/34' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/38' 2023-08-25T09:58:43.632 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/28' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/24' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/29' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/32' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/22' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/25' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/30' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/33' 2023-08-25T09:58:43.633 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/23' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/35' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/26' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './293/37' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './293' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/21' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/27' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/31' 2023-08-25T09:58:43.634 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/36' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/20' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/39' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/34' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/38' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/28' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/24' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/29' 2023-08-25T09:58:43.635 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/32' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/22' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/25' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/30' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/33' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/23' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/35' 2023-08-25T09:58:43.636 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/26' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './294/37' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './294' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/21' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/27' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/31' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/36' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/20' 2023-08-25T09:58:43.637 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/39' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/34' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/38' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/28' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/24' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/29' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/32' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/22' 2023-08-25T09:58:43.638 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/25' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/30' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/33' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/23' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/35' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/26' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './295/37' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './295' 2023-08-25T09:58:43.639 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/21' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/27' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/31' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/36' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/20' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/39' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/34' 2023-08-25T09:58:43.640 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/38' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/28' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/24' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/29' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/32' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/22' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/25' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/30' 2023-08-25T09:58:43.641 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/33' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/23' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/35' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/26' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed './296/37' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './296' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/21' 2023-08-25T09:58:43.642 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/27' 2023-08-25T09:58:45.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/31' 2023-08-25T09:58:45.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/36' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/20' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/39' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/34' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/38' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/28' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/24' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/29' 2023-08-25T09:58:45.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/32' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/22' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/25' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/30' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/33' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/23' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/35' 2023-08-25T09:58:45.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/26' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './297/37' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './297' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/21' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/27' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/31' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/36' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/20' 2023-08-25T09:58:45.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/39' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/34' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/38' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/28' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/24' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/29' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/32' 2023-08-25T09:58:45.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/22' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/25' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/30' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/33' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/23' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/35' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/26' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './298/37' 2023-08-25T09:58:45.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './298' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/21' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/27' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/31' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/36' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/20' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/39' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/34' 2023-08-25T09:58:45.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/38' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/28' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/24' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/29' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/32' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/22' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/25' 2023-08-25T09:58:45.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/30' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/33' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/23' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/35' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/26' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './299/37' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './299' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/19' 2023-08-25T09:58:45.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/15' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/20' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/12' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/10' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/11' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/16' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/14' 2023-08-25T09:58:45.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/18' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/13' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/166/17' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/166' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/19' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/15' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/20' 2023-08-25T09:58:45.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/12' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/10' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/11' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/16' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/14' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/18' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/13' 2023-08-25T09:58:45.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/147/17' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/147' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/19' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/15' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/20' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/12' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/10' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/11' 2023-08-25T09:58:45.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/16' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/14' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/18' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/13' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/151/17' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/151' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/19' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/15' 2023-08-25T09:58:45.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/20' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/12' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/10' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/11' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/16' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/14' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/18' 2023-08-25T09:58:45.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/13' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/170/17' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/170' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/19' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/15' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/20' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/12' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/10' 2023-08-25T09:58:45.961 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/11' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/16' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/14' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/18' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/13' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/167/17' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/167' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/19' 2023-08-25T09:58:45.962 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/15' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/20' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/12' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/10' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/11' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/16' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/14' 2023-08-25T09:58:45.963 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/18' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/13' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/140/17' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/140' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/19' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/15' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/20' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/12' 2023-08-25T09:58:45.964 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/10' 2023-08-25T09:58:48.776 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/11' 2023-08-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/16' 2023-08-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/14' 2023-08-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/18' 2023-08-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/13' 2023-08-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/142/17' 2023-08-25T09:58:48.777 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/142' 2023-08-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/19' 2023-08-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/15' 2023-08-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/20' 2023-08-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/12' 2023-08-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/10' 2023-08-25T09:58:48.778 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/11' 2023-08-25T09:58:48.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/16' 2023-08-25T09:58:48.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/14' 2023-08-25T09:58:48.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/18' 2023-08-25T09:58:48.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/13' 2023-08-25T09:58:48.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/150/17' 2023-08-25T09:58:48.779 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/150' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/19' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/15' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/20' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/12' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/10' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/11' 2023-08-25T09:58:48.780 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/16' 2023-08-25T09:58:48.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/14' 2023-08-25T09:58:48.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/18' 2023-08-25T09:58:48.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/13' 2023-08-25T09:58:48.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/105/17' 2023-08-25T09:58:48.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/105' 2023-08-25T09:58:48.781 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/19' 2023-08-25T09:58:48.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/15' 2023-08-25T09:58:48.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/20' 2023-08-25T09:58:48.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/12' 2023-08-25T09:58:48.782 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/10' 2023-08-25T09:58:48.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/11' 2023-08-25T09:58:48.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/16' 2023-08-25T09:58:48.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/14' 2023-08-25T09:58:48.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/18' 2023-08-25T09:58:48.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/13' 2023-08-25T09:58:48.783 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/149/17' 2023-08-25T09:58:48.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/149' 2023-08-25T09:58:48.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/19' 2023-08-25T09:58:48.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/15' 2023-08-25T09:58:48.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/20' 2023-08-25T09:58:48.784 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/12' 2023-08-25T09:58:48.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/10' 2023-08-25T09:58:48.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/11' 2023-08-25T09:58:48.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/16' 2023-08-25T09:58:48.785 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/14' 2023-08-25T09:58:48.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/18' 2023-08-25T09:58:48.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/13' 2023-08-25T09:58:48.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/109/17' 2023-08-25T09:58:48.786 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/109' 2023-08-25T09:58:48.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/19' 2023-08-25T09:58:48.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/15' 2023-08-25T09:58:48.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/20' 2023-08-25T09:58:48.787 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/12' 2023-08-25T09:58:48.788 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/10' 2023-08-25T09:58:48.788 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/11' 2023-08-25T09:58:48.788 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/16' 2023-08-25T09:58:48.788 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/14' 2023-08-25T09:58:48.789 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/18' 2023-08-25T09:58:48.789 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/13' 2023-08-25T09:58:48.789 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/158/17' 2023-08-25T09:58:48.789 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/158' 2023-08-25T09:58:48.790 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/19' 2023-08-25T09:58:48.790 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/15' 2023-08-25T09:58:48.790 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/20' 2023-08-25T09:58:48.791 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/12' 2023-08-25T09:58:48.791 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/10' 2023-08-25T09:58:48.791 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/11' 2023-08-25T09:58:48.791 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/16' 2023-08-25T09:58:48.791 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/14' 2023-08-25T09:58:48.791 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/18' 2023-08-25T09:58:48.792 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/13' 2023-08-25T09:58:48.792 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/106/17' 2023-08-25T09:58:48.792 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/106' 2023-08-25T09:58:48.792 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/19' 2023-08-25T09:58:48.793 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/15' 2023-08-25T09:58:48.793 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/20' 2023-08-25T09:58:48.793 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/12' 2023-08-25T09:58:48.793 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/10' 2023-08-25T09:58:48.793 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/11' 2023-08-25T09:58:48.793 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/16' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/14' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/18' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/13' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/189/17' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/189' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/19' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/15' 2023-08-25T09:58:48.794 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/20' 2023-08-25T09:58:48.795 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/12' 2023-08-25T09:58:48.795 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/10' 2023-08-25T09:58:48.795 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/11' 2023-08-25T09:58:48.795 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/16' 2023-08-25T09:58:48.795 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/14' 2023-08-25T09:58:48.796 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/18' 2023-08-25T09:58:48.796 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/13' 2023-08-25T09:58:48.796 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/133/17' 2023-08-25T09:58:48.796 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/133' 2023-08-25T09:58:48.797 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/19' 2023-08-25T09:58:48.797 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/15' 2023-08-25T09:58:48.797 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/20' 2023-08-25T09:58:48.797 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/12' 2023-08-25T09:58:48.797 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/10' 2023-08-25T09:58:48.797 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/11' 2023-08-25T09:58:51.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/16' 2023-08-25T09:58:51.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/14' 2023-08-25T09:58:51.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/18' 2023-08-25T09:58:51.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/13' 2023-08-25T09:58:51.438 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/173/17' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/173' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/19' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/15' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/20' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/12' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/10' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/11' 2023-08-25T09:58:51.439 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/16' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/14' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/18' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/13' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/174/17' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/174' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/19' 2023-08-25T09:58:51.440 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/15' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/20' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/12' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/10' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/11' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/16' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/14' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/18' 2023-08-25T09:58:51.441 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/13' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/155/17' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/155' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/19' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/15' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/20' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/12' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/10' 2023-08-25T09:58:51.442 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/11' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/16' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/14' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/18' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/13' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/114/17' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/114' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/19' 2023-08-25T09:58:51.443 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/15' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/20' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/12' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/10' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/11' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/16' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/14' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/18' 2023-08-25T09:58:51.444 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/13' 2023-08-25T09:58:51.445 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/159/17' 2023-08-25T09:58:51.445 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/159' 2023-08-25T09:58:51.445 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/19' 2023-08-25T09:58:51.445 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/15' 2023-08-25T09:58:51.446 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/20' 2023-08-25T09:58:51.446 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/12' 2023-08-25T09:58:51.446 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/10' 2023-08-25T09:58:51.446 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/11' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/16' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/14' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/18' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/13' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/187/17' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/187' 2023-08-25T09:58:51.447 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/19' 2023-08-25T09:58:51.448 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/15' 2023-08-25T09:58:51.448 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/20' 2023-08-25T09:58:51.448 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/12' 2023-08-25T09:58:51.448 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/10' 2023-08-25T09:58:51.449 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/11' 2023-08-25T09:58:51.449 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/16' 2023-08-25T09:58:51.449 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/14' 2023-08-25T09:58:51.450 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/18' 2023-08-25T09:58:51.450 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/13' 2023-08-25T09:58:51.451 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/185/17' 2023-08-25T09:58:51.451 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/185' 2023-08-25T09:58:51.451 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/19' 2023-08-25T09:58:51.451 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/15' 2023-08-25T09:58:51.451 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/20' 2023-08-25T09:58:51.452 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/12' 2023-08-25T09:58:51.452 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/10' 2023-08-25T09:58:51.452 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/11' 2023-08-25T09:58:51.452 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/16' 2023-08-25T09:58:51.453 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/14' 2023-08-25T09:58:51.453 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/18' 2023-08-25T09:58:51.453 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/13' 2023-08-25T09:58:51.453 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/120/17' 2023-08-25T09:58:51.454 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/120' 2023-08-25T09:58:51.454 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/19' 2023-08-25T09:58:51.454 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/15' 2023-08-25T09:58:51.455 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/20' 2023-08-25T09:58:51.455 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/12' 2023-08-25T09:58:51.455 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/10' 2023-08-25T09:58:51.455 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/11' 2023-08-25T09:58:51.456 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/16' 2023-08-25T09:58:51.456 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/14' 2023-08-25T09:58:51.456 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/18' 2023-08-25T09:58:51.457 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/13' 2023-08-25T09:58:51.458 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/110/17' 2023-08-25T09:58:51.458 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/110' 2023-08-25T09:58:51.458 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/19' 2023-08-25T09:58:51.458 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/15' 2023-08-25T09:58:51.459 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/20' 2023-08-25T09:58:51.459 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/12' 2023-08-25T09:58:51.459 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/10' 2023-08-25T09:58:51.459 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/11' 2023-08-25T09:58:51.460 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/16' 2023-08-25T09:58:53.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/14' 2023-08-25T09:58:53.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/18' 2023-08-25T09:58:53.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/13' 2023-08-25T09:58:53.579 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/137/17' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/137' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/19' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/15' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/20' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/12' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/10' 2023-08-25T09:58:53.580 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/11' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/16' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/14' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/18' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/13' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/198/17' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/198' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/19' 2023-08-25T09:58:53.581 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/15' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/20' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/12' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/10' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/11' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/16' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/14' 2023-08-25T09:58:53.582 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/18' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/13' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/136/17' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/136' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/19' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/15' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/20' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/12' 2023-08-25T09:58:53.583 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/10' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/11' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/16' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/14' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/18' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/13' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/125/17' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/125' 2023-08-25T09:58:53.584 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/19' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/15' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/20' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/12' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/10' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/11' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/16' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/14' 2023-08-25T09:58:53.585 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/18' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/13' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/194/17' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/194' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/19' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/15' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/20' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/12' 2023-08-25T09:58:53.586 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/10' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/11' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/16' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/14' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/18' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/13' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/119/17' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/119' 2023-08-25T09:58:53.587 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/19' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/15' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/20' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/12' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/10' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/11' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/16' 2023-08-25T09:58:53.588 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/14' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/18' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/13' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/157/17' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/157' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/19' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/15' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/20' 2023-08-25T09:58:53.589 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/12' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/10' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/11' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/16' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/14' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/18' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/13' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/172/17' 2023-08-25T09:58:53.590 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/172' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/19' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/15' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/20' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/12' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/10' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/11' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/16' 2023-08-25T09:58:53.591 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/14' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/18' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/13' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/186/17' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/186' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/19' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/15' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/20' 2023-08-25T09:58:53.592 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/12' 2023-08-25T09:58:53.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/10' 2023-08-25T09:58:53.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/11' 2023-08-25T09:58:53.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/16' 2023-08-25T09:58:53.593 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/14' 2023-08-25T09:58:54.604 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/18' 2023-08-25T09:58:54.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/13' 2023-08-25T09:58:54.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/190/17' 2023-08-25T09:58:54.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/190' 2023-08-25T09:58:54.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/19' 2023-08-25T09:58:54.605 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/15' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/20' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/12' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/10' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/11' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/16' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/14' 2023-08-25T09:58:54.606 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/18' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/13' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/104/17' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/104' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/19' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/15' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/20' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/12' 2023-08-25T09:58:54.607 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/10' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/11' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/16' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/14' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/18' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/13' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/164/17' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/164' 2023-08-25T09:58:54.608 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/19' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/15' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/20' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/12' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/10' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/11' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/16' 2023-08-25T09:58:54.609 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/14' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/18' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/13' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/112/17' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/112' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/19' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/15' 2023-08-25T09:58:54.610 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/20' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/12' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/10' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/11' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/16' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/14' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/18' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/13' 2023-08-25T09:58:54.611 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/100/17' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/100' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/19' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/15' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/20' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/12' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/10' 2023-08-25T09:58:54.612 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/11' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/16' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/14' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/18' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/13' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/134/17' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/134' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/19' 2023-08-25T09:58:54.613 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/15' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/20' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/12' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/10' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/11' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/16' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/14' 2023-08-25T09:58:54.614 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/18' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/13' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/115/17' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/115' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/19' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/15' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/20' 2023-08-25T09:58:54.615 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/12' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/10' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/11' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/16' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/14' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/18' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/13' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/116/17' 2023-08-25T09:58:54.616 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/116' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/19' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/15' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/20' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/12' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/10' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/11' 2023-08-25T09:58:54.617 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/16' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/14' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/18' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/13' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/195/17' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/195' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/19' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/15' 2023-08-25T09:58:54.618 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/20' 2023-08-25T09:58:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/12' 2023-08-25T09:58:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/10' 2023-08-25T09:58:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/11' 2023-08-25T09:58:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/16' 2023-08-25T09:58:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/14' 2023-08-25T09:58:54.619 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/18' 2023-08-25T09:58:54.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:58:54.865 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:58:56.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/13' 2023-08-25T09:58:56.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/138/17' 2023-08-25T09:58:56.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/138' 2023-08-25T09:58:56.118 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/19' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/15' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/20' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/12' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/10' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/11' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/16' 2023-08-25T09:58:56.119 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/14' 2023-08-25T09:58:56.120 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/18' 2023-08-25T09:58:56.120 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/13' 2023-08-25T09:58:56.120 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/135/17' 2023-08-25T09:58:56.120 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/135' 2023-08-25T09:58:56.120 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/19' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/15' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/20' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/12' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/10' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/11' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/16' 2023-08-25T09:58:56.121 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/14' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/18' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/13' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/117/17' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/117' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/19' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/15' 2023-08-25T09:58:56.122 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/20' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/12' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/10' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/11' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/16' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/14' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/18' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/13' 2023-08-25T09:58:56.123 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/143/17' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/143' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/19' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/15' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/20' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/12' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/10' 2023-08-25T09:58:56.124 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/11' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/16' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/14' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/18' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/13' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/180/17' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/180' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/19' 2023-08-25T09:58:56.125 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/15' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/20' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/12' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/10' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/11' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/16' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/14' 2023-08-25T09:58:56.126 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/18' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/13' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/126/17' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/126' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/19' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/15' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/20' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/12' 2023-08-25T09:58:56.127 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/10' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/11' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/16' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/14' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/18' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/13' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/161/17' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/161' 2023-08-25T09:58:56.128 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/19' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/15' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/20' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/12' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/10' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/11' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/16' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/14' 2023-08-25T09:58:56.129 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/18' 2023-08-25T09:58:56.130 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/13' 2023-08-25T09:58:56.130 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/176/17' 2023-08-25T09:58:56.130 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/176' 2023-08-25T09:58:56.130 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/19' 2023-08-25T09:58:56.130 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/15' 2023-08-25T09:58:56.131 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/20' 2023-08-25T09:58:56.131 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/12' 2023-08-25T09:58:56.131 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/10' 2023-08-25T09:58:56.131 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/11' 2023-08-25T09:58:56.131 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/16' 2023-08-25T09:58:56.132 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/14' 2023-08-25T09:58:56.132 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/18' 2023-08-25T09:58:56.132 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/13' 2023-08-25T09:58:56.132 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/111/17' 2023-08-25T09:58:56.132 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/111' 2023-08-25T09:58:56.133 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/19' 2023-08-25T09:58:56.133 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/15' 2023-08-25T09:58:56.133 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/20' 2023-08-25T09:58:56.133 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/12' 2023-08-25T09:58:56.133 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/10' 2023-08-25T09:58:56.134 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/11' 2023-08-25T09:58:56.134 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/16' 2023-08-25T09:58:56.134 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/14' 2023-08-25T09:58:56.134 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/18' 2023-08-25T09:58:56.134 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/13' 2023-08-25T09:58:59.024 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/191/17' 2023-08-25T09:58:59.025 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/191' 2023-08-25T09:58:59.025 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/19' 2023-08-25T09:58:59.025 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/15' 2023-08-25T09:58:59.025 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/20' 2023-08-25T09:58:59.025 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/12' 2023-08-25T09:58:59.025 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/10' 2023-08-25T09:58:59.026 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/11' 2023-08-25T09:58:59.026 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/16' 2023-08-25T09:58:59.026 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/14' 2023-08-25T09:58:59.026 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/18' 2023-08-25T09:58:59.027 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/13' 2023-08-25T09:58:59.027 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/184/17' 2023-08-25T09:58:59.028 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/184' 2023-08-25T09:58:59.028 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/19' 2023-08-25T09:58:59.028 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/15' 2023-08-25T09:58:59.028 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/20' 2023-08-25T09:58:59.029 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/12' 2023-08-25T09:58:59.029 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/10' 2023-08-25T09:58:59.029 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/11' 2023-08-25T09:58:59.029 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/16' 2023-08-25T09:58:59.029 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/14' 2023-08-25T09:58:59.030 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/18' 2023-08-25T09:58:59.030 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/13' 2023-08-25T09:58:59.030 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/192/17' 2023-08-25T09:58:59.030 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/192' 2023-08-25T09:58:59.030 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/19' 2023-08-25T09:58:59.031 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/15' 2023-08-25T09:58:59.031 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/20' 2023-08-25T09:58:59.031 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/12' 2023-08-25T09:58:59.031 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/10' 2023-08-25T09:58:59.031 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/11' 2023-08-25T09:58:59.032 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/16' 2023-08-25T09:58:59.032 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/14' 2023-08-25T09:58:59.032 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/18' 2023-08-25T09:58:59.032 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/13' 2023-08-25T09:58:59.033 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/103/17' 2023-08-25T09:58:59.033 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/103' 2023-08-25T09:58:59.033 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/19' 2023-08-25T09:58:59.033 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/15' 2023-08-25T09:58:59.033 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/20' 2023-08-25T09:58:59.034 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/12' 2023-08-25T09:58:59.034 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/10' 2023-08-25T09:58:59.034 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/11' 2023-08-25T09:58:59.034 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/16' 2023-08-25T09:58:59.035 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/14' 2023-08-25T09:58:59.035 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/18' 2023-08-25T09:58:59.035 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/13' 2023-08-25T09:58:59.035 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/128/17' 2023-08-25T09:58:59.035 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/128' 2023-08-25T09:58:59.036 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/19' 2023-08-25T09:58:59.036 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/15' 2023-08-25T09:58:59.036 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/20' 2023-08-25T09:58:59.036 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/12' 2023-08-25T09:58:59.037 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/10' 2023-08-25T09:58:59.037 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/11' 2023-08-25T09:58:59.037 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/16' 2023-08-25T09:58:59.037 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/14' 2023-08-25T09:58:59.037 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/18' 2023-08-25T09:58:59.038 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/13' 2023-08-25T09:58:59.038 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/144/17' 2023-08-25T09:58:59.038 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/144' 2023-08-25T09:58:59.038 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/19' 2023-08-25T09:58:59.038 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/15' 2023-08-25T09:58:59.039 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/20' 2023-08-25T09:58:59.039 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/12' 2023-08-25T09:58:59.039 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/10' 2023-08-25T09:58:59.039 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/11' 2023-08-25T09:58:59.040 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/16' 2023-08-25T09:58:59.040 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/14' 2023-08-25T09:58:59.040 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/18' 2023-08-25T09:58:59.040 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/13' 2023-08-25T09:58:59.040 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/146/17' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/146' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/19' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/15' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/20' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/12' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/10' 2023-08-25T09:58:59.041 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/11' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/16' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/14' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/18' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/13' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/141/17' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/141' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/19' 2023-08-25T09:58:59.042 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/15' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/20' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/12' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/10' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/11' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/16' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/14' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/18' 2023-08-25T09:58:59.043 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/13' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/182/17' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/182' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/19' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/15' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/20' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/12' 2023-08-25T09:58:59.044 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/10' 2023-08-25T09:58:59.045 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/11' 2023-08-25T09:58:59.045 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/16' 2023-08-25T09:58:59.045 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/14' 2023-08-25T09:58:59.045 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/18' 2023-08-25T09:58:59.045 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/13' 2023-08-25T09:58:59.045 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/107/17' 2023-08-25T09:59:00.158 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/107' 2023-08-25T09:59:00.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/19' 2023-08-25T09:59:00.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/15' 2023-08-25T09:59:00.159 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/20' 2023-08-25T09:59:00.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/12' 2023-08-25T09:59:00.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/10' 2023-08-25T09:59:00.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/11' 2023-08-25T09:59:00.160 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/16' 2023-08-25T09:59:00.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/14' 2023-08-25T09:59:00.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/18' 2023-08-25T09:59:00.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/13' 2023-08-25T09:59:00.161 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/178/17' 2023-08-25T09:59:00.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/178' 2023-08-25T09:59:00.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/19' 2023-08-25T09:59:00.162 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/15' 2023-08-25T09:59:00.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/20' 2023-08-25T09:59:00.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/12' 2023-08-25T09:59:00.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/10' 2023-08-25T09:59:00.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/11' 2023-08-25T09:59:00.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/16' 2023-08-25T09:59:00.163 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/14' 2023-08-25T09:59:00.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/18' 2023-08-25T09:59:00.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/13' 2023-08-25T09:59:00.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/156/17' 2023-08-25T09:59:00.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/156' 2023-08-25T09:59:00.164 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/19' 2023-08-25T09:59:00.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/15' 2023-08-25T09:59:00.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/20' 2023-08-25T09:59:00.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/12' 2023-08-25T09:59:00.165 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/10' 2023-08-25T09:59:00.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/11' 2023-08-25T09:59:00.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/16' 2023-08-25T09:59:00.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/14' 2023-08-25T09:59:00.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/18' 2023-08-25T09:59:00.166 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/13' 2023-08-25T09:59:00.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/131/17' 2023-08-25T09:59:00.167 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/131' 2023-08-25T09:59:00.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/19' 2023-08-25T09:59:00.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/15' 2023-08-25T09:59:00.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/20' 2023-08-25T09:59:00.168 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/12' 2023-08-25T09:59:00.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/10' 2023-08-25T09:59:00.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/11' 2023-08-25T09:59:00.169 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/16' 2023-08-25T09:59:00.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/14' 2023-08-25T09:59:00.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/18' 2023-08-25T09:59:00.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/13' 2023-08-25T09:59:00.170 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/123/17' 2023-08-25T09:59:00.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/123' 2023-08-25T09:59:00.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/19' 2023-08-25T09:59:00.171 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/15' 2023-08-25T09:59:00.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/20' 2023-08-25T09:59:00.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/12' 2023-08-25T09:59:00.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/10' 2023-08-25T09:59:00.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/11' 2023-08-25T09:59:00.172 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/16' 2023-08-25T09:59:00.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/14' 2023-08-25T09:59:00.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/18' 2023-08-25T09:59:00.173 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/13' 2023-08-25T09:59:00.174 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/160/17' 2023-08-25T09:59:00.175 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/160' 2023-08-25T09:59:00.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/19' 2023-08-25T09:59:00.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/15' 2023-08-25T09:59:00.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/20' 2023-08-25T09:59:00.176 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/12' 2023-08-25T09:59:00.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/10' 2023-08-25T09:59:00.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/11' 2023-08-25T09:59:00.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/16' 2023-08-25T09:59:00.177 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/14' 2023-08-25T09:59:00.178 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/18' 2023-08-25T09:59:00.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/13' 2023-08-25T09:59:00.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/165/17' 2023-08-25T09:59:00.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/165' 2023-08-25T09:59:00.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/19' 2023-08-25T09:59:00.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/15' 2023-08-25T09:59:00.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/20' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/12' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/10' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/11' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/16' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/14' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/18' 2023-08-25T09:59:00.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/13' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/101/17' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/101' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/19' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/15' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/20' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/12' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/10' 2023-08-25T09:59:00.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/11' 2023-08-25T09:59:00.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/16' 2023-08-25T09:59:00.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/14' 2023-08-25T09:59:00.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/18' 2023-08-25T09:59:00.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/13' 2023-08-25T09:59:00.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/148/17' 2023-08-25T09:59:00.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/148' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/19' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/15' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/20' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/12' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/10' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/11' 2023-08-25T09:59:00.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/16' 2023-08-25T09:59:00.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/14' 2023-08-25T09:59:00.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/18' 2023-08-25T09:59:00.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/13' 2023-08-25T09:59:00.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/168/17' 2023-08-25T09:59:00.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/168' 2023-08-25T09:59:01.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/19' 2023-08-25T09:59:01.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/15' 2023-08-25T09:59:01.227 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/20' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/12' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/10' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/11' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/16' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/14' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/18' 2023-08-25T09:59:01.228 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/13' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/179/17' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/179' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/19' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/15' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/20' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/12' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/10' 2023-08-25T09:59:01.229 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/11' 2023-08-25T09:59:01.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/16' 2023-08-25T09:59:01.230 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/14' 2023-08-25T09:59:01.232 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/18' 2023-08-25T09:59:01.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/13' 2023-08-25T09:59:01.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/132/17' 2023-08-25T09:59:01.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/132' 2023-08-25T09:59:01.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/19' 2023-08-25T09:59:01.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/15' 2023-08-25T09:59:01.233 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/20' 2023-08-25T09:59:01.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/12' 2023-08-25T09:59:01.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/10' 2023-08-25T09:59:01.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/11' 2023-08-25T09:59:01.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/16' 2023-08-25T09:59:01.234 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/14' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/18' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/13' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/199/17' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/199' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/19' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/15' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/20' 2023-08-25T09:59:01.235 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/12' 2023-08-25T09:59:01.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/10' 2023-08-25T09:59:01.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/11' 2023-08-25T09:59:01.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/16' 2023-08-25T09:59:01.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/14' 2023-08-25T09:59:01.236 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/18' 2023-08-25T09:59:01.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/13' 2023-08-25T09:59:01.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/163/17' 2023-08-25T09:59:01.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/163' 2023-08-25T09:59:01.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/19' 2023-08-25T09:59:01.237 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/15' 2023-08-25T09:59:01.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/20' 2023-08-25T09:59:01.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/12' 2023-08-25T09:59:01.238 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/10' 2023-08-25T09:59:01.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/11' 2023-08-25T09:59:01.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/16' 2023-08-25T09:59:01.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/14' 2023-08-25T09:59:01.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/18' 2023-08-25T09:59:01.240 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/13' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/153/17' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/153' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/19' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/15' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/20' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/12' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/10' 2023-08-25T09:59:01.241 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/11' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/16' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/14' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/18' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/13' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/122/17' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/122' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/19' 2023-08-25T09:59:01.242 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/15' 2023-08-25T09:59:01.243 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/20' 2023-08-25T09:59:01.243 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/12' 2023-08-25T09:59:01.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/10' 2023-08-25T09:59:01.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/11' 2023-08-25T09:59:01.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/16' 2023-08-25T09:59:01.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/14' 2023-08-25T09:59:01.244 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/18' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/13' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/162/17' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/162' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/19' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/15' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/20' 2023-08-25T09:59:01.245 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/12' 2023-08-25T09:59:01.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/10' 2023-08-25T09:59:01.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/11' 2023-08-25T09:59:01.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/16' 2023-08-25T09:59:01.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/14' 2023-08-25T09:59:01.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/18' 2023-08-25T09:59:01.246 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/13' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/197/17' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/197' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/19' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/15' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/20' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/12' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/10' 2023-08-25T09:59:01.247 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/11' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/16' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/14' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/18' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/13' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/113/17' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/113' 2023-08-25T09:59:01.248 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/19' 2023-08-25T09:59:02.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/15' 2023-08-25T09:59:02.179 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/20' 2023-08-25T09:59:02.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/12' 2023-08-25T09:59:02.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/10' 2023-08-25T09:59:02.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/11' 2023-08-25T09:59:02.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/16' 2023-08-25T09:59:02.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/14' 2023-08-25T09:59:02.180 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/18' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/13' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/152/17' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/152' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/19' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/15' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/20' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/12' 2023-08-25T09:59:02.181 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/10' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/11' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/16' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/14' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/18' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/13' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/121/17' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/121' 2023-08-25T09:59:02.182 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/19' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/15' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/20' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/12' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/10' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/11' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/16' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/14' 2023-08-25T09:59:02.183 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/18' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/13' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/118/17' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/118' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/19' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/15' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/20' 2023-08-25T09:59:02.184 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/12' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/10' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/11' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/16' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/14' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/18' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/13' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/196/17' 2023-08-25T09:59:02.185 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/196' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/19' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/15' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/20' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/12' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/10' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/11' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/16' 2023-08-25T09:59:02.186 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/14' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/18' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/13' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/124/17' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/124' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/19' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/15' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/20' 2023-08-25T09:59:02.187 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/12' 2023-08-25T09:59:02.188 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/10' 2023-08-25T09:59:02.188 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/11' 2023-08-25T09:59:02.188 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/16' 2023-08-25T09:59:02.188 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/14' 2023-08-25T09:59:02.188 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/18' 2023-08-25T09:59:02.188 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/13' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/188/17' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/188' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/19' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/15' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/20' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/12' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/10' 2023-08-25T09:59:02.189 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/11' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/16' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/14' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/18' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/13' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/145/17' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/145' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/19' 2023-08-25T09:59:02.190 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/15' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/20' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/12' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/10' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/11' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/16' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/14' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/18' 2023-08-25T09:59:02.191 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/13' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/130/17' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/130' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/19' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/15' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/20' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/12' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/10' 2023-08-25T09:59:02.192 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/11' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/16' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/14' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/18' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/13' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/193/17' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/193' 2023-08-25T09:59:02.193 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/19' 2023-08-25T09:59:02.194 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/15' 2023-08-25T09:59:04.943 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/20' 2023-08-25T09:59:04.943 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/12' 2023-08-25T09:59:04.943 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/10' 2023-08-25T09:59:04.943 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/11' 2023-08-25T09:59:04.943 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/16' 2023-08-25T09:59:04.944 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/14' 2023-08-25T09:59:04.944 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/18' 2023-08-25T09:59:04.945 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/13' 2023-08-25T09:59:04.945 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/175/17' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/175' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/19' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/15' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/20' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/12' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/10' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/11' 2023-08-25T09:59:04.946 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/16' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/14' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/18' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/13' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/181/17' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/181' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/19' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/15' 2023-08-25T09:59:04.947 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/20' 2023-08-25T09:59:04.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/12' 2023-08-25T09:59:04.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/10' 2023-08-25T09:59:04.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/11' 2023-08-25T09:59:04.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/16' 2023-08-25T09:59:04.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/14' 2023-08-25T09:59:04.948 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/18' 2023-08-25T09:59:04.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/13' 2023-08-25T09:59:04.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/127/17' 2023-08-25T09:59:04.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/127' 2023-08-25T09:59:04.949 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/19' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/15' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/20' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/12' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/10' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/11' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/16' 2023-08-25T09:59:04.950 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/14' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/18' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/13' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/108/17' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/108' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/19' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/15' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/20' 2023-08-25T09:59:04.951 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/12' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/10' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/11' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/16' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/14' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/18' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/13' 2023-08-25T09:59:04.952 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/183/17' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/183' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/19' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/15' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/20' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/12' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/10' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/11' 2023-08-25T09:59:04.953 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/16' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/14' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/18' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/13' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/177/17' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/177' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/19' 2023-08-25T09:59:04.954 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/15' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/20' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/12' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/10' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/11' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/16' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/14' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/18' 2023-08-25T09:59:04.955 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/13' 2023-08-25T09:59:04.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/169/17' 2023-08-25T09:59:04.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/169' 2023-08-25T09:59:04.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/19' 2023-08-25T09:59:04.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/15' 2023-08-25T09:59:04.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/20' 2023-08-25T09:59:04.956 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/12' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/10' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/11' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/16' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/14' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/18' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/13' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/154/17' 2023-08-25T09:59:04.957 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/154' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/19' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/15' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/20' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/12' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/10' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/11' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/16' 2023-08-25T09:59:04.958 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/14' 2023-08-25T09:59:04.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/18' 2023-08-25T09:59:04.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/13' 2023-08-25T09:59:04.959 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/171/17' 2023-08-25T09:59:04.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/171' 2023-08-25T09:59:04.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/19' 2023-08-25T09:59:04.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/15' 2023-08-25T09:59:04.960 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/20' 2023-08-25T09:59:05.275 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/12' 2023-08-25T09:59:05.276 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/10' 2023-08-25T09:59:05.276 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/11' 2023-08-25T09:59:05.276 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/16' 2023-08-25T09:59:05.276 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/14' 2023-08-25T09:59:05.276 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/18' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/13' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/139/17' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/139' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/19' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/15' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/20' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/12' 2023-08-25T09:59:05.277 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/10' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/11' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/16' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/14' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/18' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/13' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/102/17' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/102' 2023-08-25T09:59:05.278 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/19' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/15' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/20' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/12' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/10' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/11' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/16' 2023-08-25T09:59:05.279 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/14' 2023-08-25T09:59:05.280 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/18' 2023-08-25T09:59:05.280 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/13' 2023-08-25T09:59:05.280 INFO:tasks.workunit.client.0.smithi071.stdout:removed './copyofsnap1/test1/129/17' 2023-08-25T09:59:05.280 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1/129' 2023-08-25T09:59:05.280 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1/test1' 2023-08-25T09:59:05.280 INFO:tasks.workunit.client.0.smithi071.stdout:removed directory './copyofsnap1' 2023-08-25T09:59:05.281 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:59:05.282 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:59:05.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:59:05.419 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-08-25T09:59:05.419 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-08-25T09:59:05.510 INFO:tasks.workunit.client.0.smithi071.stderr:+ set -e 2023-08-25T09:59:05.510 INFO:tasks.workunit.client.0.smithi071.stderr:+ touch foo 2023-08-25T09:59:05.520 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod +x foo 2023-08-25T09:59:05.523 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir .snap/s 2023-08-25T09:59:05.570 INFO:tasks.workunit.client.0.smithi071.stderr:+ find .snap/s/foo -executable 2023-08-25T09:59:05.571 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep foo 2023-08-25T09:59:05.596 INFO:tasks.workunit.client.0.smithi071.stdout:.snap/s/foo 2023-08-25T09:59:05.597 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir .snap/s 2023-08-25T09:59:05.634 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm foo 2023-08-25T09:59:06.152 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:59:06.153 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:59:06.154 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:59:06.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:59:06.293 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-08-25T09:59:06.293 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-08-25T09:59:06.381 INFO:tasks.workunit.client.0.smithi071.stderr:+ set -e 2023-08-25T09:59:06.381 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir foo 2023-08-25T09:59:06.388 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo x 2023-08-25T09:59:06.392 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:07.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:07.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.341+0000 7f275f59e640 1 Processor -- start 2023-08-25T09:59:07.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.341+0000 7f275f59e640 1 -- start start 2023-08-25T09:59:07.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.343+0000 7f275f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 0x7f27600ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.343+0000 7f275f59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27600b1af0 con 0x7f27600cb190 2023-08-25T09:59:07.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.343+0000 7f275f59e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276015d190 con 0x7f276015cd00 2023-08-25T09:59:07.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.343+0000 7f275f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27600c98f0 con 0x7f27600cb570 2023-08-25T09:59:07.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.343+0000 7f275e59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f27600cb190 0x7f27600c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48248/0 (socket says 172.21.15.71:48248) 2023-08-25T09:59:07.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.343+0000 7f275e59c640 1 -- 172.21.15.71:0/1495547524 learned_addr learned my addr 172.21.15.71:0/1495547524 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:07.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f2757fff640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 0x7f27600ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298961541 0 0) 0x7f27600b1af0 con 0x7f27600cb190 2023-08-25T09:59:07.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f274c003520 con 0x7f27600cb190 2023-08-25T09:59:07.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 821262213 0 0) 0x7f274c003520 con 0x7f27600cb190 2023-08-25T09:59:07.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 msgr2=0x7f27600ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:07.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 0x7f27600ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 >> v1:172.21.15.164:6789/0 conn(0x7f276015cd00 legacy=0x7f2760158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27600b0fb0 con 0x7f27600cb190 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f2750001670 con 0x7f27600cb190 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 >> v1:172.21.15.71:6789/0 conn(0x7f27600cb190 legacy=0x7f27600c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c003a10 0x7f274c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f274c0047c0 0x7f274c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 0x7f274c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27600c98f0 con 0x7f274c003a10 2023-08-25T09:59:07.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c00bb10 con 0x7f274c0047c0 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.344+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c00bcd0 con 0x7f27600cb570 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275e59c640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c003a10 0x7f274c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275ed9d640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 0x7f274c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275e59c640 1 -- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 msgr2=0x7f274c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f2757fff640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f274c0047c0 0x7f274c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275e59c640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27600cb570 0x7f274c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275e59c640 1 -- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f274c0047c0 msgr2=0x7f274c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:07.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275e59c640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f274c0047c0 0x7f274c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.345+0000 7f275e59c640 1 -- 172.21.15.71:0/1495547524 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c003520 con 0x7f274c003a10 2023-08-25T09:59:07.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.346+0000 7f275e59c640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c003a10 0x7f274c003e30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f275000c190 tx=0x7f274c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189e2a528d60ce40 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:07.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.346+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2750005ad0 con 0x7f274c003a10 2023-08-25T09:59:07.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.346+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2750005c30 con 0x7f274c003a10 2023-08-25T09:59:07.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.346+0000 7f275dd9b640 1 -- 172.21.15.71:0/1495547524 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f275002c8a0 con 0x7f274c003a10 2023-08-25T09:59:07.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275f59e640 1 -- 172.21.15.71:0/1495547524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c003a10 msgr2=0x7f274c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275f59e640 1 --2- 172.21.15.71:0/1495547524 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f274c003a10 0x7f274c003e30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f275000c190 tx=0x7f274c0057f0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275ed9d640 1 -- 172.21.15.71:0/1495547524 reap_dead start 2023-08-25T09:59:07.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275f59e640 1 -- 172.21.15.71:0/1495547524 shutdown_connections 2023-08-25T09:59:07.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275f59e640 1 -- 172.21.15.71:0/1495547524 >> 172.21.15.71:0/1495547524 conn(0x7f276006ae20 msgr2=0x7f276006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:07.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275f59e640 1 -- 172.21.15.71:0/1495547524 shutdown_connections 2023-08-25T09:59:07.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.347+0000 7f275f59e640 1 -- 172.21.15.71:0/1495547524 wait complete. 2023-08-25T09:59:07.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.349+0000 7f275f59e640 1 Processor -- start 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.349+0000 7f275f59e640 1 -- start start 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f275f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f274c003a10 0x7f2760151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f275f59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 0x7f27600c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f275f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27600c2220 0x7f2760154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f275f59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27600c3330 con 0x7f2760155d00 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f275f59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2760156780 con 0x7f27600c2220 2023-08-25T09:59:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f275f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27601521c0 con 0x7f274c003a10 2023-08-25T09:59:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f2757fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 0x7f27600c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f2757fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 0x7f27600c1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42924/0 (socket says 172.21.15.71:42924) 2023-08-25T09:59:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.350+0000 7f2757fff640 1 -- 172.21.15.71:0/2617339214 learned_addr learned my addr 172.21.15.71:0/2617339214 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f275e59c640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f274c003a10 0x7f2760151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2757fff640 1 -- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f274c003a10 msgr2=0x7f2760151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2757fff640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f274c003a10 0x7f2760151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f275ed9d640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27600c2220 0x7f2760154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2757fff640 1 -- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27600c2220 msgr2=0x7f2760154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2757fff640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27600c2220 0x7f2760154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2757fff640 1 -- 172.21.15.71:0/2617339214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27601f14a0 con 0x7f2760155d00 2023-08-25T09:59:07.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2757fff640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 0x7f27600c1ce0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2748010860 tx=0x7f2748010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2748009610 con 0x7f2760155d00 2023-08-25T09:59:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274800eb60 con 0x7f2760155d00 2023-08-25T09:59:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274801c400 con 0x7f2760155d00 2023-08-25T09:59:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.351+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27601f1780 con 0x7f2760155d00 2023-08-25T09:59:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.352+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27601f1d40 con 0x7f2760155d00 2023-08-25T09:59:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.353+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276004e420 con 0x7f2760155d00 2023-08-25T09:59:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.353+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f274801c560 con 0x7f2760155d00 2023-08-25T09:59:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.354+0000 7f2756ffd640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f274006a080 0x7f274006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.354+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f2748088f50 con 0x7f2760155d00 2023-08-25T09:59:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.355+0000 7f275e59c640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f274006a080 0x7f274006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.355+0000 7f275e59c640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f274006a080 0x7f274006c540 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2750034000 tx=0x7f2750009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.359+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f27480541d0 con 0x7f2760155d00 2023-08-25T09:59:07.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.628+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2760152d50 con 0x7f2760155d00 2023-08-25T09:59:07.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.629+0000 7f2756ffd640 1 -- 172.21.15.71:0/2617339214 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f27480577f0 con 0x7f2760155d00 2023-08-25T09:59:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:07.632 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:03.969309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:active","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:07.633 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 77 2023-08-25T09:59:07.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f274006a080 msgr2=0x7f274006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f274006a080 0x7f274006c540 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2750034000 tx=0x7f2750009030 comp rx=0 tx=0).stop 2023-08-25T09:59:07.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 msgr2=0x7f27600c1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 0x7f27600c1ce0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2748010860 tx=0x7f2748010d20 comp rx=0 tx=0).stop 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 shutdown_connections 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27600c2220 0x7f2760154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f274006a080 0x7f274006c540 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2760155d00 0x7f27600c1ce0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 --2- 172.21.15.71:0/2617339214 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f274c003a10 0x7f2760151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 >> 172.21.15.71:0/2617339214 conn(0x7f276006ae20 msgr2=0x7f27600cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:07.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 shutdown_connections 2023-08-25T09:59:07.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.632+0000 7f275f59e640 1 -- 172.21.15.71:0/2617339214 wait complete. 2023-08-25T09:59:07.654 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 5 -> 1 2023-08-25T09:59:07.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2023-08-25T09:59:07.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.856+0000 7f44b2b79640 1 Processor -- start 2023-08-25T09:59:07.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.856+0000 7f44b2b79640 1 -- start start 2023-08-25T09:59:07.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.857+0000 7f44b2b79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac06cc70 0x7f44ac0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.857+0000 7f44b2b79640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44ac0c3940 0x7f44ac1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.857+0000 7f44b2b79640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ac06a960 con 0x7f44ac1594d0 2023-08-25T09:59:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.857+0000 7f44b2b79640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44ac06ac20 con 0x7f44ac0c3940 2023-08-25T09:59:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.857+0000 7f44b2b79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac06af00 con 0x7f44ac06cc70 2023-08-25T09:59:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b0915640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f44ac1594d0 0x7f44ac14f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48264/0 (socket says 172.21.15.71:48264) 2023-08-25T09:59:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b0915640 1 -- 172.21.15.71:0/1848027709 learned_addr learned my addr 172.21.15.71:0/1848027709 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b1116640 1 --2- 172.21.15.71:0/1848027709 >> v2:172.21.15.71:3300/0 conn(0x7f44ac0c3940 0x7f44ac1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44abfff640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac06cc70 0x7f44ac0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44ab7fe640 1 -- 172.21.15.71:0/1848027709 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191426980 0 0) 0x7f44ac06a960 con 0x7f44ac1594d0 2023-08-25T09:59:07.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44ab7fe640 1 -- 172.21.15.71:0/1848027709 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f449c003520 con 0x7f44ac1594d0 2023-08-25T09:59:07.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b1116640 1 -- 172.21.15.71:0/1848027709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac06cc70 msgr2=0x7f44ac0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:07.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b1116640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac06cc70 0x7f44ac0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b1116640 1 -- 172.21.15.71:0/1848027709 >> v1:172.21.15.71:6789/0 conn(0x7f44ac1594d0 legacy=0x7f44ac14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44b1116640 1 -- 172.21.15.71:0/1848027709 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac0c2e60 con 0x7f44ac0c3940 2023-08-25T09:59:07.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.858+0000 7f44abfff640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac06cc70 0x7f44ac0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:07.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.859+0000 7f44b1116640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1598b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009d90 tx=0x7f44a402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298b056a25f57c8d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:07.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.860+0000 7f44ab7fe640 1 -- 172.21.15.71:0/1848027709 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a402cc60 con 0x7f44ac0c3940 2023-08-25T09:59:07.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.860+0000 7f44ab7fe640 1 -- 172.21.15.71:0/1848027709 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a402cdc0 con 0x7f44ac0c3940 2023-08-25T09:59:07.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.860+0000 7f44ab7fe640 1 -- 172.21.15.71:0/1848027709 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a4034660 con 0x7f44ac0c3940 2023-08-25T09:59:07.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.860+0000 7f44b2b79640 1 -- 172.21.15.71:0/1848027709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 msgr2=0x7f44ac1598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.860+0000 7f44b2b79640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1598b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009d90 tx=0x7f44a402beb0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.861+0000 7f44b2b79640 1 -- 172.21.15.71:0/1848027709 shutdown_connections 2023-08-25T09:59:07.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.861+0000 7f44b2b79640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1598b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.861+0000 7f44b2b79640 1 --2- 172.21.15.71:0/1848027709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac06cc70 0x7f44ac0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.861+0000 7f44b2b79640 1 -- 172.21.15.71:0/1848027709 >> 172.21.15.71:0/1848027709 conn(0x7f44ac0654f0 msgr2=0x7f44ac065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:07.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.861+0000 7f44b2b79640 1 -- 172.21.15.71:0/1848027709 shutdown_connections 2023-08-25T09:59:07.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.861+0000 7f44b2b79640 1 -- 172.21.15.71:0/1848027709 wait complete. 2023-08-25T09:59:07.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.862+0000 7f44b2b79640 1 Processor -- start 2023-08-25T09:59:07.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 -- start start 2023-08-25T09:59:07.873 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44ac06cc70 0x7f44ac1e4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac1594d0 0x7f44ac147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac149d60 con 0x7f44ac0c3940 2023-08-25T09:59:07.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac149f40 con 0x7f44ac06cc70 2023-08-25T09:59:07.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.863+0000 7f44b2b79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac14a220 con 0x7f44ac1594d0 2023-08-25T09:59:07.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1e50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42940/0 (socket says 172.21.15.71:42940) 2023-08-25T09:59:07.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 -- 172.21.15.71:0/2566365055 learned_addr learned my addr 172.21.15.71:0/2566365055 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:07.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 -- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac1594d0 msgr2=0x7f44ac147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac1594d0 0x7f44ac147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 -- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44ac06cc70 msgr2=0x7f44ac1e4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:07.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44b0915640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44ac06cc70 0x7f44ac1e4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44ac06cc70 0x7f44ac1e4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:07.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.864+0000 7f44abfff640 1 -- 172.21.15.71:0/2566365055 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a4009a70 con 0x7f44ac0c3940 2023-08-25T09:59:07.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.865+0000 7f44abfff640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1e50a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f449800c770 tx=0x7f449800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:07.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.865+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498007820 con 0x7f44ac0c3940 2023-08-25T09:59:07.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.865+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4498007980 con 0x7f44ac0c3940 2023-08-25T09:59:07.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.865+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498007cf0 con 0x7f44ac0c3940 2023-08-25T09:59:07.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.865+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac14a500 con 0x7f44ac0c3940 2023-08-25T09:59:07.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.865+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac0c5400 con 0x7f44ac0c3940 2023-08-25T09:59:07.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.866+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ac066510 con 0x7f44ac0c3940 2023-08-25T09:59:07.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.867+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f44980113f0 con 0x7f44ac0c3940 2023-08-25T09:59:07.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.868+0000 7f44a97fa640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449006a050 0x7f449006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:07.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.868+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f4498088b10 con 0x7f44ac0c3940 2023-08-25T09:59:07.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.868+0000 7f44b0915640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449006a050 0x7f449006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:07.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.869+0000 7f44b0915640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449006a050 0x7f449006c510 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000f60 tx=0x7f44a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:07.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:07.872+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4498053ce0 con 0x7f44ac0c3940 2023-08-25T09:59:07.952 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir foo/.snap/ss 2023-08-25T09:59:07.991 INFO:tasks.workunit.client.0.smithi071.stderr:+ ln foo/x foo/xx 2023-08-25T09:59:08.002 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat foo/.snap/ss/x 2023-08-25T09:59:08.026 INFO:tasks.workunit.client.0.smithi071.stdout:x 2023-08-25T09:59:08.027 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir foo/.snap/ss 2023-08-25T09:59:08.053 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo a 2023-08-25T09:59:08.061 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo b 2023-08-25T09:59:08.067 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir foo/.snap/s 2023-08-25T09:59:08.101 INFO:tasks.workunit.client.0.smithi071.stderr:++ cat foo/.snap/s/a 2023-08-25T09:59:08.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.120+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f44ac0c3110 con 0x7f44ac0c3940 2023-08-25T09:59:08.134 INFO:tasks.workunit.client.0.smithi071.stderr:+ r=a 2023-08-25T09:59:08.134 INFO:tasks.workunit.client.0.smithi071.stderr:+ '[' -z a ']' 2023-08-25T09:59:08.134 INFO:tasks.workunit.client.0.smithi071.stderr:+ ln foo/b foo/b2 2023-08-25T09:59:08.140 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat foo/.snap/s/b 2023-08-25T09:59:08.159 INFO:tasks.workunit.client.0.smithi071.stdout:b 2023-08-25T09:59:08.160 INFO:tasks.workunit.client.0.smithi071.stdout:this used to hang: 2023-08-25T09:59:08.160 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 'this used to hang:' 2023-08-25T09:59:08.160 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo more 2023-08-25T09:59:08.169 INFO:tasks.workunit.client.0.smithi071.stdout:oh, it didn't hang! good job. 2023-08-25T09:59:08.170 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2023-08-25T09:59:08.170 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat foo/b 2023-08-25T09:59:08.177 INFO:tasks.workunit.client.0.smithi071.stdout:b 2023-08-25T09:59:08.177 INFO:tasks.workunit.client.0.smithi071.stdout:more 2023-08-25T09:59:08.178 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir foo/.snap/s 2023-08-25T09:59:08.204 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm -r foo 2023-08-25T09:59:08.277 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:59:08.278 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:59:08.278 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:59:08.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:59:08.728 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-08-25T09:59:08.729 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-08-25T09:59:08.818 INFO:tasks.workunit.client.0.smithi071.stderr:+ set -e 2023-08-25T09:59:08.819 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:08.826 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir .snap/bar 2023-08-25T09:59:08.862 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/foo 2023-08-25T09:59:08.887 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/foo 2023-08-25T09:59:08.916 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:08.926 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/bar/a 2023-08-25T09:59:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: File: .snap/bar/a 2023-08-25T09:59:08.967 INFO:tasks.workunit.client.0.smithi071.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2023-08-25T09:59:08.968 INFO:tasks.workunit.client.0.smithi071.stdout:Device: 27h/39d Inode: 845524441763063 Links: 2 2023-08-25T09:59:08.968 INFO:tasks.workunit.client.0.smithi071.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-08-25T09:59:08.968 INFO:tasks.workunit.client.0.smithi071.stdout:Context: system_u:object_r:fusefs_t:s0 2023-08-25T09:59:08.968 INFO:tasks.workunit.client.0.smithi071.stdout:Access: 2023-08-25 09:59:08.821311409 +0000 2023-08-25T09:59:08.969 INFO:tasks.workunit.client.0.smithi071.stdout:Modify: 2023-08-25 09:59:08.821311409 +0000 2023-08-25T09:59:08.969 INFO:tasks.workunit.client.0.smithi071.stdout:Change: 2023-08-25 09:59:08.821311409 +0000 2023-08-25T09:59:08.969 INFO:tasks.workunit.client.0.smithi071.stdout: Birth: - 2023-08-25T09:59:08.970 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir .snap/bar 2023-08-25T09:59:08.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.985+0000 7f44a97fa640 1 -- 172.21.15.71:0/2566365055 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v78) v1 ==== 105+0+0 (secure 0 0 0) 0x7f4498057300 con 0x7f44ac0c3940 2023-08-25T09:59:08.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449006a050 msgr2=0x7f449006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:08.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449006a050 0x7f449006c510 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f44a0000f60 tx=0x7f44a0009040 comp rx=0 tx=0).stop 2023-08-25T09:59:08.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 msgr2=0x7f44ac1e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:08.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1e50a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f449800c770 tx=0x7f449800cc30 comp rx=0 tx=0).stop 2023-08-25T09:59:08.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 shutdown_connections 2023-08-25T09:59:08.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f449006a050 0x7f449006c510 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:08.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44ac1594d0 0x7f44ac147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:08.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44ac0c3940 0x7f44ac1e50a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:08.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 --2- 172.21.15.71:0/2566365055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44ac06cc70 0x7f44ac1e4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:08.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.990+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 >> 172.21.15.71:0/2566365055 conn(0x7f44ac0654f0 msgr2=0x7f44ac06b920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:08.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.991+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 shutdown_connections 2023-08-25T09:59:08.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:08.991+0000 7f44b2b79640 1 -- 172.21.15.71:0/2566365055 wait complete. 2023-08-25T09:59:08.998 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:59:08.999 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:59:09.000 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:59:09.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:59:09.013 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-08-25T09:59:09.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 Processor -- start 2023-08-25T09:59:09.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 -- start start 2023-08-25T09:59:09.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f04200c3a40 0x7f04200c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:09.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04201509d0 0x7f0420150e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:09.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04200c1dc0 con 0x7f04200c3660 2023-08-25T09:59:09.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0420157a80 con 0x7f04200c3a40 2023-08-25T09:59:09.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.255+0000 7f0426657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0420151360 con 0x7f04201509d0 2023-08-25T09:59:09.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041ffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f04200c3660 0x7f042006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48272/0 (socket says 172.21.15.71:48272) 2023-08-25T09:59:09.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041ffff640 1 -- 172.21.15.71:0/4241545644 learned_addr learned my addr 172.21.15.71:0/4241545644 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:09.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041f7fe640 1 --2- 172.21.15.71:0/4241545644 >> v2:172.21.15.71:3300/0 conn(0x7f04200c3a40 0x7f04200c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f0424bf4640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04201509d0 0x7f0420150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041effd640 1 -- 172.21.15.71:0/4241545644 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819664094 0 0) 0x7f04200c1dc0 con 0x7f04200c3660 2023-08-25T09:59:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041effd640 1 -- 172.21.15.71:0/4241545644 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0408003520 con 0x7f04200c3660 2023-08-25T09:59:09.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041f7fe640 1 -- 172.21.15.71:0/4241545644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04201509d0 msgr2=0x7f0420150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:09.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041f7fe640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04201509d0 0x7f0420150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041f7fe640 1 -- 172.21.15.71:0/4241545644 >> v1:172.21.15.71:6789/0 conn(0x7f04200c3660 legacy=0x7f042006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:09.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f041f7fe640 1 -- 172.21.15.71:0/4241545644 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0420156e00 con 0x7f04200c3a40 2023-08-25T09:59:09.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.256+0000 7f0424bf4640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04201509d0 0x7f0420150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:09.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f041f7fe640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04200c3a40 0x7f04200c1710 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f04140097a0 tx=0x7f041402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa97a5d6e9b5b153 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:09.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f041effd640 1 -- 172.21.15.71:0/4241545644 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414030800 con 0x7f04200c3a40 2023-08-25T09:59:09.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f041effd640 1 -- 172.21.15.71:0/4241545644 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0414030960 con 0x7f04200c3a40 2023-08-25T09:59:09.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f041effd640 1 -- 172.21.15.71:0/4241545644 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0414030cd0 con 0x7f04200c3a40 2023-08-25T09:59:09.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f0426657640 1 -- 172.21.15.71:0/4241545644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04200c3a40 msgr2=0x7f04200c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:09.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f0426657640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04200c3a40 0x7f04200c1710 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f04140097a0 tx=0x7f041402b940 comp rx=0 tx=0).stop 2023-08-25T09:59:09.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f0426657640 1 -- 172.21.15.71:0/4241545644 shutdown_connections 2023-08-25T09:59:09.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f0426657640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04201509d0 0x7f0420150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f0426657640 1 --2- 172.21.15.71:0/4241545644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04200c3a40 0x7f04200c1710 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.258+0000 7f0426657640 1 -- 172.21.15.71:0/4241545644 >> 172.21.15.71:0/4241545644 conn(0x7f04200654f0 msgr2=0x7f0420065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:09.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.259+0000 7f0426657640 1 -- 172.21.15.71:0/4241545644 shutdown_connections 2023-08-25T09:59:09.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.259+0000 7f0426657640 1 -- 172.21.15.71:0/4241545644 wait complete. 2023-08-25T09:59:09.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.260+0000 7f0426657640 1 Processor -- start 2023-08-25T09:59:09.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.260+0000 7f0426657640 1 -- start start 2023-08-25T09:59:09.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.261+0000 7f0426657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04200c3660 0x7f04201ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:09.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.261+0000 7f0426657640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04200c3a40 0x7f04201efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:09.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.261+0000 7f0426657640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 0x7f0420149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:09.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.261+0000 7f0426657640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04201ef180 con 0x7f04201509d0 2023-08-25T09:59:09.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.261+0000 7f0426657640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0420152a80 con 0x7f04200c3a40 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.261+0000 7f0426657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0420152d60 con 0x7f04200c3660 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 0x7f0420149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 0x7f0420149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42956/0 (socket says 172.21.15.71:42956) 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 -- 172.21.15.71:0/3972575349 learned_addr learned my addr 172.21.15.71:0/3972575349 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f041ffff640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04200c3660 0x7f04201ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f041f7fe640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04200c3a40 0x7f04201efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:09.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 -- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04200c3660 msgr2=0x7f04201ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:09.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04200c3660 0x7f04201ef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 -- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04200c3a40 msgr2=0x7f04201efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:09.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04200c3a40 0x7f04201efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f0424bf4640 1 -- 172.21.15.71:0/3972575349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414009400 con 0x7f04201509d0 2023-08-25T09:59:09.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.262+0000 7f041f7fe640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04200c3a40 0x7f04201efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:09.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.263+0000 7f0424bf4640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 0x7f0420149750 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f041800cef0 tx=0x7f041800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.263+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041800d840 con 0x7f04201509d0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.263+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041800d9c0 con 0x7f04201509d0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.263+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041800dd70 con 0x7f04201509d0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.263+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0420153040 con 0x7f04201509d0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.263+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04201ef3d0 con 0x7f04201509d0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.264+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04200c29c0 con 0x7f04201509d0 2023-08-25T09:59:09.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.265+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f041801e3b0 con 0x7f04201509d0 2023-08-25T09:59:09.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.266+0000 7f041cff9640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f040c06a080 0x7f040c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:09.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.267+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7215+0+0 (secure 0 0 0) 0x7f04180894a0 con 0x7f04201509d0 2023-08-25T09:59:09.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.267+0000 7f041ffff640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f040c06a080 0x7f040c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:09.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.267+0000 7f041ffff640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f040c06a080 0x7f040c06c540 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f04100011e0 tx=0x7f0410009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:09.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.271+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f04180547c0 con 0x7f04201509d0 2023-08-25T09:59:09.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.498+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04201ef5f0 con 0x7f04201509d0 2023-08-25T09:59:09.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.500+0000 7f041cff9640 1 -- 172.21.15.71:0/3972575349 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f0418057de0 con 0x7f04201509d0 2023-08-25T09:59:09.502 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:09.503 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:09.503 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:09.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f040c06a080 msgr2=0x7f040c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:09.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f040c06a080 0x7f040c06c540 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f04100011e0 tx=0x7f0410009040 comp rx=0 tx=0).stop 2023-08-25T09:59:09.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 msgr2=0x7f0420149750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:09.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 0x7f0420149750 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f041800cef0 tx=0x7f041800a710 comp rx=0 tx=0).stop 2023-08-25T09:59:09.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 shutdown_connections 2023-08-25T09:59:09.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04200c3a40 0x7f04201efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f040c06a080 0x7f040c06c540 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f04201509d0 0x7f0420149750 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 --2- 172.21.15.71:0/3972575349 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f04200c3660 0x7f04201ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:09.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 >> 172.21.15.71:0/3972575349 conn(0x7f04200654f0 msgr2=0x7f0420070630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:09.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 shutdown_connections 2023-08-25T09:59:09.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:09.503+0000 7f0426657640 1 -- 172.21.15.71:0/3972575349 wait complete. 2023-08-25T09:59:09.522 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T09:59:09.524 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T09:59:11.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:11.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.741+0000 7f48d32e0640 1 Processor -- start 2023-08-25T09:59:11.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.741+0000 7f48d32e0640 1 -- start start 2023-08-25T09:59:11.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d32e0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f48cc152db0 0x7f48cc0c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d32e0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f48cc0c60e0 0x7f48cc0c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d32e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc071be0 0x7f48cc158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d32e0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48cc158b90 con 0x7f48cc0c60e0 2023-08-25T09:59:11.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d32e0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48cc0c5260 con 0x7f48cc152db0 2023-08-25T09:59:11.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d32e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0c6580 con 0x7f48cc071be0 2023-08-25T09:59:11.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d1add640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f48cc0c60e0 0x7f48cc0c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d1add640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f48cc0c60e0 0x7f48cc0c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39082/0 (socket says 172.21.15.71:39082) 2023-08-25T09:59:11.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.743+0000 7f48d1add640 1 -- 172.21.15.71:0/1158111621 learned_addr learned my addr 172.21.15.71:0/1158111621 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:11.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d22de640 1 --2- 172.21.15.71:0/1158111621 >> v2:172.21.15.164:3300/0 conn(0x7f48cc152db0 0x7f48cc0c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d2adf640 1 --2- 172.21.15.71:0/1158111621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc071be0 0x7f48cc158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d1add640 1 -- 172.21.15.71:0/1158111621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc071be0 msgr2=0x7f48cc158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:11.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d1add640 1 --2- 172.21.15.71:0/1158111621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc071be0 0x7f48cc158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d1add640 1 -- 172.21.15.71:0/1158111621 >> v2:172.21.15.164:3300/0 conn(0x7f48cc152db0 msgr2=0x7f48cc0c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d1add640 1 --2- 172.21.15.71:0/1158111621 >> v2:172.21.15.164:3300/0 conn(0x7f48cc152db0 0x7f48cc0c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.744+0000 7f48d1add640 1 -- 172.21.15.71:0/1158111621 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc15cc30 con 0x7f48cc0c60e0 2023-08-25T09:59:11.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.745+0000 7f48d1add640 1 --2- 172.21.15.71:0/1158111621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc0c60e0 0x7f48cc0c6d00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f48cc06b4f0 tx=0x7f48bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c246a4d252abff server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:11.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.748+0000 7f48d12dc640 1 -- 172.21.15.71:0/1158111621 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc031870 con 0x7f48cc0c60e0 2023-08-25T09:59:11.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.748+0000 7f48d12dc640 1 -- 172.21.15.71:0/1158111621 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc0319d0 con 0x7f48cc0c60e0 2023-08-25T09:59:11.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.748+0000 7f48d12dc640 1 -- 172.21.15.71:0/1158111621 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48bc031d60 con 0x7f48cc0c60e0 2023-08-25T09:59:11.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.748+0000 7f48d32e0640 1 -- 172.21.15.71:0/1158111621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc0c60e0 msgr2=0x7f48cc0c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:11.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.748+0000 7f48d32e0640 1 --2- 172.21.15.71:0/1158111621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc0c60e0 0x7f48cc0c6d00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f48cc06b4f0 tx=0x7f48bc02b910 comp rx=0 tx=0).stop 2023-08-25T09:59:11.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 -- 172.21.15.71:0/1158111621 shutdown_connections 2023-08-25T09:59:11.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 --2- 172.21.15.71:0/1158111621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc071be0 0x7f48cc158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 --2- 172.21.15.71:0/1158111621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc0c60e0 0x7f48cc0c6d00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 --2- 172.21.15.71:0/1158111621 >> v2:172.21.15.164:3300/0 conn(0x7f48cc152db0 0x7f48cc0c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 -- 172.21.15.71:0/1158111621 >> 172.21.15.71:0/1158111621 conn(0x7f48cc06d0d0 msgr2=0x7f48cc0ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:11.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 -- 172.21.15.71:0/1158111621 shutdown_connections 2023-08-25T09:59:11.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.749+0000 7f48d32e0640 1 -- 172.21.15.71:0/1158111621 wait complete. 2023-08-25T09:59:11.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.750+0000 7f48d32e0640 1 Processor -- start 2023-08-25T09:59:11.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.751+0000 7f48d32e0640 1 -- start start 2023-08-25T09:59:11.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d32e0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 0x7f48cc0c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d32e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc0c60e0 0x7f48cc0c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d32e0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48cc152db0 0x7f48cc0c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d32e0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc14e310 con 0x7f48cc071be0 2023-08-25T09:59:11.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d32e0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc0705b0 con 0x7f48cc152db0 2023-08-25T09:59:11.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d32e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc06fd50 con 0x7f48cc0c60e0 2023-08-25T09:59:11.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 0x7f48cc0c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 0x7f48cc0c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39086/0 (socket says 172.21.15.71:39086) 2023-08-25T09:59:11.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 -- 172.21.15.71:0/2602446668 learned_addr learned my addr 172.21.15.71:0/2602446668 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:11.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d1add640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc0c60e0 0x7f48cc0c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d2adf640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48cc152db0 0x7f48cc0c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 -- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc0c60e0 msgr2=0x7f48cc0c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:11.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc0c60e0 0x7f48cc0c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 -- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48cc152db0 msgr2=0x7f48cc0c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:11.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48cc152db0 0x7f48cc0c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:11.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d22de640 1 -- 172.21.15.71:0/2602446668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0094d0 con 0x7f48cc071be0 2023-08-25T09:59:11.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d2adf640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48cc152db0 0x7f48cc0c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:11.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.752+0000 7f48d1add640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc0c60e0 0x7f48cc0c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:11.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.753+0000 7f48d22de640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 0x7f48cc0c8930 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f48c800d800 tx=0x7f48c800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:11.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.754+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c800b660 con 0x7f48cc071be0 2023-08-25T09:59:11.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.754+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c800b7c0 con 0x7f48cc071be0 2023-08-25T09:59:11.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.754+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c800bb50 con 0x7f48cc071be0 2023-08-25T09:59:11.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.754+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48cc06fff0 con 0x7f48cc071be0 2023-08-25T09:59:11.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.755+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48cc0c9370 con 0x7f48cc071be0 2023-08-25T09:59:11.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.756+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48cc0c91d0 con 0x7f48cc071be0 2023-08-25T09:59:11.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.756+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f48c800bcb0 con 0x7f48cc071be0 2023-08-25T09:59:11.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.758+0000 7f48c2ffd640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f48b006a080 0x7f48b006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:11.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.758+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7391+0+0 (secure 0 0 0) 0x7f48c8089070 con 0x7f48cc071be0 2023-08-25T09:59:11.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.758+0000 7f48d1add640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f48b006a080 0x7f48b006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:11.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.759+0000 7f48d1add640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f48b006a080 0x7f48b006c540 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f48bc036000 tx=0x7f48bc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:11.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.763+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f48c80bb990 con 0x7f48cc071be0 2023-08-25T09:59:11.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:11.996+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48cc04e420 con 0x7f48cc071be0 2023-08-25T09:59:12.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.004+0000 7f48c2ffd640 1 -- 172.21.15.71:0/2602446668 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f48cc0c91d0 con 0x7f48cc071be0 2023-08-25T09:59:12.007 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:12.007 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:12.008 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:12.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.008+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f48b006a080 msgr2=0x7f48b006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:12.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.008+0000 7f48d32e0640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f48b006a080 0x7f48b006c540 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f48bc036000 tx=0x7f48bc03c040 comp rx=0 tx=0).stop 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.008+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 msgr2=0x7f48cc0c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.008+0000 7f48d32e0640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 0x7f48cc0c8930 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f48c800d800 tx=0x7f48c800dcc0 comp rx=0 tx=0).stop 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 shutdown_connections 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48cc152db0 0x7f48cc0c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f48b006a080 0x7f48b006c540 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48cc0c60e0 0x7f48cc0c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 --2- 172.21.15.71:0/2602446668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f48cc071be0 0x7f48cc0c8930 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:12.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 >> 172.21.15.71:0/2602446668 conn(0x7f48cc06d0d0 msgr2=0x7f48cc158ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:12.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 shutdown_connections 2023-08-25T09:59:12.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:12.009+0000 7f48d32e0640 1 -- 172.21.15.71:0/2602446668 wait complete. 2023-08-25T09:59:12.034 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:14.036 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:14.066 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-08-25T09:59:14.067 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-08-25T09:59:14.177 INFO:tasks.workunit.client.0.smithi071.stderr:+ set -e 2023-08-25T09:59:14.178 INFO:tasks.workunit.client.0.smithi071.stderr:++ seq 1 20 2023-08-25T09:59:14.179 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:14.180 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:14.186 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:14.186 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:14.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.249+0000 7fa6b2376640 1 Processor -- start 2023-08-25T09:59:14.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.249+0000 7fa6b2376640 1 -- start start 2023-08-25T09:59:14.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6b2376640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa6ac0c3660 0x7fa6ac0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6b2376640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3a40 0x7fa6ac06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6b2376640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0c1dc0 con 0x7fa6ac1509d0 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6b2376640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ac157a80 con 0x7fa6ac0c3660 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6b2376640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac06cf70 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6abfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa6ac0c3660 0x7fa6ac0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6abfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa6ac0c3660 0x7fa6ac0c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39096/0 (socket says 172.21.15.71:39096) 2023-08-25T09:59:14.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6abfff640 1 -- 172.21.15.71:0/950381321 learned_addr learned my addr 172.21.15.71:0/950381321 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:14.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6abfff640 1 -- 172.21.15.71:0/950381321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3a40 msgr2=0x7fa6ac06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6ab7fe640 1 --2- 172.21.15.71:0/950381321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3a40 0x7fa6ac06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6abfff640 1 --2- 172.21.15.71:0/950381321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3a40 0x7fa6ac06ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.251+0000 7fa6abfff640 1 -- 172.21.15.71:0/950381321 >> v1:172.21.15.164:6789/0 conn(0x7fa6ac1509d0 legacy=0x7fa6ac150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.252+0000 7fa6abfff640 1 -- 172.21.15.71:0/950381321 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac156e00 con 0x7fa6ac0c3660 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.253+0000 7fa6abfff640 1 --2- 172.21.15.71:0/950381321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3660 0x7fa6ac0c1710 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009870 tx=0x7fa69c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0915b14abce8e7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.253+0000 7fa6aaffd640 1 -- 172.21.15.71:0/950381321 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c030840 con 0x7fa6ac0c3660 2023-08-25T09:59:14.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.253+0000 7fa6aaffd640 1 -- 172.21.15.71:0/950381321 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c0309a0 con 0x7fa6ac0c3660 2023-08-25T09:59:14.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.253+0000 7fa6aaffd640 1 -- 172.21.15.71:0/950381321 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c030d30 con 0x7fa6ac0c3660 2023-08-25T09:59:14.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 -- 172.21.15.71:0/950381321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3660 msgr2=0x7fa6ac0c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 --2- 172.21.15.71:0/950381321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3660 0x7fa6ac0c1710 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009870 tx=0x7fa69c02b980 comp rx=0 tx=0).stop 2023-08-25T09:59:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 -- 172.21.15.71:0/950381321 shutdown_connections 2023-08-25T09:59:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 --2- 172.21.15.71:0/950381321 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3a40 0x7fa6ac06ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 --2- 172.21.15.71:0/950381321 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3660 0x7fa6ac0c1710 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 -- 172.21.15.71:0/950381321 >> 172.21.15.71:0/950381321 conn(0x7fa6ac0654f0 msgr2=0x7fa6ac065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 -- 172.21.15.71:0/950381321 shutdown_connections 2023-08-25T09:59:14.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.254+0000 7fa6b2376640 1 -- 172.21.15.71:0/950381321 wait complete. 2023-08-25T09:59:14.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.255+0000 7fa6b2376640 1 Processor -- start 2023-08-25T09:59:14.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.256+0000 7fa6b2376640 1 -- start start 2023-08-25T09:59:14.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b2376640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3660 0x7fa6ac155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:14.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b2376640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 0x7fa6ac159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:14.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b2376640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6ac1509d0 0x7fa6ac159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:14.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b2376640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac158a80 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b2376640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac06ff80 con 0x7fa6ac1509d0 2023-08-25T09:59:14.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b2376640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1f29d0 con 0x7fa6ac0c3660 2023-08-25T09:59:14.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 0x7fa6ac159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:14.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 0x7fa6ac159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39108/0 (socket says 172.21.15.71:39108) 2023-08-25T09:59:14.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 -- 172.21.15.71:0/4140920704 learned_addr learned my addr 172.21.15.71:0/4140920704 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:14.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b0913640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6ac1509d0 0x7fa6ac159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:14.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 -- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3660 msgr2=0x7fa6ac155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:14.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6abfff640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3660 0x7fa6ac155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:14.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3660 0x7fa6ac155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 -- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6ac1509d0 msgr2=0x7fa6ac159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6ac1509d0 0x7fa6ac159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6ab7fe640 1 -- 172.21.15.71:0/4140920704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0094d0 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.257+0000 7fa6b0913640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6ac1509d0 0x7fa6ac159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.258+0000 7fa6ab7fe640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 0x7fa6ac159840 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c7f0 tx=0x7fa6a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.258+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0004220 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.258+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a0004380 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.258+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac070220 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.258+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0007840 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.258+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac159040 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.259+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac0c29c0 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.260+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa6a00079a0 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.261+0000 7fa6a8ff9640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa69806a080 0x7fa69806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:14.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.261+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7359+0+0 (secure 0 0 0) 0x7fa6a00887f0 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.262+0000 7fa6abfff640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa69806a080 0x7fa69806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:14.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.262+0000 7fa6abfff640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa69806a080 0x7fa69806c540 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa69c036000 tx=0x7fa69c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:14.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.266+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa6a0053930 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.497+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6ac069160 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.497+0000 7fa6a8ff9640 1 -- 172.21.15.71:0/4140920704 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fa6a0056f50 con 0x7fa6ac0c3a40 2023-08-25T09:59:14.499 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:14.500 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:14.500 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:14.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa69806a080 msgr2=0x7fa69806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:14.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa69806a080 0x7fa69806c540 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa69c036000 tx=0x7fa69c03b040 comp rx=0 tx=0).stop 2023-08-25T09:59:14.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 msgr2=0x7fa6ac159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:14.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 0x7fa6ac159840 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c7f0 tx=0x7fa6a000ccb0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 shutdown_connections 2023-08-25T09:59:14.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6ac1509d0 0x7fa6ac159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa69806a080 0x7fa69806c540 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6ac0c3a40 0x7fa6ac159840 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 --2- 172.21.15.71:0/4140920704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6ac0c3660 0x7fa6ac155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:14.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 >> 172.21.15.71:0/4140920704 conn(0x7fa6ac0654f0 msgr2=0x7fa6ac0c86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:14.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 shutdown_connections 2023-08-25T09:59:14.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:14.500+0000 7fa6b2376640 1 -- 172.21.15.71:0/4140920704 wait complete. 2023-08-25T09:59:14.521 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:14.548 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:14.636 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:14.643 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:14.985 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:15.012 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:15.013 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:15.013 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:15.013 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:15.013 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:15.013 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:15.013 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:15.014 INFO:tasks.workunit.client.0.smithi071.stdout: 106331 ? 00:00:00 ps 2023-08-25T09:59:15.026 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:15.126 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:15.138 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:15.152 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:15.153 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:15.162 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:15.162 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:15.898 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:15.925 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:15.932 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:16.078 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:16.100 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:16.101 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:16.101 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:16.101 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:16.101 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:16.101 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:16.102 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:16.102 INFO:tasks.workunit.client.0.smithi071.stdout: 106341 ? 00:00:00 ps 2023-08-25T09:59:16.109 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:16.156 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:16.169 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:16.191 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:16.192 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:16.201 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:16.201 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:16.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:16.591 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:16.615 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:16.620 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:16.736 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:16.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.753+0000 7fbcdc912640 1 Processor -- start 2023-08-25T09:59:16.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.753+0000 7fbcdc912640 1 -- start start 2023-08-25T09:59:16.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbcdc912640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd81594d0 0x7fbcd80c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:16.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbcdc912640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbcd806cc70 0x7fbcd814f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:16.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbcdc912640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd806a960 con 0x7fbcd80c3940 2023-08-25T09:59:16.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbcdc912640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd806ac20 con 0x7fbcd806cc70 2023-08-25T09:59:16.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbcdc912640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd806af00 con 0x7fbcd81594d0 2023-08-25T09:59:16.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbccffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbcd806cc70 0x7fbcd814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbccffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbcd806cc70 0x7fbcd814f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39110/0 (socket says 172.21.15.71:39110) 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbccffff640 1 -- 172.21.15.71:0/1451726948 learned_addr learned my addr 172.21.15.71:0/1451726948 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.755+0000 7fbcd659c640 1 --2- 172.21.15.71:0/1451726948 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd81594d0 0x7fbcd80c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.756+0000 7fbccffff640 1 -- 172.21.15.71:0/1451726948 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd81594d0 msgr2=0x7fbcd80c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.756+0000 7fbccffff640 1 --2- 172.21.15.71:0/1451726948 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd81594d0 0x7fbcd80c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.756+0000 7fbccffff640 1 -- 172.21.15.71:0/1451726948 >> v1:172.21.15.164:6789/0 conn(0x7fbcd80c3940 legacy=0x7fbcd81598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:16.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.756+0000 7fbccffff640 1 -- 172.21.15.71:0/1451726948 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd80c2e60 con 0x7fbcd806cc70 2023-08-25T09:59:16.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.757+0000 7fbccffff640 1 --2- 172.21.15.71:0/1451726948 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd814f850 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000bb10 tx=0x7fbcc002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89be0e5a1277f0a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:16.759 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:16.759 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:16.760 INFO:tasks.workunit.client.0.smithi071.stdout: 106371 ? 00:00:00 ps 2023-08-25T09:59:16.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcd5d9b640 1 -- 172.21.15.71:0/1451726948 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0036440 con 0x7fbcd806cc70 2023-08-25T09:59:16.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcd5d9b640 1 -- 172.21.15.71:0/1451726948 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc00365a0 con 0x7fbcd806cc70 2023-08-25T09:59:16.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcd5d9b640 1 -- 172.21.15.71:0/1451726948 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0036930 con 0x7fbcd806cc70 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 -- 172.21.15.71:0/1451726948 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 msgr2=0x7fbcd814f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 --2- 172.21.15.71:0/1451726948 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd814f850 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000bb10 tx=0x7fbcc002e150 comp rx=0 tx=0).stop 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 -- 172.21.15.71:0/1451726948 shutdown_connections 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 --2- 172.21.15.71:0/1451726948 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd814f850 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 --2- 172.21.15.71:0/1451726948 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd81594d0 0x7fbcd80c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 -- 172.21.15.71:0/1451726948 >> 172.21.15.71:0/1451726948 conn(0x7fbcd80654f0 msgr2=0x7fbcd8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.758+0000 7fbcdc912640 1 -- 172.21.15.71:0/1451726948 shutdown_connections 2023-08-25T09:59:16.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.759+0000 7fbcdc912640 1 -- 172.21.15.71:0/1451726948 wait complete. 2023-08-25T09:59:16.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.760+0000 7fbcdc912640 1 Processor -- start 2023-08-25T09:59:16.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.761+0000 7fbcdc912640 1 -- start start 2023-08-25T09:59:16.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.761+0000 7fbcdc912640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd80c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:16.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcdc912640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd80c3940 0x7fbcd81eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:16.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcdc912640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcd81594d0 0x7fbcd80c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcdc912640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81ee5d0 con 0x7fbcd806cc70 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcdc912640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd814ef40 con 0x7fbcd81594d0 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcdc912640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81edfa0 con 0x7fbcd80c3940 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd80c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd80c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39118/0 (socket says 172.21.15.71:39118) 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 -- 172.21.15.71:0/3708057928 learned_addr learned my addr 172.21.15.71:0/3708057928 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:16.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbccffff640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd80c3940 0x7fbcd81eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 -- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd80c3940 msgr2=0x7fbcd81eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd80c3940 0x7fbcd81eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 -- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcd81594d0 msgr2=0x7fbcd80c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd6d9d640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcd81594d0 0x7fbcd80c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcd81594d0 0x7fbcd80c8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.762+0000 7fbcd659c640 1 -- 172.21.15.71:0/3708057928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc000b770 con 0x7fbcd806cc70 2023-08-25T09:59:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.763+0000 7fbcd659c640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd80c80f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800d800 tx=0x7fbcc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:16.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.763+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800b6c0 con 0x7fbcd806cc70 2023-08-25T09:59:16.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.763+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc800b820 con 0x7fbcd806cc70 2023-08-25T09:59:16.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.763+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800bbb0 con 0x7fbcd806cc70 2023-08-25T09:59:16.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.763+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8144b60 con 0x7fbcd806cc70 2023-08-25T09:59:16.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.763+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd81f1fc0 con 0x7fbcd806cc70 2023-08-25T09:59:16.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.764+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd8069160 con 0x7fbcd806cc70 2023-08-25T09:59:16.768 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:16.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.765+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbcc800bd10 con 0x7fbcd806cc70 2023-08-25T09:59:16.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.766+0000 7fbcceffd640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbcb806a050 0x7fbcb806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:16.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.766+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7327+0+0 (secure 0 0 0) 0x7fbcc8089140 con 0x7fbcd806cc70 2023-08-25T09:59:16.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.767+0000 7fbccffff640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbcb806a050 0x7fbcb806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:16.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.767+0000 7fbccffff640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbcb806a050 0x7fbcb806c510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0038000 tx=0x7fbcc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:16.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:16.771+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbcc8054210 con 0x7fbcd806cc70 2023-08-25T09:59:16.809 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:16.822 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:16.841 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:16.841 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:16.850 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:16.850 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:16.884 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:16.908 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:16.913 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:16.992 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:17.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.007+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcd80c33f0 con 0x7fbcd806cc70 2023-08-25T09:59:17.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.008+0000 7fbcceffd640 1 -- 172.21.15.71:0/3708057928 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fbcc8057830 con 0x7fbcd806cc70 2023-08-25T09:59:17.010 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:17.011 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:17.012 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:17.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbcb806a050 msgr2=0x7fbcb806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:17.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbcb806a050 0x7fbcb806c510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0038000 tx=0x7fbcc0008040 comp rx=0 tx=0).stop 2023-08-25T09:59:17.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 msgr2=0x7fbcd80c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:17.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd80c80f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800d800 tx=0x7fbcc800dcc0 comp rx=0 tx=0).stop 2023-08-25T09:59:17.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 shutdown_connections 2023-08-25T09:59:17.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbcd81594d0 0x7fbcd80c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:17.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbcb806a050 0x7fbcb806c510 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:17.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbcd80c3940 0x7fbcd81eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:17.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 --2- 172.21.15.71:0/3708057928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbcd806cc70 0x7fbcd80c80f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:17.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 >> 172.21.15.71:0/3708057928 conn(0x7fbcd80654f0 msgr2=0x7fbcd814d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:17.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 shutdown_connections 2023-08-25T09:59:17.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:17.011+0000 7fbcdc912640 1 -- 172.21.15.71:0/3708057928 wait complete. 2023-08-25T09:59:17.016 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:17.016 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:17.017 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:17.017 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:17.017 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:17.017 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:17.017 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:17.017 INFO:tasks.workunit.client.0.smithi071.stdout: 106401 ? 00:00:00 ps 2023-08-25T09:59:17.022 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:17.032 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:17.065 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:17.078 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:17.099 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:17.099 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:17.110 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:17.110 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:17.146 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:17.172 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:17.179 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:17.285 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:17.304 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:17.304 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:17.304 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:17.304 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:17.305 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:17.305 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:17.305 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:17.305 INFO:tasks.workunit.client.0.smithi071.stdout: 106413 ? 00:00:00 ps 2023-08-25T09:59:17.315 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:17.369 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:17.385 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:17.408 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:17.408 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:17.419 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:17.420 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:17.457 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:17.487 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:17.494 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:17.560 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:17.583 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:17.584 INFO:tasks.workunit.client.0.smithi071.stdout: 106423 ? 00:00:00 ps 2023-08-25T09:59:17.597 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:17.653 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:17.669 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:17.693 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:17.693 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:17.703 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:17.704 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:17.742 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:17.769 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:17.776 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:17.867 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:17.890 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:17.890 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:17.890 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:17.891 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:17.891 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:17.891 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:17.891 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:17.891 INFO:tasks.workunit.client.0.smithi071.stdout: 106433 ? 00:00:00 ps 2023-08-25T09:59:18.400 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:18.456 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:18.472 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:18.490 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:18.491 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:18.498 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:18.499 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:18.537 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:18.564 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:18.570 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:18.635 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:18.661 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:18.661 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:18.661 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:18.662 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:18.662 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:18.662 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:18.662 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:18.662 INFO:tasks.workunit.client.0.smithi071.stdout: 106443 ? 00:00:00 ps 2023-08-25T09:59:18.672 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:18.915 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:18.929 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:18.954 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:18.955 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:18.966 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:18.966 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:19.012 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:19.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:19.043 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:19.050 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:19.127 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:19.149 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:19.149 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:19.149 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:19.149 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:19.149 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:19.150 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:19.150 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:19.150 INFO:tasks.workunit.client.0.smithi071.stdout: 106473 ? 00:00:00 ps 2023-08-25T09:59:19.157 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:19.209 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:19.222 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:19.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.248+0000 7f5d6276b640 1 Processor -- start 2023-08-25T09:59:19.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.248+0000 7f5d6276b640 1 -- start start 2023-08-25T09:59:19.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.250+0000 7f5d6276b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c159fc0 0x7f5d5c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.250+0000 7f5d6276b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c06a960 con 0x7f5d5c1594d0 2023-08-25T09:59:19.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.250+0000 7f5d6276b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c06ac20 con 0x7f5d5c06cc70 2023-08-25T09:59:19.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.250+0000 7f5d6276b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c06af00 con 0x7f5d5c159fc0 2023-08-25T09:59:19.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.250+0000 7f5d5bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5d5c1594d0 0x7f5d5c14f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56684/0 (socket says 172.21.15.71:56684) 2023-08-25T09:59:19.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.250+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2928325093 learned_addr learned my addr 172.21.15.71:0/2928325093 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:19.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269261001 0 0) 0x7f5d5c06a960 con 0x7f5d5c1594d0 2023-08-25T09:59:19.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d60d08640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c159fc0 0x7f5d5c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d44005730 con 0x7f5d5c1594d0 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4009758158 0 0) 0x7f5d44005730 con 0x7f5d5c1594d0 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c159fc0 msgr2=0x7f5d5c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c159fc0 0x7f5d5c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 >> v1:172.21.15.164:6789/0 conn(0x7f5d5c06cc70 legacy=0x7f5d5c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0c2e60 con 0x7f5d5c1594d0 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f5d4c0019f0 con 0x7f5d5c1594d0 2023-08-25T09:59:19.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 >> v1:172.21.15.71:6789/0 conn(0x7f5d5c1594d0 legacy=0x7f5d5c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d44006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d440069d0 0x7f5d44006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d440073e0 0x7f5d4400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c06af00 con 0x7f5d44005c20 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4400e1e0 con 0x7f5d440069d0 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4400e3a0 con 0x7f5d440073e0 2023-08-25T09:59:19.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.251+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d44006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d5b7fe640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d440069d0 0x7f5d44006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d60d08640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d440073e0 0x7f5d4400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d440073e0 msgr2=0x7f5d4400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:19.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d440073e0 0x7f5d4400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d440069d0 msgr2=0x7f5d44006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:19.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d440069d0 0x7f5d44006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.252+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2928325093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d44005730 con 0x7f5d44005c20 2023-08-25T09:59:19.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.253+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d44006040 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00a830 tx=0x7f5d44007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a508087ed31206 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:19.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.253+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0072d0 con 0x7f5d44005c20 2023-08-25T09:59:19.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.253+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c02f070 con 0x7f5d44005c20 2023-08-25T09:59:19.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.253+0000 7f5d5affd640 1 -- 172.21.15.71:0/2928325093 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c007650 con 0x7f5d44005c20 2023-08-25T09:59:19.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d6276b640 1 -- 172.21.15.71:0/2928325093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 msgr2=0x7f5d44006040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2928325093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d44006040 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00a830 tx=0x7f5d44007a00 comp rx=0 tx=0).stop 2023-08-25T09:59:19.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d60d08640 1 -- 172.21.15.71:0/2928325093 reap_dead start 2023-08-25T09:59:19.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d6276b640 1 -- 172.21.15.71:0/2928325093 shutdown_connections 2023-08-25T09:59:19.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d6276b640 1 -- 172.21.15.71:0/2928325093 >> 172.21.15.71:0/2928325093 conn(0x7f5d5c0654f0 msgr2=0x7f5d5c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:19.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d6276b640 1 -- 172.21.15.71:0/2928325093 shutdown_connections 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.254+0000 7f5d6276b640 1 -- 172.21.15.71:0/2928325093 wait complete. 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.256+0000 7f5d6276b640 1 Processor -- start 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.256+0000 7f5d6276b640 1 -- start start 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d6276b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d5c147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d6276b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d5c0c9aa0 0x7f5d5c1de8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d6276b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c0c9f00 0x7f5d5c1dede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d6276b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1e14c0 con 0x7f5d44005c20 2023-08-25T09:59:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d6276b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c157e80 con 0x7f5d5c0c9aa0 2023-08-25T09:59:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d6276b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c14ce10 con 0x7f5d5c0c9f00 2023-08-25T09:59:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d5c147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d5bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d5c147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39138/0 (socket says 172.21.15.71:39138) 2023-08-25T09:59:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2263417285 learned_addr learned my addr 172.21.15.71:0/2263417285 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d5b7fe640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d5c0c9aa0 0x7f5d5c1de8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.257+0000 7f5d60d08640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c0c9f00 0x7f5d5c1dede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c0c9f00 msgr2=0x7f5d5c1dede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c0c9f00 0x7f5d5c1dede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d5c0c9aa0 msgr2=0x7f5d5c1de8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d5c0c9aa0 0x7f5d5c1de8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d5bfff640 1 -- 172.21.15.71:0/2263417285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c1578e0 con 0x7f5d44005c20 2023-08-25T09:59:19.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d5bfff640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d5c147350 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00a830 tx=0x7f5d4c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:19.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c004d60 con 0x7f5d44005c20 2023-08-25T09:59:19.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c0070d0 con 0x7f5d44005c20 2023-08-25T09:59:19.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c007460 con 0x7f5d44005c20 2023-08-25T09:59:19.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c157ba0 con 0x7f5d44005c20 2023-08-25T09:59:19.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.258+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c1e1150 con 0x7f5d44005c20 2023-08-25T09:59:19.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.259+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c06cae0 con 0x7f5d44005c20 2023-08-25T09:59:19.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.260+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5d4c031030 con 0x7f5d44005c20 2023-08-25T09:59:19.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.261+0000 7f5d58ff9640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5d4806a050 0x7f5d4806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:19.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.261+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7327+0+0 (secure 0 0 0) 0x7f5d4c0aa240 con 0x7f5d44005c20 2023-08-25T09:59:19.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.262+0000 7f5d5b7fe640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5d4806a050 0x7f5d4806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:19.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.262+0000 7f5d5b7fe640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5d4806a050 0x7f5d4806c510 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c1585b0 tx=0x7f5d50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:19.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.265+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5d4c075310 con 0x7f5d44005c20 2023-08-25T09:59:19.374 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:19.374 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:19.383 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:19.384 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:19.419 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:19.447 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:19.453 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:19.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.500+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d5c1e12e0 con 0x7f5d44005c20 2023-08-25T09:59:19.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.501+0000 7f5d58ff9640 1 -- 172.21.15.71:0/2263417285 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f5d4c045020 con 0x7f5d44005c20 2023-08-25T09:59:19.503 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:19.503 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:19.503 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:19.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5d4806a050 msgr2=0x7f5d4806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5d4806a050 0x7f5d4806c510 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c1585b0 tx=0x7f5d50009040 comp rx=0 tx=0).stop 2023-08-25T09:59:19.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 msgr2=0x7f5d5c147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d5c147350 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00a830 tx=0x7f5d4c001d80 comp rx=0 tx=0).stop 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 shutdown_connections 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d5c0c9aa0 0x7f5d5c1de8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5d4806a050 0x7f5d4806c510 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d5c0c9f00 0x7f5d5c1dede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.503+0000 7f5d6276b640 1 --2- 172.21.15.71:0/2263417285 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d44005c20 0x7f5d5c147350 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:19.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.504+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 >> 172.21.15.71:0/2263417285 conn(0x7f5d5c0654f0 msgr2=0x7f5d5c0c2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:19.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.504+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 shutdown_connections 2023-08-25T09:59:19.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:19.504+0000 7f5d6276b640 1 -- 172.21.15.71:0/2263417285 wait complete. 2023-08-25T09:59:19.523 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:19.536 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:19.557 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:19.557 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:19.558 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:19.558 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:19.558 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:19.559 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:19.559 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:19.559 INFO:tasks.workunit.client.0.smithi071.stdout: 106506 ? 00:00:00 ps 2023-08-25T09:59:19.566 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:19.616 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:19.636 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:19.656 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:19.656 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:19.665 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:19.666 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:19.704 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:19.731 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:19.738 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:19.864 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:19.886 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:19.886 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:19.886 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:19.887 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:19.887 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:19.887 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:19.887 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:19.888 INFO:tasks.workunit.client.0.smithi071.stdout: 106516 ? 00:00:00 ps 2023-08-25T09:59:19.895 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:19.948 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:19.962 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:19.984 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:19.994 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:19.995 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:20.034 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:20.062 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:20.069 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:20.145 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:20.168 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:20.169 INFO:tasks.workunit.client.0.smithi071.stdout: 106526 ? 00:00:00 ps 2023-08-25T09:59:20.179 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:20.231 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:20.245 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:20.267 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:20.267 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:20.277 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:20.314 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:20.341 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:20.348 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:20.411 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:20.434 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:20.434 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:20.434 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:20.434 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:20.435 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:20.435 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:20.435 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:20.435 INFO:tasks.workunit.client.0.smithi071.stdout: 106536 ? 00:00:00 ps 2023-08-25T09:59:20.447 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:20.507 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:20.535 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:20.767 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:20.768 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:20.779 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:20.780 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:20.821 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:20.850 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:20.857 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:20.946 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:20.969 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:20.969 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:20.969 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:20.970 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:20.970 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:20.970 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:20.970 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:20.971 INFO:tasks.workunit.client.0.smithi071.stdout: 106546 ? 00:00:00 ps 2023-08-25T09:59:20.979 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:21.073 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:21.444 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:21.454 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:21.454 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:21.462 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:21.463 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:21.497 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:21.523 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:21.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:21.533 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:21.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.751+0000 7f0eae1c2640 1 Processor -- start 2023-08-25T09:59:21.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.751+0000 7f0eae1c2640 1 -- start start 2023-08-25T09:59:21.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.753+0000 7f0eae1c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ea80c3660 0x7f0ea80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:21.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.753+0000 7f0eae1c2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ea81509d0 0x7f0ea8150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:21.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.753+0000 7f0eae1c2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea80c1dc0 con 0x7f0ea80c3a40 2023-08-25T09:59:21.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0eae1c2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea8157a80 con 0x7f0ea81509d0 2023-08-25T09:59:21.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0eae1c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea8151320 con 0x7f0ea80c3660 2023-08-25T09:59:21.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ea81509d0 0x7f0ea8150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:21.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ea81509d0 0x7f0ea8150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46038/0 (socket says 172.21.15.71:46038) 2023-08-25T09:59:21.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 -- 172.21.15.71:0/343421982 learned_addr learned my addr 172.21.15.71:0/343421982 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:21.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea77fe640 1 --2- 172.21.15.71:0/343421982 >> v2:172.21.15.164:3300/0 conn(0x7f0ea80c3660 0x7f0ea80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:21.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 -- 172.21.15.71:0/343421982 >> v2:172.21.15.164:3300/0 conn(0x7f0ea80c3660 msgr2=0x7f0ea80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:21.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 --2- 172.21.15.71:0/343421982 >> v2:172.21.15.164:3300/0 conn(0x7f0ea80c3660 0x7f0ea80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:21.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 -- 172.21.15.71:0/343421982 >> v1:172.21.15.71:6789/0 conn(0x7f0ea80c3a40 legacy=0x7f0ea806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:21.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.754+0000 7f0ea7fff640 1 -- 172.21.15.71:0/343421982 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea8156e00 con 0x7f0ea81509d0 2023-08-25T09:59:21.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.756+0000 7f0ea7fff640 1 --2- 172.21.15.71:0/343421982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea81509d0 0x7f0ea8150de0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8066440 tx=0x7f0e9002ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a608ce3e802b8cc server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:21.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.756+0000 7f0ea67fc640 1 -- 172.21.15.71:0/343421982 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9002f870 con 0x7f0ea81509d0 2023-08-25T09:59:21.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.756+0000 7f0ea67fc640 1 -- 172.21.15.71:0/343421982 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9002f9d0 con 0x7f0ea81509d0 2023-08-25T09:59:21.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.756+0000 7f0ea67fc640 1 -- 172.21.15.71:0/343421982 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9002fd60 con 0x7f0ea81509d0 2023-08-25T09:59:21.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 -- 172.21.15.71:0/343421982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea81509d0 msgr2=0x7f0ea8150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/343421982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea81509d0 0x7f0ea8150de0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8066440 tx=0x7f0e9002ba50 comp rx=0 tx=0).stop 2023-08-25T09:59:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 -- 172.21.15.71:0/343421982 shutdown_connections 2023-08-25T09:59:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/343421982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea81509d0 0x7f0ea8150de0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/343421982 >> v2:172.21.15.164:3300/0 conn(0x7f0ea80c3660 0x7f0ea80c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 -- 172.21.15.71:0/343421982 >> 172.21.15.71:0/343421982 conn(0x7f0ea80654f0 msgr2=0x7f0ea8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:21.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 -- 172.21.15.71:0/343421982 shutdown_connections 2023-08-25T09:59:21.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.757+0000 7f0eae1c2640 1 -- 172.21.15.71:0/343421982 wait complete. 2023-08-25T09:59:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.759+0000 7f0eae1c2640 1 Processor -- start 2023-08-25T09:59:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.760+0000 7f0eae1c2640 1 -- start start 2023-08-25T09:59:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0eae1c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 0x7f0ea814e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0eae1c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ea80c3a40 0x7f0ea806f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0eae1c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ea81509d0 0x7f0ea80c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:21.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0eae1c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea806eed0 con 0x7f0ea80c3660 2023-08-25T09:59:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0eae1c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea814e900 con 0x7f0ea81509d0 2023-08-25T09:59:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0eae1c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea814e0b0 con 0x7f0ea80c3a40 2023-08-25T09:59:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0ea77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 0x7f0ea814e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0ea77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 0x7f0ea814e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46052/0 (socket says 172.21.15.71:46052) 2023-08-25T09:59:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0ea77fe640 1 -- 172.21.15.71:0/1197262942 learned_addr learned my addr 172.21.15.71:0/1197262942 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0ea7fff640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ea81509d0 0x7f0ea80c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:21.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.761+0000 7f0ea6ffd640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ea80c3a40 0x7f0ea806f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:21.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.762+0000 7f0ea77fe640 1 -- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ea80c3a40 msgr2=0x7f0ea806f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.762+0000 7f0ea77fe640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ea80c3a40 0x7f0ea806f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:21.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.762+0000 7f0ea77fe640 1 -- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ea81509d0 msgr2=0x7f0ea80c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:21.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.762+0000 7f0ea77fe640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ea81509d0 0x7f0ea80c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:21.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.762+0000 7f0ea77fe640 1 -- 172.21.15.71:0/1197262942 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e900094d0 con 0x7f0ea80c3660 2023-08-25T09:59:21.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.762+0000 7f0ea77fe640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 0x7f0ea814e3c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000efc0 tx=0x7f0ea000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:21.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.764+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0014480 con 0x7f0ea80c3660 2023-08-25T09:59:21.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.764+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea000f040 con 0x7f0ea80c3660 2023-08-25T09:59:21.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.764+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0014820 con 0x7f0ea80c3660 2023-08-25T09:59:21.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.764+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea806f830 con 0x7f0ea80c3660 2023-08-25T09:59:21.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.764+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea814daf0 con 0x7f0ea80c3660 2023-08-25T09:59:21.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.766+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea80c29c0 con 0x7f0ea80c3660 2023-08-25T09:59:21.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.767+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0ea0014980 con 0x7f0ea80c3660 2023-08-25T09:59:21.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.768+0000 7f0e9ffff640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0e8806a0d0 0x7f0e8806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:21.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.768+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7327+0+0 (secure 0 0 0) 0x7f0ea00894e0 con 0x7f0ea80c3660 2023-08-25T09:59:21.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.768+0000 7f0ea6ffd640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0e8806a0d0 0x7f0e8806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:21.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.769+0000 7f0ea6ffd640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0e8806a0d0 0x7f0e8806c590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80c0490 tx=0x7f0e98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:21.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:21.773+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0ea00547f0 con 0x7f0ea80c3660 2023-08-25T09:59:22.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.000+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ea8067240 con 0x7f0ea80c3660 2023-08-25T09:59:22.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.001+0000 7f0e9ffff640 1 -- 172.21.15.71:0/1197262942 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f0ea8067240 con 0x7f0ea80c3660 2023-08-25T09:59:22.003 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:22.004 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:22.004 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:22.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0e8806a0d0 msgr2=0x7f0e8806c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:22.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0e8806a0d0 0x7f0e8806c590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ea80c0490 tx=0x7f0e98009040 comp rx=0 tx=0).stop 2023-08-25T09:59:22.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 msgr2=0x7f0ea814e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:22.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 0x7f0ea814e3c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000efc0 tx=0x7f0ea000c490 comp rx=0 tx=0).stop 2023-08-25T09:59:22.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 shutdown_connections 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ea81509d0 0x7f0ea80c5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0e8806a0d0 0x7f0e8806c590 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ea80c3a40 0x7f0ea806f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 --2- 172.21.15.71:0/1197262942 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ea80c3660 0x7f0ea814e3c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 >> 172.21.15.71:0/1197262942 conn(0x7f0ea80654f0 msgr2=0x7f0ea80c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 shutdown_connections 2023-08-25T09:59:22.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:22.004+0000 7f0eae1c2640 1 -- 172.21.15.71:0/1197262942 wait complete. 2023-08-25T09:59:22.024 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T09:59:22.025 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T09:59:22.148 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:22.171 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:22.171 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:22.171 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:22.171 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:22.172 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:22.172 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:22.172 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:22.172 INFO:tasks.workunit.client.0.smithi071.stdout: 106598 ? 00:00:00 ps 2023-08-25T09:59:22.183 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:22.234 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:22.245 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:22.462 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:22.462 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:22.473 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:22.473 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:22.506 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:22.532 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:22.536 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:22.680 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:22.706 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:22.707 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:22.707 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:22.707 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:22.707 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:22.707 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:22.708 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:22.708 INFO:tasks.workunit.client.0.smithi071.stdout: 106608 ? 00:00:00 ps 2023-08-25T09:59:22.721 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:22.772 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:22.783 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:22.799 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:22.799 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:22.808 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:22.809 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:22.882 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:22.913 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:22.919 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:23.014 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:23.037 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:23.038 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:23.038 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:23.038 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:23.038 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:23.039 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:23.039 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:23.039 INFO:tasks.workunit.client.0.smithi071.stdout: 106618 ? 00:00:00 ps 2023-08-25T09:59:23.048 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:23.148 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:23.159 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:23.173 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:23.174 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:23.182 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:23.182 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:23.217 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:23.241 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:23.249 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:23.345 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:23.367 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:23.367 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:23.367 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:23.368 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:23.368 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:23.368 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:23.368 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:23.368 INFO:tasks.workunit.client.0.smithi071.stdout: 106628 ? 00:00:00 ps 2023-08-25T09:59:23.379 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:23.429 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:23.442 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:23.458 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:23.458 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:23.469 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:23.469 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:23.611 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:23.643 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:23.651 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:23.733 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:23.756 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:23.756 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:23.756 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:23.756 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:23.756 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:23.757 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:23.757 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:23.757 INFO:tasks.workunit.client.0.smithi071.stdout: 106638 ? 00:00:00 ps 2023-08-25T09:59:23.767 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:24.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:24.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.248+0000 7f2a9ff0c640 1 Processor -- start 2023-08-25T09:59:24.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.248+0000 7f2a9ff0c640 1 -- start start 2023-08-25T09:59:24.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.250+0000 7f2a9ff0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a980c4560 0x7f2a9806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:24.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.250+0000 7f2a9ff0c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2a98070cf0 0x7f2a981637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:24.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.250+0000 7f2a9ff0c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9806eb60 con 0x7f2a98152d80 2023-08-25T09:59:24.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.250+0000 7f2a9ff0c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a9806ca30 con 0x7f2a98070cf0 2023-08-25T09:59:24.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.250+0000 7f2a9ff0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9806cf90 con 0x7f2a980c4560 2023-08-25T09:59:24.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9d4a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a980c4560 0x7f2a9806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:24.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9e4a9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2a98070cf0 0x7f2a981637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:24.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9d4a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a980c4560 0x7f2a9806e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:44712/0 (socket says 172.21.15.71:44712) 2023-08-25T09:59:24.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9d4a7640 1 -- 172.21.15.71:0/1005439567 learned_addr learned my addr 172.21.15.71:0/1005439567 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:24.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9e4a9640 1 -- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a980c4560 msgr2=0x7f2a9806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:24.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9e4a9640 1 --2- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a980c4560 0x7f2a9806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9e4a9640 1 -- 172.21.15.71:0/1005439567 >> v1:172.21.15.164:6789/0 conn(0x7f2a98152d80 legacy=0x7f2a980c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:24.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.251+0000 7f2a9e4a9640 1 -- 172.21.15.71:0/1005439567 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a98158fd0 con 0x7f2a98070cf0 2023-08-25T09:59:24.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.253+0000 7f2a9e4a9640 1 --2- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a981637b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009d90 tx=0x7f2a9002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f3e655136076e0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:24.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.253+0000 7f2a9cca6640 1 -- 172.21.15.71:0/1005439567 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9002cbc0 con 0x7f2a98070cf0 2023-08-25T09:59:24.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.253+0000 7f2a9cca6640 1 -- 172.21.15.71:0/1005439567 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9002cd20 con 0x7f2a98070cf0 2023-08-25T09:59:24.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.253+0000 7f2a9cca6640 1 -- 172.21.15.71:0/1005439567 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90034660 con 0x7f2a98070cf0 2023-08-25T09:59:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 msgr2=0x7f2a981637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a981637b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009d90 tx=0x7f2a9002be10 comp rx=0 tx=0).stop 2023-08-25T09:59:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1005439567 shutdown_connections 2023-08-25T09:59:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a981637b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1005439567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a980c4560 0x7f2a9806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1005439567 >> 172.21.15.71:0/1005439567 conn(0x7f2a980654f0 msgr2=0x7f2a98065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:24.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1005439567 shutdown_connections 2023-08-25T09:59:24.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.254+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1005439567 wait complete. 2023-08-25T09:59:24.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.256+0000 7f2a9ff0c640 1 Processor -- start 2023-08-25T09:59:24.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.256+0000 7f2a9ff0c640 1 -- start start 2023-08-25T09:59:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9ff0c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a98157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9ff0c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 0x7f2a980c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9ff0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a98152d80 0x7f2a98155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9ff0c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98154220 con 0x7f2a980c4560 2023-08-25T09:59:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9ff0c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98158e50 con 0x7f2a98070cf0 2023-08-25T09:59:24.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9ff0c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a98154820 con 0x7f2a98152d80 2023-08-25T09:59:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9d4a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 0x7f2a980c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9d4a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 0x7f2a980c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46066/0 (socket says 172.21.15.71:46066) 2023-08-25T09:59:24.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.257+0000 7f2a9d4a7640 1 -- 172.21.15.71:0/1415215685 learned_addr learned my addr 172.21.15.71:0/1415215685 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:24.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9dca8640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a98157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:24.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9e4a9640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a98152d80 0x7f2a98155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:24.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9d4a7640 1 -- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a98152d80 msgr2=0x7f2a98155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:24.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9d4a7640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a98152d80 0x7f2a98155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9d4a7640 1 -- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 msgr2=0x7f2a98157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9d4a7640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a98157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9d4a7640 1 -- 172.21.15.71:0/1415215685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90009a70 con 0x7f2a980c4560 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9d4a7640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 0x7f2a980c9980 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e870 tx=0x7f2a8800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800c8e0 con 0x7f2a980c4560 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8800ca40 con 0x7f2a980c4560 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800cdf0 con 0x7f2a980c4560 2023-08-25T09:59:24.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.258+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a981f24c0 con 0x7f2a980c4560 2023-08-25T09:59:24.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.259+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a98154dc0 con 0x7f2a980c4560 2023-08-25T09:59:24.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.260+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a98069160 con 0x7f2a980c4560 2023-08-25T09:59:24.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.261+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2a88013470 con 0x7f2a980c4560 2023-08-25T09:59:24.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.262+0000 7f2a8e7fc640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7c06a080 0x7f2a7c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:24.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.262+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7327+0+0 (secure 0 0 0) 0x7f2a88088ae0 con 0x7f2a980c4560 2023-08-25T09:59:24.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.262+0000 7f2a9dca8640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7c06a080 0x7f2a7c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:24.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.263+0000 7f2a9dca8640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7c06a080 0x7f2a7c06c540 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2a94000f60 tx=0x7f2a94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:24.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.266+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2a88053cd0 con 0x7f2a980c4560 2023-08-25T09:59:24.397 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:24.407 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:24.420 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 20` 2023-08-25T09:59:24.420 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a 2023-08-25T09:59:24.428 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/one 2023-08-25T09:59:24.428 INFO:tasks.workunit.client.0.smithi071.stderr:+ cat 2023-08-25T09:59:24.457 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir a/.snap/two 2023-08-25T09:59:24.483 INFO:tasks.workunit.client.0.smithi071.stderr:+ chmod 777 a/foo 2023-08-25T09:59:24.488 INFO:tasks.workunit.client.0.smithi071.stderr:+ sync 2023-08-25T09:59:24.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.506+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a9806c2e0 con 0x7f2a980c4560 2023-08-25T09:59:24.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.507+0000 7f2a8e7fc640 1 -- 172.21.15.71:0/1415215685 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f2a880572f0 con 0x7f2a980c4560 2023-08-25T09:59:24.510 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:24.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:24.510 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:24.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7c06a080 msgr2=0x7f2a7c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:24.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7c06a080 0x7f2a7c06c540 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2a94000f60 tx=0x7f2a94009040 comp rx=0 tx=0).stop 2023-08-25T09:59:24.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 msgr2=0x7f2a980c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:24.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 0x7f2a980c9980 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e870 tx=0x7f2a8800ed30 comp rx=0 tx=0).stop 2023-08-25T09:59:24.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 shutdown_connections 2023-08-25T09:59:24.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7c06a080 0x7f2a7c06c540 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a98152d80 0x7f2a98155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a980c4560 0x7f2a980c9980 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 --2- 172.21.15.71:0/1415215685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a98070cf0 0x7f2a98157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:24.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 >> 172.21.15.71:0/1415215685 conn(0x7f2a980654f0 msgr2=0x7f2a980c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:24.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 shutdown_connections 2023-08-25T09:59:24.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:24.510+0000 7f2a9ff0c640 1 -- 172.21.15.71:0/1415215685 wait complete. 2023-08-25T09:59:24.531 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:24.617 INFO:tasks.workunit.client.0.smithi071.stderr:+ ps 2023-08-25T09:59:24.644 INFO:tasks.workunit.client.0.smithi071.stdout: PID TTY TIME CMD 2023-08-25T09:59:24.644 INFO:tasks.workunit.client.0.smithi071.stdout: 879 ? 00:00:00 systemd 2023-08-25T09:59:24.644 INFO:tasks.workunit.client.0.smithi071.stdout: 907 ? 00:00:00 (sd-pam) 2023-08-25T09:59:24.644 INFO:tasks.workunit.client.0.smithi071.stdout: 75249 ? 00:00:03 sshd 2023-08-25T09:59:24.644 INFO:tasks.workunit.client.0.smithi071.stdout: 98509 ? 00:00:00 bash 2023-08-25T09:59:24.645 INFO:tasks.workunit.client.0.smithi071.stdout: 106280 ? 00:00:00 timeout 2023-08-25T09:59:24.645 INFO:tasks.workunit.client.0.smithi071.stdout: 106301 ? 00:00:00 snaptest-double 2023-08-25T09:59:24.645 INFO:tasks.workunit.client.0.smithi071.stdout: 106690 ? 00:00:00 ps 2023-08-25T09:59:24.655 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a/.snap/one a/.snap/two 2023-08-25T09:59:24.705 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm a/foo 2023-08-25T09:59:24.717 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir a 2023-08-25T09:59:24.736 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:59:24.737 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:59:24.738 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:59:24.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:59:24.873 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-08-25T09:59:24.873 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-08-25T09:59:24.964 INFO:tasks.workunit.client.0.smithi071.stderr:+ mkdir .snap/foo 2023-08-25T09:59:25.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:59:25.012 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:59:25.563 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2023-08-25T09:59:25.564 INFO:tasks.workunit.client.0.smithi071.stdout:We want ENOENT, not ESTALE, here. 2023-08-25T09:59:25.564 INFO:tasks.workunit.client.0.smithi071.stderr:++ seq 1 100 2023-08-25T09:59:25.567 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.568 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/1 2023-08-25T09:59:25.569 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.580 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-08-25T09:59:25.581 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.582 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.582 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/2 2023-08-25T09:59:25.592 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-08-25T09:59:25.593 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.594 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/3 2023-08-25T09:59:25.594 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.601 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-08-25T09:59:25.602 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.602 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/4 2023-08-25T09:59:25.603 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.610 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-08-25T09:59:25.611 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.611 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/5 2023-08-25T09:59:25.612 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.618 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-08-25T09:59:25.619 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.620 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/6 2023-08-25T09:59:25.620 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.629 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-08-25T09:59:25.629 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.630 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/7 2023-08-25T09:59:25.630 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.641 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-08-25T09:59:25.642 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.643 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/8 2023-08-25T09:59:25.644 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.650 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-08-25T09:59:25.651 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.652 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/9 2023-08-25T09:59:25.659 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-08-25T09:59:25.660 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.660 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/10 2023-08-25T09:59:25.660 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.666 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-08-25T09:59:25.667 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.667 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/11 2023-08-25T09:59:25.667 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.674 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-08-25T09:59:25.675 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.676 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/12 2023-08-25T09:59:25.676 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.683 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-08-25T09:59:25.683 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.684 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/13 2023-08-25T09:59:25.684 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.691 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-08-25T09:59:25.691 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.692 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/14 2023-08-25T09:59:25.692 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.698 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-08-25T09:59:25.699 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.700 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/15 2023-08-25T09:59:25.700 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.706 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-08-25T09:59:25.707 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.707 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/16 2023-08-25T09:59:25.708 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.714 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-08-25T09:59:25.715 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.715 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/17 2023-08-25T09:59:25.716 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.722 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-08-25T09:59:25.723 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/18 2023-08-25T09:59:25.724 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.731 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-08-25T09:59:25.732 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.733 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/19 2023-08-25T09:59:25.734 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.738 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-08-25T09:59:25.739 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.739 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/20 2023-08-25T09:59:25.740 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.746 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-08-25T09:59:25.747 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.747 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/21 2023-08-25T09:59:25.748 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.757 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-08-25T09:59:25.758 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.759 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/22 2023-08-25T09:59:25.760 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.776 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-08-25T09:59:25.777 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.778 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/23 2023-08-25T09:59:25.779 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.790 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-08-25T09:59:25.791 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.791 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:25.791 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/24 2023-08-25T09:59:25.799 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-08-25T09:59:25.800 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:25.801 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/25 2023-08-25T09:59:25.802 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.423 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-08-25T09:59:26.424 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.426 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/26 2023-08-25T09:59:26.427 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.435 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-08-25T09:59:26.436 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.437 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/27 2023-08-25T09:59:26.437 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.446 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-08-25T09:59:26.446 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.447 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/28 2023-08-25T09:59:26.449 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.459 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-08-25T09:59:26.459 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.460 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/29 2023-08-25T09:59:26.461 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.470 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-08-25T09:59:26.470 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.471 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/30 2023-08-25T09:59:26.471 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.480 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-08-25T09:59:26.480 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.481 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/31 2023-08-25T09:59:26.481 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.490 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-08-25T09:59:26.490 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.491 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/32 2023-08-25T09:59:26.492 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.500 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-08-25T09:59:26.501 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.502 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/33 2023-08-25T09:59:26.503 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.511 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-08-25T09:59:26.511 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.512 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/34 2023-08-25T09:59:26.512 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.520 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-08-25T09:59:26.521 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.521 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/35 2023-08-25T09:59:26.522 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.529 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-08-25T09:59:26.530 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.530 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/36 2023-08-25T09:59:26.531 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:26.539 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-08-25T09:59:26.539 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.540 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/37 2023-08-25T09:59:26.540 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.548 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-08-25T09:59:26.548 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.549 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/38 2023-08-25T09:59:26.549 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.557 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-08-25T09:59:26.558 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.558 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/39 2023-08-25T09:59:26.559 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.565 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-08-25T09:59:26.566 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.566 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/40 2023-08-25T09:59:26.567 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.574 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-08-25T09:59:26.574 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.575 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/41 2023-08-25T09:59:26.575 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.582 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-08-25T09:59:26.583 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.583 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/42 2023-08-25T09:59:26.584 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.591 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-08-25T09:59:26.592 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.592 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/43 2023-08-25T09:59:26.593 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.601 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-08-25T09:59:26.602 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.602 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/44 2023-08-25T09:59:26.602 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.609 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-08-25T09:59:26.610 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.610 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/45 2023-08-25T09:59:26.611 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.617 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-08-25T09:59:26.617 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.618 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/46 2023-08-25T09:59:26.618 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.624 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-08-25T09:59:26.624 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.625 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/47 2023-08-25T09:59:26.625 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.633 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-08-25T09:59:26.634 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.634 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/48 2023-08-25T09:59:26.635 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.644 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-08-25T09:59:26.644 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.648 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/49 2023-08-25T09:59:26.649 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.653 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-08-25T09:59:26.654 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.655 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/50 2023-08-25T09:59:26.655 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.661 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-08-25T09:59:26.662 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.662 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/51 2023-08-25T09:59:26.663 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.669 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-08-25T09:59:26.670 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.670 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/52 2023-08-25T09:59:26.671 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.676 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-08-25T09:59:26.677 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.677 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/53 2023-08-25T09:59:26.678 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.685 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-08-25T09:59:26.686 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.687 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/54 2023-08-25T09:59:26.687 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.692 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-08-25T09:59:26.693 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.693 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/55 2023-08-25T09:59:26.694 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.702 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-08-25T09:59:26.703 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.703 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/56 2023-08-25T09:59:26.704 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.710 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-08-25T09:59:26.711 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.713 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/57 2023-08-25T09:59:26.714 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.718 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-08-25T09:59:26.719 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.719 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/58 2023-08-25T09:59:26.720 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.727 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-08-25T09:59:26.728 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.728 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/59 2023-08-25T09:59:26.731 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.735 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-08-25T09:59:26.737 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.738 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/60 2023-08-25T09:59:26.738 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.745 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-08-25T09:59:26.745 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.746 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/61 2023-08-25T09:59:26.747 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.750+0000 7fa07f90c640 1 Processor -- start 2023-08-25T09:59:26.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.750+0000 7fa07f90c640 1 -- start start 2023-08-25T09:59:26.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.752+0000 7fa07f90c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078152d80 0x7fa07806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.752+0000 7fa07f90c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0780c4560 0x7fa0780c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.755 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-08-25T09:59:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.752+0000 7fa07f90c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa078070cf0 0x7fa0780c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.752+0000 7fa07f90c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07806eb60 con 0x7fa078070cf0 2023-08-25T09:59:26.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07f90c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07806ca30 con 0x7fa0780c4560 2023-08-25T09:59:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07f90c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07806cf90 con 0x7fa078152d80 2023-08-25T09:59:26.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0780c4560 0x7fa0780c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0780c4560 0x7fa0780c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:39690/0 (socket says 172.21.15.71:39690) 2023-08-25T09:59:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 -- 172.21.15.71:0/410957097 learned_addr learned my addr 172.21.15.71:0/410957097 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07d6a8640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078152d80 0x7fa07806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:26.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 -- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078152d80 msgr2=0x7fa07806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078152d80 0x7fa07806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 -- 172.21.15.71:0/410957097 >> v2:172.21.15.71:3300/0 conn(0x7fa078070cf0 msgr2=0x7fa0780c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:26.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 --2- 172.21.15.71:0/410957097 >> v2:172.21.15.71:3300/0 conn(0x7fa078070cf0 0x7fa0780c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07cea7640 1 -- 172.21.15.71:0/410957097 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078158fd0 con 0x7fa0780c4560 2023-08-25T09:59:26.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.753+0000 7fa07d6a8640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078152d80 0x7fa07806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:59:26.761 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.761 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/62 2023-08-25T09:59:26.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.755+0000 7fa07cea7640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0780c6080 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa078071d70 tx=0x7fa06802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16826fe240f597b7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:26.762 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.756+0000 7fa06ffff640 1 -- 172.21.15.71:0/410957097 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680317e0 con 0x7fa0780c4560 2023-08-25T09:59:26.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.756+0000 7fa06ffff640 1 -- 172.21.15.71:0/410957097 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa068031940 con 0x7fa0780c4560 2023-08-25T09:59:26.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.756+0000 7fa06ffff640 1 -- 172.21.15.71:0/410957097 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068031cd0 con 0x7fa0780c4560 2023-08-25T09:59:26.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.757+0000 7fa07f90c640 1 -- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 msgr2=0x7fa0780c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:26.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.757+0000 7fa07f90c640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0780c6080 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa078071d70 tx=0x7fa06802b880 comp rx=0 tx=0).stop 2023-08-25T09:59:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 -- 172.21.15.71:0/410957097 shutdown_connections 2023-08-25T09:59:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 --2- 172.21.15.71:0/410957097 >> v2:172.21.15.71:3300/0 conn(0x7fa078070cf0 0x7fa0780c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0780c6080 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 --2- 172.21.15.71:0/410957097 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078152d80 0x7fa07806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 -- 172.21.15.71:0/410957097 >> 172.21.15.71:0/410957097 conn(0x7fa0780654f0 msgr2=0x7fa078065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:26.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 -- 172.21.15.71:0/410957097 shutdown_connections 2023-08-25T09:59:26.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.758+0000 7fa07f90c640 1 -- 172.21.15.71:0/410957097 wait complete. 2023-08-25T09:59:26.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.761+0000 7fa07f90c640 1 Processor -- start 2023-08-25T09:59:26.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.762+0000 7fa07f90c640 1 -- start start 2023-08-25T09:59:26.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07f90c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078070cf0 0x7fa0781ec240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07f90c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0781ec780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.768 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-08-25T09:59:26.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07f90c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa078152d80 0x7fa07814aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07f90c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781f1670 con 0x7fa078152d80 2023-08-25T09:59:26.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07f90c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07814e930 con 0x7fa0780c4560 2023-08-25T09:59:26.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07f90c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781f11e0 con 0x7fa078070cf0 2023-08-25T09:59:26.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0781ec780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:26.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07d6a8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078070cf0 0x7fa0781ec240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:26.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07dea9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa078152d80 0x7fa07814aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:26.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0781ec780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:39702/0 (socket says 172.21.15.71:39702) 2023-08-25T09:59:26.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 -- 172.21.15.71:0/2428733787 learned_addr learned my addr 172.21.15.71:0/2428733787 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:26.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07d6a8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078070cf0 0x7fa0781ec240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:44754/0 (socket says 172.21.15.71:44754) 2023-08-25T09:59:26.772 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 -- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078070cf0 msgr2=0x7fa0781ec240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:26.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078070cf0 0x7fa0781ec240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 -- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa078152d80 msgr2=0x7fa07814aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:26.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa078152d80 0x7fa07814aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:26.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.763+0000 7fa07cea7640 1 -- 172.21.15.71:0/2428733787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0680094d0 con 0x7fa0780c4560 2023-08-25T09:59:26.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.764+0000 7fa07dea9640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa078152d80 0x7fa07814aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T09:59:26.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.764+0000 7fa07cea7640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0781ec780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa068000c00 tx=0x7fa0680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:26.774 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/63 2023-08-25T09:59:26.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.764+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068032880 con 0x7fa0780c4560 2023-08-25T09:59:26.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.764+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0781f02f0 con 0x7fa0780c4560 2023-08-25T09:59:26.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.764+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0781f08b0 con 0x7fa0780c4560 2023-08-25T09:59:26.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.765+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0680329e0 con 0x7fa0780c4560 2023-08-25T09:59:26.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.765+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068032d90 con 0x7fa0780c4560 2023-08-25T09:59:26.778 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.766+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa058005250 con 0x7fa0780c4560 2023-08-25T09:59:26.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.767+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa0680453f0 con 0x7fa0780c4560 2023-08-25T09:59:26.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.768+0000 7fa06dffb640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa05c06a050 0x7fa05c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:26.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.768+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fa0680aac60 con 0x7fa0780c4560 2023-08-25T09:59:26.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.768+0000 7fa07d6a8640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa05c06a050 0x7fa05c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:26.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.769+0000 7fa07d6a8640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa05c06a050 0x7fa05c06c510 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa074006fd0 tx=0x7fa074008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:26.782 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-08-25T09:59:26.783 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.783 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/64 2023-08-25T09:59:26.783 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:26.773+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa068075ec0 con 0x7fa0780c4560 2023-08-25T09:59:26.784 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-08-25T09:59:26.785 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.785 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/65 2023-08-25T09:59:26.785 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.786 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-08-25T09:59:26.786 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.787 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/66 2023-08-25T09:59:26.787 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.789 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-08-25T09:59:26.790 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.790 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/67 2023-08-25T09:59:26.790 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.795 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-08-25T09:59:26.796 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.797 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/68 2023-08-25T09:59:26.797 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.801 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-08-25T09:59:26.801 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.802 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/69 2023-08-25T09:59:26.802 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.806 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-08-25T09:59:26.807 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.810 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/70 2023-08-25T09:59:26.810 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.813 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-08-25T09:59:26.813 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.814 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/71 2023-08-25T09:59:26.814 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.819 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-08-25T09:59:26.820 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.820 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/72 2023-08-25T09:59:26.821 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.826 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-08-25T09:59:26.826 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/73 2023-08-25T09:59:26.827 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.833 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-08-25T09:59:26.835 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.835 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/74 2023-08-25T09:59:26.835 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.842 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-08-25T09:59:26.843 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.844 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/75 2023-08-25T09:59:26.844 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.849 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-08-25T09:59:26.850 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.851 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.851 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/76 2023-08-25T09:59:26.858 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-08-25T09:59:26.859 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.859 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/77 2023-08-25T09:59:26.859 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.866 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-08-25T09:59:26.867 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.867 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/78 2023-08-25T09:59:26.868 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.875 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-08-25T09:59:26.876 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.877 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.877 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/79 2023-08-25T09:59:26.884 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-08-25T09:59:26.885 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.885 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/80 2023-08-25T09:59:26.885 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.893 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-08-25T09:59:26.893 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.894 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/81 2023-08-25T09:59:26.894 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.900 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-08-25T09:59:26.901 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.901 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/82 2023-08-25T09:59:26.902 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.908 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-08-25T09:59:26.908 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.909 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/83 2023-08-25T09:59:26.909 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.915 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-08-25T09:59:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.916 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/84 2023-08-25T09:59:26.917 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.923 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-08-25T09:59:26.924 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.924 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/85 2023-08-25T09:59:26.926 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.933 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-08-25T09:59:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.933 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/86 2023-08-25T09:59:26.934 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.939 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-08-25T09:59:26.939 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/87 2023-08-25T09:59:26.940 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.947 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-08-25T09:59:26.948 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/88 2023-08-25T09:59:26.949 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.955 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-08-25T09:59:26.956 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.956 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/89 2023-08-25T09:59:26.957 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.964 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-08-25T09:59:26.965 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.965 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/90 2023-08-25T09:59:26.966 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.973 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-08-25T09:59:26.974 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.974 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/91 2023-08-25T09:59:26.974 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.982 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-08-25T09:59:26.983 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.983 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/92 2023-08-25T09:59:26.984 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:26.991 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-08-25T09:59:26.992 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:26.993 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/93 2023-08-25T09:59:26.993 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.026+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa058005a80 con 0x7fa0780c4560 2023-08-25T09:59:27.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.027+0000 7fa06dffb640 1 -- 172.21.15.71:0/2428733787 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fa0680794e0 con 0x7fa0780c4560 2023-08-25T09:59:27.029 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:27.030 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:27.030 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:27.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa05c06a050 msgr2=0x7fa05c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa05c06a050 0x7fa05c06c510 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa074006fd0 tx=0x7fa074008040 comp rx=0 tx=0).stop 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 msgr2=0x7fa0781ec780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0781ec780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa068000c00 tx=0x7fa0680046c0 comp rx=0 tx=0).stop 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 shutdown_connections 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0780c4560 0x7fa0781ec780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa05c06a050 0x7fa05c06c510 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:27.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa078152d80 0x7fa07814aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:27.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 --2- 172.21.15.71:0/2428733787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa078070cf0 0x7fa0781ec240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:27.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.030+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 >> 172.21.15.71:0/2428733787 conn(0x7fa0780654f0 msgr2=0x7fa0780c2020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:27.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.031+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 shutdown_connections 2023-08-25T09:59:27.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:27.031+0000 7fa07f90c640 1 -- 172.21.15.71:0/2428733787 wait complete. 2023-08-25T09:59:27.050 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:27.772 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-08-25T09:59:27.773 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.774 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/94 2023-08-25T09:59:27.775 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.786 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-08-25T09:59:27.786 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.788 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/95 2023-08-25T09:59:27.789 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.798 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-08-25T09:59:27.798 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.800 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/96 2023-08-25T09:59:27.801 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.810 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-08-25T09:59:27.810 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.811 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/97 2023-08-25T09:59:27.812 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.820 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-08-25T09:59:27.821 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.822 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/98 2023-08-25T09:59:27.823 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.834 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-08-25T09:59:27.834 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.835 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/99 2023-08-25T09:59:27.836 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.845 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-08-25T09:59:27.846 INFO:tasks.workunit.client.0.smithi071.stderr:+ for f in `seq 1 100` 2023-08-25T09:59:27.847 INFO:tasks.workunit.client.0.smithi071.stderr:+ stat .snap/foo/100 2023-08-25T09:59:27.848 INFO:tasks.workunit.client.0.smithi071.stderr:+ grep 'No such file' 2023-08-25T09:59:27.857 INFO:tasks.workunit.client.0.smithi071.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-08-25T09:59:27.858 INFO:tasks.workunit.client.0.smithi071.stderr:+ rmdir .snap/foo 2023-08-25T09:59:28.506 INFO:tasks.workunit.client.0.smithi071.stdout:OK 2023-08-25T09:59:28.506 INFO:tasks.workunit.client.0.smithi071.stderr:+ echo OK 2023-08-25T09:59:28.507 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-08-25T09:59:28.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-08-25T09:59:28.630 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-08-25T09:59:28.631 DEBUG:teuthology.orchestra.run.smithi071:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-08-25T09:59:28.727 INFO:tasks.workunit.client.0.smithi071.stderr:+ set -e 2023-08-25T09:59:28.727 INFO:tasks.workunit.client.0.smithi071.stderr:+ retried=false 2023-08-25T09:59:28.727 INFO:tasks.workunit.client.0.smithi071.stderr:+ trap -- retry EXIT 2023-08-25T09:59:28.727 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm -rf ceph 2023-08-25T09:59:28.731 INFO:tasks.workunit.client.0.smithi071.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-08-25T09:59:28.747 INFO:tasks.workunit.client.0.smithi071.stderr:Cloning into 'ceph'... 2023-08-25T09:59:29.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:29.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.288+0000 7efcad8d3640 1 Processor -- start 2023-08-25T09:59:29.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.288+0000 7efcad8d3640 1 -- start start 2023-08-25T09:59:29.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.290+0000 7efcad8d3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efca806bf00 0x7efca8070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:29.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.290+0000 7efcad8d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 0x7efca80713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:29.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.290+0000 7efcad8d3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca806a900 con 0x7efca8153170 2023-08-25T09:59:29.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.290+0000 7efcad8d3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca806fd80 con 0x7efca806bf00 2023-08-25T09:59:29.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.290+0000 7efcad8d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca806f7e0 con 0x7efca8159920 2023-08-25T09:59:29.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.291+0000 7efca77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 0x7efca80713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:29.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.291+0000 7efca77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 0x7efca80713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:44766/0 (socket says 172.21.15.71:44766) 2023-08-25T09:59:29.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.291+0000 7efca77fe640 1 -- 172.21.15.71:0/457009128 learned_addr learned my addr 172.21.15.71:0/457009128 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:29.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.291+0000 7efca67fc640 1 --2- 172.21.15.71:0/457009128 >> v2:172.21.15.164:3300/0 conn(0x7efca806bf00 0x7efca8070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.292+0000 7efca77fe640 1 -- 172.21.15.71:0/457009128 >> v2:172.21.15.164:3300/0 conn(0x7efca806bf00 msgr2=0x7efca8070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.292+0000 7efca77fe640 1 --2- 172.21.15.71:0/457009128 >> v2:172.21.15.164:3300/0 conn(0x7efca806bf00 0x7efca8070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.292+0000 7efca77fe640 1 -- 172.21.15.71:0/457009128 >> v1:172.21.15.164:6789/0 conn(0x7efca8153170 legacy=0x7efca806a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.292+0000 7efca77fe640 1 -- 172.21.15.71:0/457009128 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8070970 con 0x7efca8159920 2023-08-25T09:59:29.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.294+0000 7efca77fe640 1 --2- 172.21.15.71:0/457009128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 0x7efca80713f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efca0009e50 tx=0x7efca002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf17e85f18d8413 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:29.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.294+0000 7efca5ffb640 1 -- 172.21.15.71:0/457009128 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca002ca70 con 0x7efca8159920 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.294+0000 7efca5ffb640 1 -- 172.21.15.71:0/457009128 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca002cbd0 con 0x7efca8159920 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.294+0000 7efca5ffb640 1 -- 172.21.15.71:0/457009128 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca00343f0 con 0x7efca8159920 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 -- 172.21.15.71:0/457009128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 msgr2=0x7efca80713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 --2- 172.21.15.71:0/457009128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 0x7efca80713f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efca0009e50 tx=0x7efca002bed0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 -- 172.21.15.71:0/457009128 shutdown_connections 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 --2- 172.21.15.71:0/457009128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8159920 0x7efca80713f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 --2- 172.21.15.71:0/457009128 >> v2:172.21.15.164:3300/0 conn(0x7efca806bf00 0x7efca8070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 -- 172.21.15.71:0/457009128 >> 172.21.15.71:0/457009128 conn(0x7efca80654f0 msgr2=0x7efca8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:29.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 -- 172.21.15.71:0/457009128 shutdown_connections 2023-08-25T09:59:29.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.295+0000 7efcad8d3640 1 -- 172.21.15.71:0/457009128 wait complete. 2023-08-25T09:59:29.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.297+0000 7efcad8d3640 1 Processor -- start 2023-08-25T09:59:29.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.298+0000 7efcad8d3640 1 -- start start 2023-08-25T09:59:29.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efcad8d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efca806bf00 0x7efca80c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:29.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efcad8d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8153170 0x7efca80c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:29.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efcad8d3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 0x7efca80c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:29.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efcad8d3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c3ee0 con 0x7efca8159920 2023-08-25T09:59:29.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efcad8d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81eee90 con 0x7efca806bf00 2023-08-25T09:59:29.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efcad8d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c31b0 con 0x7efca8153170 2023-08-25T09:59:29.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 0x7efca80c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:29.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 0x7efca80c4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46100/0 (socket says 172.21.15.71:46100) 2023-08-25T09:59:29.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 -- 172.21.15.71:0/1926938239 learned_addr learned my addr 172.21.15.71:0/1926938239 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:29.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca67fc640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8153170 0x7efca80c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:29.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca6ffd640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efca806bf00 0x7efca80c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:29.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 -- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8153170 msgr2=0x7efca80c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:29.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8153170 0x7efca80c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 -- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efca806bf00 msgr2=0x7efca80c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:29.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efca806bf00 0x7efca80c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.299+0000 7efca77fe640 1 -- 172.21.15.71:0/1926938239 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0009b30 con 0x7efca8159920 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.300+0000 7efca77fe640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 0x7efca80c4160 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efca0009e50 tx=0x7efca0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.300+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0040670 con 0x7efca8159920 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.300+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0040810 con 0x7efca8159920 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.300+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0040bc0 con 0x7efca8159920 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.300+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca81ee8f0 con 0x7efca8159920 2023-08-25T09:59:29.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.300+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca80c2880 con 0x7efca8159920 2023-08-25T09:59:29.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.302+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca8069160 con 0x7efca8159920 2023-08-25T09:59:29.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.303+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efca0040d20 con 0x7efca8159920 2023-08-25T09:59:29.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.304+0000 7efc8ffff640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efc9406a050 0x7efc9406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:29.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.305+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7383+0+0 (secure 0 0 0) 0x7efca00abbe0 con 0x7efca8159920 2023-08-25T09:59:29.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.305+0000 7efca6ffd640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efc9406a050 0x7efc9406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:29.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.305+0000 7efca6ffd640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efc9406a050 0x7efc9406c510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efc98000f60 tx=0x7efc98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:29.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.308+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efca0076d90 con 0x7efca8159920 2023-08-25T09:59:29.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.540+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efca80c32f0 con 0x7efca8159920 2023-08-25T09:59:29.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.540+0000 7efc8ffff640 1 -- 172.21.15.71:0/1926938239 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7efca0009510 con 0x7efca8159920 2023-08-25T09:59:29.542 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:29.543 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:29.543 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:29.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efc9406a050 msgr2=0x7efc9406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:29.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efc9406a050 0x7efc9406c510 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efc98000f60 tx=0x7efc98009040 comp rx=0 tx=0).stop 2023-08-25T09:59:29.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 msgr2=0x7efca80c4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:29.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 0x7efca80c4160 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7efca0009e50 tx=0x7efca0004a40 comp rx=0 tx=0).stop 2023-08-25T09:59:29.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 shutdown_connections 2023-08-25T09:59:29.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efc9406a050 0x7efc9406c510 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efca8159920 0x7efca80c4160 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efca8153170 0x7efca80c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 --2- 172.21.15.71:0/1926938239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efca806bf00 0x7efca80c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:29.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.543+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 >> 172.21.15.71:0/1926938239 conn(0x7efca80654f0 msgr2=0x7efca8150d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:29.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.544+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 shutdown_connections 2023-08-25T09:59:29.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:29.544+0000 7efcad8d3640 1 -- 172.21.15.71:0/1926938239 wait complete. 2023-08-25T09:59:29.564 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:31.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:31.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.781+0000 7fe3721cf640 1 Processor -- start 2023-08-25T09:59:31.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.781+0000 7fe3721cf640 1 -- start start 2023-08-25T09:59:31.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.782+0000 7fe3721cf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe36c1510e0 0x7fe36c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.782+0000 7fe3721cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0c1dc0 con 0x7fe36c0c3660 2023-08-25T09:59:31.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.782+0000 7fe3721cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c156e00 con 0x7fe36c0c3a40 2023-08-25T09:59:31.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.782+0000 7fe3721cf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c157a80 con 0x7fe36c1510e0 2023-08-25T09:59:31.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.782+0000 7fe36b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe36c0c3660 0x7fe36c06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36344/0 (socket says 172.21.15.71:36344) 2023-08-25T09:59:31.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.782+0000 7fe36b7fe640 1 -- 172.21.15.71:0/2624172668 learned_addr learned my addr 172.21.15.71:0/2624172668 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:31.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530065005 0 0) 0x7fe36c0c1dc0 con 0x7fe36c0c3660 2023-08-25T09:59:31.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe354003520 con 0x7fe36c0c3660 2023-08-25T09:59:31.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36bfff640 1 --2- 172.21.15.71:0/2624172668 >> v2:172.21.15.164:3300/0 conn(0x7fe36c1510e0 0x7fe36c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:31.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030398729 0 0) 0x7fe354003520 con 0x7fe36c0c3660 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 >> v2:172.21.15.164:3300/0 conn(0x7fe36c1510e0 msgr2=0x7fe36c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 --2- 172.21.15.71:0/2624172668 >> v2:172.21.15.164:3300/0 conn(0x7fe36c1510e0 0x7fe36c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 >> v1:172.21.15.164:6789/0 conn(0x7fe36c0c3a40 legacy=0x7fe36c1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c1570e0 con 0x7fe36c0c3660 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe35c001670 con 0x7fe36c0c3660 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 >> v1:172.21.15.71:6789/0 conn(0x7fe36c0c3660 legacy=0x7fe36c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:31.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3540039e0 0x7fe354003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3540047c0 0x7fe354004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe35400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c157a80 con 0x7fe3540051d0 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35400c110 con 0x7fe3540047c0 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.783+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35400c2d0 con 0x7fe3540039e0 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36bfff640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe35400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:31.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36b7fe640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3540039e0 0x7fe354003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:31.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36bfff640 1 -- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3540039e0 msgr2=0x7fe354003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:31.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36bfff640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3540039e0 0x7fe354003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36bfff640 1 -- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3540047c0 msgr2=0x7fe354004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:31.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36bfff640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe3540047c0 0x7fe354004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36bfff640 1 -- 172.21.15.71:0/2624172668 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354003520 con 0x7fe3540051d0 2023-08-25T09:59:31.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.784+0000 7fe36b7fe640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3540039e0 0x7fe354003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:31.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.785+0000 7fe36bfff640 1 -- 172.21.15.71:0/2624172668 reap_dead start 2023-08-25T09:59:31.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.785+0000 7fe36bfff640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe35400b980 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe364007360 tx=0x7fe3540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a386d019417f58a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:31.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.785+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36402ccf0 con 0x7fe3540051d0 2023-08-25T09:59:31.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.785+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe36402ce50 con 0x7fe3540051d0 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.785+0000 7fe36a7fc640 1 -- 172.21.15.71:0/2624172668 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe364003bd0 con 0x7fe3540051d0 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 -- 172.21.15.71:0/2624172668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 msgr2=0x7fe35400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe35400b980 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fe364007360 tx=0x7fe3540057f0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 -- 172.21.15.71:0/2624172668 shutdown_connections 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 --2- 172.21.15.71:0/2624172668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe35400b980 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 -- 172.21.15.71:0/2624172668 >> 172.21.15.71:0/2624172668 conn(0x7fe36c0654f0 msgr2=0x7fe36c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 -- 172.21.15.71:0/2624172668 shutdown_connections 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.786+0000 7fe3721cf640 1 -- 172.21.15.71:0/2624172668 wait complete. 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.788+0000 7fe3721cf640 1 Processor -- start 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.788+0000 7fe3721cf640 1 -- start start 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe3721cf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe36c0c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe3721cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe36c157e20 0x7fe36c14e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe3721cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c06f2b0 0x7fe36c06f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe3721cf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c06ec50 con 0x7fe3540051d0 2023-08-25T09:59:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe3721cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c9b30 con 0x7fe36c157e20 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe3721cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c06e620 con 0x7fe36c06f2b0 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe36c0c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe36c0c95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40312/0 (socket says 172.21.15.71:40312) 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 -- 172.21.15.71:0/3840407969 learned_addr learned my addr 172.21.15.71:0/3840407969 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 -- 172.21.15.71:0/3840407969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c06f2b0 msgr2=0x7fe36c06f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c06f2b0 0x7fe36c06f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 -- 172.21.15.71:0/3840407969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe36c157e20 msgr2=0x7fe36c14e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe36c157e20 0x7fe36c14e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.789+0000 7fe36b7fe640 1 -- 172.21.15.71:0/3840407969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c15a810 con 0x7fe3540051d0 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.790+0000 7fe36b7fe640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe36c0c95f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00d810 tx=0x7fe35c00dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.790+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0074e0 con 0x7fe3540051d0 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.790+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c007640 con 0x7fe3540051d0 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.790+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0049e0 con 0x7fe3540051d0 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.790+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c06f070 con 0x7fe3540051d0 2023-08-25T09:59:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.790+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c15ad60 con 0x7fe3540051d0 2023-08-25T09:59:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.791+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c067a60 con 0x7fe3540051d0 2023-08-25T09:59:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.792+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe35c004b40 con 0x7fe3540051d0 2023-08-25T09:59:31.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.793+0000 7fe353fff640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe35806a050 0x7fe35806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:31.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.793+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fe35c0884c0 con 0x7fe3540051d0 2023-08-25T09:59:31.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.793+0000 7fe36affd640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe35806a050 0x7fe35806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:31.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.794+0000 7fe36affd640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe35806a050 0x7fe35806c510 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe36c14b8c0 tx=0x7fe3600079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:31.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:31.797+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe35c0535e0 con 0x7fe3540051d0 2023-08-25T09:59:32.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.023+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe36c157670 con 0x7fe3540051d0 2023-08-25T09:59:32.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.024+0000 7fe353fff640 1 -- 172.21.15.71:0/3840407969 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fe35c056c00 con 0x7fe3540051d0 2023-08-25T09:59:32.026 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:32.027 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:32.027 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:32.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe35806a050 msgr2=0x7fe35806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:32.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe35806a050 0x7fe35806c510 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe36c14b8c0 tx=0x7fe3600079b0 comp rx=0 tx=0).stop 2023-08-25T09:59:32.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 msgr2=0x7fe36c0c95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:32.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe36c0c95f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00d810 tx=0x7fe35c00dcd0 comp rx=0 tx=0).stop 2023-08-25T09:59:32.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 shutdown_connections 2023-08-25T09:59:32.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe36c157e20 0x7fe36c14e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:32.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe35806a050 0x7fe35806c510 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:32.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c06f2b0 0x7fe36c06f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:32.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 --2- 172.21.15.71:0/3840407969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3540051d0 0x7fe36c0c95f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:32.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 >> 172.21.15.71:0/3840407969 conn(0x7fe36c0654f0 msgr2=0x7fe36c0c45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:32.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 shutdown_connections 2023-08-25T09:59:32.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:32.027+0000 7fe3721cf640 1 -- 172.21.15.71:0/3840407969 wait complete. 2023-08-25T09:59:32.048 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:34.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.284+0000 7f77c751c640 1 Processor -- start 2023-08-25T09:59:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.284+0000 7f77c751c640 1 -- start start 2023-08-25T09:59:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.285+0000 7f77c751c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 0x7f77c00c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.285+0000 7f77c751c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00c1a60 con 0x7f77c006a540 2023-08-25T09:59:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.285+0000 7f77c751c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00c1d20 con 0x7f77c00bfbc0 2023-08-25T09:59:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.285+0000 7f77c751c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c00c0600 con 0x7f77c0152da0 2023-08-25T09:59:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.286+0000 7f77c5ab9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f77c006a540 0x7f77c0163790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36352/0 (socket says 172.21.15.71:36352) 2023-08-25T09:59:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.286+0000 7f77c5ab9640 1 -- 172.21.15.71:0/631955664 learned_addr learned my addr 172.21.15.71:0/631955664 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.286+0000 7f77c52b8640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 0x7f77c00c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.286+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743328754 0 0) 0x7f77c00c1a60 con 0x7f77c006a540 2023-08-25T09:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.286+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77ac003520 con 0x7f77c006a540 2023-08-25T09:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013543922 0 0) 0x7f77c00c1d20 con 0x7f77c00bfbc0 2023-08-25T09:59:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c00c1a60 con 0x7f77c00bfbc0 2023-08-25T09:59:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565246575 0 0) 0x7f77ac003520 con 0x7f77c006a540 2023-08-25T09:59:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 msgr2=0x7f77c00c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 0x7f77c00c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 >> v1:172.21.15.164:6789/0 conn(0x7f77c00bfbc0 legacy=0x7f77c015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c00c08e0 con 0x7f77c006a540 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f77b80019f0 con 0x7f77c006a540 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 >> v1:172.21.15.71:6789/0 conn(0x7f77c006a540 legacy=0x7f77c0163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 0x7f77ac003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77ac004d00 0x7f77ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c00c0600 con 0x7f77ac004100 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac00bac0 con 0x7f77ac004d00 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.287+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac00bc80 con 0x7f77c0152da0 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c52b8640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 0x7f77ac003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c5ab9640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77ac004d00 0x7f77ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c4ab7640 1 -- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 msgr2=0x7f77ac003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:34.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c0152da0 0x7f77ac003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c4ab7640 1 -- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77ac004d00 msgr2=0x7f77ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77ac004d00 0x7f77ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c4ab7640 1 -- 172.21.15.71:0/631955664 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac003520 con 0x7f77ac004100 2023-08-25T09:59:34.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.288+0000 7f77c5ab9640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77ac004d00 0x7f77ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T09:59:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.290+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77ac0047c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f77b00018d0 tx=0x7f77ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302c8878b6b5d10b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.290+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b002f050 con 0x7f77ac004100 2023-08-25T09:59:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.290+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b00052d0 con 0x7f77ac004100 2023-08-25T09:59:34.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.290+0000 7f77b7fff640 1 -- 172.21.15.71:0/631955664 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0003810 con 0x7f77ac004100 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.291+0000 7f77c751c640 1 -- 172.21.15.71:0/631955664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 msgr2=0x7f77ac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.291+0000 7f77c751c640 1 --2- 172.21.15.71:0/631955664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77ac0047c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f77b00018d0 tx=0x7f77ac0057f0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.291+0000 7f77c5ab9640 1 -- 172.21.15.71:0/631955664 reap_dead start 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.291+0000 7f77c751c640 1 -- 172.21.15.71:0/631955664 shutdown_connections 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.291+0000 7f77c751c640 1 -- 172.21.15.71:0/631955664 >> 172.21.15.71:0/631955664 conn(0x7f77c006d350 msgr2=0x7f77c00c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.292+0000 7f77c751c640 1 -- 172.21.15.71:0/631955664 shutdown_connections 2023-08-25T09:59:34.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.292+0000 7f77c751c640 1 -- 172.21.15.71:0/631955664 wait complete. 2023-08-25T09:59:34.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.293+0000 7f77c751c640 1 Processor -- start 2023-08-25T09:59:34.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.294+0000 7f77c751c640 1 -- start start 2023-08-25T09:59:34.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c751c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77c015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c751c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c01599d0 0x7f77c015c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c751c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c015af20 0x7f77c0148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c751c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c015b450 con 0x7f77ac004100 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c751c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c01588e0 con 0x7f77c01599d0 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c751c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c01591d0 con 0x7f77c015af20 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77c015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77c015bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40332/0 (socket says 172.21.15.71:40332) 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 -- 172.21.15.71:0/2763439895 learned_addr learned my addr 172.21.15.71:0/2763439895 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c01599d0 0x7f77c015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c5ab9640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c015af20 0x7f77c0148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 -- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c015af20 msgr2=0x7f77c0148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c015af20 0x7f77c0148520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 -- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c01599d0 msgr2=0x7f77c015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c01599d0 0x7f77c015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c52b8640 1 -- 172.21.15.71:0/2763439895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c0158b80 con 0x7f77ac004100 2023-08-25T09:59:34.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.295+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c01599d0 0x7f77c015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:34.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.296+0000 7f77c52b8640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77c015bbe0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0119a0 tx=0x7f77bc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:34.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.296+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00faf0 con 0x7f77ac004100 2023-08-25T09:59:34.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.296+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc00fc50 con 0x7f77ac004100 2023-08-25T09:59:34.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.296+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77bc014440 con 0x7f77ac004100 2023-08-25T09:59:34.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.296+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c015b6c0 con 0x7f77ac004100 2023-08-25T09:59:34.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.296+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c0148a60 con 0x7f77ac004100 2023-08-25T09:59:34.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.297+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c015b850 con 0x7f77ac004100 2023-08-25T09:59:34.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.298+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f77bc0145a0 con 0x7f77ac004100 2023-08-25T09:59:34.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.299+0000 7f77b5ffb640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77a406a050 0x7f77a406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:34.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.300+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f77bc0893b0 con 0x7f77ac004100 2023-08-25T09:59:34.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.300+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77a406a050 0x7f77a406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:34.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.301+0000 7f77c4ab7640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77a406a050 0x7f77a406c510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f77b0034000 tx=0x7f77b0031000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:34.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.304+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f77bc0544e0 con 0x7f77ac004100 2023-08-25T09:59:34.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.538+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77c014ec00 con 0x7f77ac004100 2023-08-25T09:59:34.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.539+0000 7f77b5ffb640 1 -- 172.21.15.71:0/2763439895 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f77bc022090 con 0x7f77ac004100 2023-08-25T09:59:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:34.541 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:34.541 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:34.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77a406a050 msgr2=0x7f77a406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77a406a050 0x7f77a406c510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f77b0034000 tx=0x7f77b0031000 comp rx=0 tx=0).stop 2023-08-25T09:59:34.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 msgr2=0x7f77c015bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:34.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77c015bbe0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0119a0 tx=0x7f77bc011e60 comp rx=0 tx=0).stop 2023-08-25T09:59:34.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 shutdown_connections 2023-08-25T09:59:34.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c01599d0 0x7f77c015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77a406a050 0x7f77a406c510 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77c015af20 0x7f77c0148520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 --2- 172.21.15.71:0/2763439895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77ac004100 0x7f77c015bbe0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:34.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 >> 172.21.15.71:0/2763439895 conn(0x7f77c006d350 msgr2=0x7f77c00c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:34.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.541+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 shutdown_connections 2023-08-25T09:59:34.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:34.542+0000 7f77c751c640 1 -- 172.21.15.71:0/2763439895 wait complete. 2023-08-25T09:59:34.563 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T09:59:34.564 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T09:59:36.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.800+0000 7fb088a45640 1 Processor -- start 2023-08-25T09:59:36.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.800+0000 7fb088a45640 1 -- start start 2023-08-25T09:59:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.801+0000 7fb088a45640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb084154370 0x7fb08415a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.801+0000 7fb088a45640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.801+0000 7fb088a45640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb084071a50 con 0x7fb0840c02f0 2023-08-25T09:59:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.801+0000 7fb088a45640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0840af960 con 0x7fb084154370 2023-08-25T09:59:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.801+0000 7fb088a45640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0840b0200 con 0x7fb0840c7440 2023-08-25T09:59:36.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb084154370 0x7fb08415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:36.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb084154370 0x7fb08415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40342/0 (socket says 172.21.15.71:40342) 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 -- 172.21.15.71:0/3088682988 learned_addr learned my addr 172.21.15.71:0/3088682988 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb082ffd640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 -- 172.21.15.71:0/3088682988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 msgr2=0x7fb08406d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 -- 172.21.15.71:0/3088682988 >> v1:172.21.15.164:6789/0 conn(0x7fb0840c02f0 legacy=0x7fb084069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb0837fe640 1 -- 172.21.15.71:0/3088682988 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb084071ca0 con 0x7fb084154370 2023-08-25T09:59:36.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.802+0000 7fb082ffd640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:36.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.804+0000 7fb0837fe640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08415a030 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb074009870 tx=0x7fb07402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb6636cbf9b4fc1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.804+0000 7fb0827fc640 1 -- 172.21.15.71:0/3088682988 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0740317c0 con 0x7fb084154370 2023-08-25T09:59:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.804+0000 7fb0827fc640 1 -- 172.21.15.71:0/3088682988 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb074031920 con 0x7fb084154370 2023-08-25T09:59:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.804+0000 7fb0827fc640 1 -- 172.21.15.71:0/3088682988 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb074031cb0 con 0x7fb084154370 2023-08-25T09:59:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 -- 172.21.15.71:0/3088682988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 msgr2=0x7fb08415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08415a030 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb074009870 tx=0x7fb07402b860 comp rx=0 tx=0).stop 2023-08-25T09:59:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 -- 172.21.15.71:0/3088682988 shutdown_connections 2023-08-25T09:59:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 --2- 172.21.15.71:0/3088682988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08415a030 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 -- 172.21.15.71:0/3088682988 >> 172.21.15.71:0/3088682988 conn(0x7fb0840cb4b0 msgr2=0x7fb0840cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 -- 172.21.15.71:0/3088682988 shutdown_connections 2023-08-25T09:59:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.805+0000 7fb088a45640 1 -- 172.21.15.71:0/3088682988 wait complete. 2023-08-25T09:59:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.807+0000 7fb088a45640 1 Processor -- start 2023-08-25T09:59:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.808+0000 7fb088a45640 1 -- start start 2023-08-25T09:59:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb088a45640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0840c02f0 0x7fb0840c9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb088a45640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb088a45640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08406fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb088a45640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb084151d20 con 0x7fb084154370 2023-08-25T09:59:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb088a45640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08406c970 con 0x7fb0840c02f0 2023-08-25T09:59:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb088a45640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb084151780 con 0x7fb0840c7440 2023-08-25T09:59:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb083fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08406fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb083fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08406fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40352/0 (socket says 172.21.15.71:40352) 2023-08-25T09:59:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb083fff640 1 -- 172.21.15.71:0/2087307115 learned_addr learned my addr 172.21.15.71:0/2087307115 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb082ffd640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.809+0000 7fb0837fe640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0840c02f0 0x7fb0840c9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:36.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb083fff640 1 -- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 msgr2=0x7fb08406f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:36.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb083fff640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:36.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb083fff640 1 -- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0840c02f0 msgr2=0x7fb0840c9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb083fff640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0840c02f0 0x7fb0840c9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb083fff640 1 -- 172.21.15.71:0/2087307115 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0740094d0 con 0x7fb084154370 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb083fff640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08406fc30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00b990 tx=0x7fb07c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c00cba0 con 0x7fb084154370 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07c00cd00 con 0x7fb084154370 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c014680 con 0x7fb084154370 2023-08-25T09:59:36.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb084151a40 con 0x7fb084154370 2023-08-25T09:59:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.810+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08406c170 con 0x7fb084154370 2023-08-25T09:59:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.812+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08404e420 con 0x7fb084154370 2023-08-25T09:59:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.813+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb07c01d3b0 con 0x7fb084154370 2023-08-25T09:59:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.814+0000 7fb06bfff640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb07006a080 0x7fb07006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:36.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.814+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7327+0+0 (secure 0 0 0) 0x7fb07c0889d0 con 0x7fb084154370 2023-08-25T09:59:36.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.814+0000 7fb0837fe640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb07006a080 0x7fb07006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:36.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.815+0000 7fb0837fe640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb07006a080 0x7fb07006c540 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb074036000 tx=0x7fb07403d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:36.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:36.820+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb07c053bc0 con 0x7fb084154370 2023-08-25T09:59:37.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.088+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0840ade80 con 0x7fb084154370 2023-08-25T09:59:37.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.090+0000 7fb06bfff640 1 -- 172.21.15.71:0/2087307115 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fb07c0571e0 con 0x7fb084154370 2023-08-25T09:59:37.092 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:37.092 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:37.092 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:37.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb07006a080 msgr2=0x7fb07006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:37.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb07006a080 0x7fb07006c540 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb074036000 tx=0x7fb07403d040 comp rx=0 tx=0).stop 2023-08-25T09:59:37.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 msgr2=0x7fb08406fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08406fc30 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00b990 tx=0x7fb07c00be50 comp rx=0 tx=0).stop 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 shutdown_connections 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb07006a080 0x7fb07006c540 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb084154370 0x7fb08406fc30 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0840c7440 0x7fb08406f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 --2- 172.21.15.71:0/2087307115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0840c02f0 0x7fb0840c9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:37.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.092+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 >> 172.21.15.71:0/2087307115 conn(0x7fb0840cb4b0 msgr2=0x7fb0840c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:37.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.093+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 shutdown_connections 2023-08-25T09:59:37.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:37.093+0000 7fb088a45640 1 -- 172.21.15.71:0/2087307115 wait complete. 2023-08-25T09:59:37.112 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:39.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.344+0000 7f4fc9198640 1 Processor -- start 2023-08-25T09:59:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.344+0000 7f4fc9198640 1 -- start start 2023-08-25T09:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.345+0000 7f4fc9198640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4fc40c3a40 0x7f4fc40c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc9198640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4fc41509d0 0x7f4fc4150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc9198640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40c1dc0 con 0x7f4fc40c3660 2023-08-25T09:59:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc9198640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc4157a80 con 0x7f4fc40c3a40 2023-08-25T09:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc9198640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc4151320 con 0x7f4fc41509d0 2023-08-25T09:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4fc40c3a40 0x7f4fc40c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4fc40c3a40 0x7f4fc40c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40362/0 (socket says 172.21.15.71:40362) 2023-08-25T09:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 -- 172.21.15.71:0/1131937844 learned_addr learned my addr 172.21.15.71:0/1131937844 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc359e640 1 --2- 172.21.15.71:0/1131937844 >> v2:172.21.15.164:3300/0 conn(0x7f4fc41509d0 0x7f4fc4150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 -- 172.21.15.71:0/1131937844 >> v2:172.21.15.164:3300/0 conn(0x7f4fc41509d0 msgr2=0x7f4fc4150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 --2- 172.21.15.71:0/1131937844 >> v2:172.21.15.164:3300/0 conn(0x7f4fc41509d0 0x7f4fc4150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 -- 172.21.15.71:0/1131937844 >> v1:172.21.15.164:6789/0 conn(0x7f4fc40c3660 legacy=0x7f4fc406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:39.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc259c640 1 -- 172.21.15.71:0/1131937844 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4156e00 con 0x7f4fc40c3a40 2023-08-25T09:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.346+0000 7f4fc359e640 1 --2- 172.21.15.71:0/1131937844 >> v2:172.21.15.164:3300/0 conn(0x7f4fc41509d0 0x7f4fc4150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.348+0000 7f4fc259c640 1 --2- 172.21.15.71:0/1131937844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc40c1710 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40663f0 tx=0x7f4fb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1416e4c405aec2ea server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.348+0000 7f4fc1d9b640 1 -- 172.21.15.71:0/1131937844 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8030860 con 0x7f4fc40c3a40 2023-08-25T09:59:39.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.348+0000 7f4fc1d9b640 1 -- 172.21.15.71:0/1131937844 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb80309c0 con 0x7f4fc40c3a40 2023-08-25T09:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.348+0000 7f4fc1d9b640 1 -- 172.21.15.71:0/1131937844 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8030d50 con 0x7f4fc40c3a40 2023-08-25T09:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.348+0000 7f4fc9198640 1 -- 172.21.15.71:0/1131937844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 msgr2=0x7f4fc40c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.348+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1131937844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc40c1710 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40663f0 tx=0x7f4fb802b9a0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.349+0000 7f4fc9198640 1 -- 172.21.15.71:0/1131937844 shutdown_connections 2023-08-25T09:59:39.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.349+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1131937844 >> v2:172.21.15.164:3300/0 conn(0x7f4fc41509d0 0x7f4fc4150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.349+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1131937844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc40c1710 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.349+0000 7f4fc9198640 1 -- 172.21.15.71:0/1131937844 >> 172.21.15.71:0/1131937844 conn(0x7f4fc40654f0 msgr2=0x7f4fc4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:39.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.349+0000 7f4fc9198640 1 -- 172.21.15.71:0/1131937844 shutdown_connections 2023-08-25T09:59:39.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.349+0000 7f4fc9198640 1 -- 172.21.15.71:0/1131937844 wait complete. 2023-08-25T09:59:39.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.351+0000 7f4fc9198640 1 Processor -- start 2023-08-25T09:59:39.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.351+0000 7f4fc9198640 1 -- start start 2023-08-25T09:59:39.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc9198640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fc40c3660 0x7f4fc41e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:39.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc9198640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc414a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:39.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc9198640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fc41509d0 0x7f4fc414a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:39.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc9198640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc41f2860 con 0x7f4fc40c3a40 2023-08-25T09:59:39.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc9198640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc41535d0 con 0x7f4fc40c3660 2023-08-25T09:59:39.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc9198640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc4153870 con 0x7f4fc41509d0 2023-08-25T09:59:39.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc414a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:39.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc414a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40370/0 (socket says 172.21.15.71:40370) 2023-08-25T09:59:39.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc259c640 1 -- 172.21.15.71:0/1737659189 learned_addr learned my addr 172.21.15.71:0/1737659189 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:39.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.352+0000 7f4fc359e640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fc41509d0 0x7f4fc414a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:39.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc259c640 1 -- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fc41509d0 msgr2=0x7f4fc414a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:39.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc2d9d640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fc40c3660 0x7f4fc41e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:39.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc259c640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fc41509d0 0x7f4fc414a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc259c640 1 -- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fc40c3660 msgr2=0x7f4fc41e57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:39.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc259c640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fc40c3660 0x7f4fc41e57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc259c640 1 -- 172.21.15.71:0/1737659189 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb80094d0 con 0x7f4fc40c3a40 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc359e640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fc41509d0 0x7f4fc414a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc259c640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc414a300 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000c00 tx=0x7f4fb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8030830 con 0x7f4fc40c3a40 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8043400 con 0x7f4fc40c3a40 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb80437b0 con 0x7f4fc40c3a40 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.353+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc4153b50 con 0x7f4fc40c3a40 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.354+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc4154110 con 0x7f4fc40c3a40 2023-08-25T09:59:39.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.355+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc40c29c0 con 0x7f4fc40c3a40 2023-08-25T09:59:39.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.356+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4fb8043910 con 0x7f4fc40c3a40 2023-08-25T09:59:39.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.357+0000 7f4fab7fe640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fb006a050 0x7f4fb006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:39.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.357+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7327+0+0 (secure 0 0 0) 0x7f4fb803e070 con 0x7f4fc40c3a40 2023-08-25T09:59:39.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.357+0000 7f4fc2d9d640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fb006a050 0x7f4fb006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:39.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.358+0000 7f4fc2d9d640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fb006a050 0x7f4fb006c510 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40015d0 tx=0x7f4fb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:39.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.362+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4fb8077030 con 0x7f4fc40c3a40 2023-08-25T09:59:39.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.612+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fc4069160 con 0x7f4fc40c3a40 2023-08-25T09:59:39.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.613+0000 7f4fab7fe640 1 -- 172.21.15.71:0/1737659189 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f4fb8034090 con 0x7f4fc40c3a40 2023-08-25T09:59:39.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:39.615 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:39.615 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:39.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fb006a050 msgr2=0x7f4fb006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:39.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fb006a050 0x7f4fb006c510 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40015d0 tx=0x7f4fb4009040 comp rx=0 tx=0).stop 2023-08-25T09:59:39.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 msgr2=0x7f4fc414a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:39.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc414a300 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8000c00 tx=0x7f4fb8004730 comp rx=0 tx=0).stop 2023-08-25T09:59:39.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 shutdown_connections 2023-08-25T09:59:39.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fb006a050 0x7f4fb006c510 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4fc41509d0 0x7f4fc414a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fc40c3a40 0x7f4fc414a300 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 --2- 172.21.15.71:0/1737659189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4fc40c3660 0x7f4fc41e57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:39.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 >> 172.21.15.71:0/1737659189 conn(0x7f4fc40654f0 msgr2=0x7f4fc4070630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:39.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 shutdown_connections 2023-08-25T09:59:39.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:39.615+0000 7f4fc9198640 1 -- 172.21.15.71:0/1737659189 wait complete. 2023-08-25T09:59:39.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:41.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:41.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.872+0000 7fb673342640 1 Processor -- start 2023-08-25T09:59:41.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.872+0000 7fb673342640 1 -- start start 2023-08-25T09:59:41.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.874+0000 7fb673342640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0cacb0 0x7fb66c15c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:41.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.874+0000 7fb673342640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb66c15cc80 0x7fb66c0cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:41.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.874+0000 7fb673342640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c15d0d0 con 0x7fb66c152d80 2023-08-25T09:59:41.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.874+0000 7fb673342640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb66c1591c0 con 0x7fb66c15cc80 2023-08-25T09:59:41.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.874+0000 7fb673342640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1594a0 con 0x7fb66c0cacb0 2023-08-25T09:59:41.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb66c15cc80 0x7fb66c0cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:41.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb66c15cc80 0x7fb66c0cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37868/0 (socket says 172.21.15.71:37868) 2023-08-25T09:59:41.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 -- 172.21.15.71:0/3330902607 learned_addr learned my addr 172.21.15.71:0/3330902607 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:41.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6708dd640 1 --2- 172.21.15.71:0/3330902607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0cacb0 0x7fb66c15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:41.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 -- 172.21.15.71:0/3330902607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0cacb0 msgr2=0x7fb66c15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:41.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 --2- 172.21.15.71:0/3330902607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0cacb0 0x7fb66c15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:41.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 -- 172.21.15.71:0/3330902607 >> v1:172.21.15.164:6789/0 conn(0x7fb66c152d80 legacy=0x7fb66c0c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:41.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.875+0000 7fb6718df640 1 -- 172.21.15.71:0/3330902607 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c159780 con 0x7fb66c15cc80 2023-08-25T09:59:41.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.877+0000 7fb6718df640 1 --2- 172.21.15.71:0/3330902607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c15cc80 0x7fb66c0cb140 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb664009e50 tx=0x7fb66402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410ab66d4873cbd2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:41.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb663fff640 1 -- 172.21.15.71:0/3330902607 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66402cb90 con 0x7fb66c15cc80 2023-08-25T09:59:41.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb663fff640 1 -- 172.21.15.71:0/3330902607 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66402ccf0 con 0x7fb66c15cc80 2023-08-25T09:59:41.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb663fff640 1 -- 172.21.15.71:0/3330902607 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb664034610 con 0x7fb66c15cc80 2023-08-25T09:59:41.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 -- 172.21.15.71:0/3330902607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c15cc80 msgr2=0x7fb66c0cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 --2- 172.21.15.71:0/3330902607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c15cc80 0x7fb66c0cb140 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb664009e50 tx=0x7fb66402bff0 comp rx=0 tx=0).stop 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 -- 172.21.15.71:0/3330902607 shutdown_connections 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 --2- 172.21.15.71:0/3330902607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c15cc80 0x7fb66c0cb140 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 --2- 172.21.15.71:0/3330902607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c0cacb0 0x7fb66c15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 -- 172.21.15.71:0/3330902607 >> 172.21.15.71:0/3330902607 conn(0x7fb66c06b8f0 msgr2=0x7fb66c0654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 -- 172.21.15.71:0/3330902607 shutdown_connections 2023-08-25T09:59:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.878+0000 7fb673342640 1 -- 172.21.15.71:0/3330902607 wait complete. 2023-08-25T09:59:41.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.880+0000 7fb673342640 1 Processor -- start 2023-08-25T09:59:41.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.880+0000 7fb673342640 1 -- start start 2023-08-25T09:59:41.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb673342640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 0x7fb66c150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:41.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb673342640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c152d80 0x7fb66c1423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:41.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb673342640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c15cc80 0x7fb66c0b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:41.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb673342640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c15a130 con 0x7fb66c0cacb0 2023-08-25T09:59:41.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb673342640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1f1800 con 0x7fb66c15cc80 2023-08-25T09:59:41.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb673342640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1428e0 con 0x7fb66c152d80 2023-08-25T09:59:41.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 0x7fb66c150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:41.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 0x7fb66c150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37874/0 (socket says 172.21.15.71:37874) 2023-08-25T09:59:41.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 -- 172.21.15.71:0/3353598665 learned_addr learned my addr 172.21.15.71:0/3353598665 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:41.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6708dd640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c152d80 0x7fb66c1423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:41.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 -- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c152d80 msgr2=0x7fb66c1423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:41.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c152d80 0x7fb66c1423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:41.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 -- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c15cc80 msgr2=0x7fb66c0b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:41.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6718df640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c15cc80 0x7fb66c0b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:41.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c15cc80 0x7fb66c0b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:41.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6710de640 1 -- 172.21.15.71:0/3353598665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb664009b30 con 0x7fb66c0cacb0 2023-08-25T09:59:41.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.882+0000 7fb6708dd640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c152d80 0x7fb66c1423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:41.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.883+0000 7fb6710de640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 0x7fb66c150750 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb66800c450 tx=0x7fb66800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:41.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.883+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6680052d0 con 0x7fb66c0cacb0 2023-08-25T09:59:41.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.883+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb668005430 con 0x7fb66c0cacb0 2023-08-25T09:59:41.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.883+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c1f1a30 con 0x7fb66c0cacb0 2023-08-25T09:59:41.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.883+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668012870 con 0x7fb66c0cacb0 2023-08-25T09:59:41.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.883+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c1f1ff0 con 0x7fb66c0cacb0 2023-08-25T09:59:41.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.885+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c0b1a60 con 0x7fb66c0cacb0 2023-08-25T09:59:41.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.886+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb6680129d0 con 0x7fb66c0cacb0 2023-08-25T09:59:41.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.886+0000 7fb661ffb640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a050 0x7fb65006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:41.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.887+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb668088630 con 0x7fb66c0cacb0 2023-08-25T09:59:41.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.887+0000 7fb6708dd640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a050 0x7fb65006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:41.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.888+0000 7fb6708dd640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a050 0x7fb65006c510 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0c41d0 tx=0x7fb65c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:41.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:41.891+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb6680537d0 con 0x7fb66c0cacb0 2023-08-25T09:59:42.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.147+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb66c0b2450 con 0x7fb66c0cacb0 2023-08-25T09:59:42.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.148+0000 7fb661ffb640 1 -- 172.21.15.71:0/3353598665 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fb668056df0 con 0x7fb66c0cacb0 2023-08-25T09:59:42.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:42.150 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:42.150 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:42.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a050 msgr2=0x7fb65006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:42.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a050 0x7fb65006c510 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0c41d0 tx=0x7fb65c008040 comp rx=0 tx=0).stop 2023-08-25T09:59:42.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 msgr2=0x7fb66c150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:42.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 0x7fb66c150750 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb66800c450 tx=0x7fb66800c910 comp rx=0 tx=0).stop 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 shutdown_connections 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb66c15cc80 0x7fb66c0b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb65006a050 0x7fb65006c510 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb66c152d80 0x7fb66c1423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 --2- 172.21.15.71:0/3353598665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66c0cacb0 0x7fb66c150750 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 >> 172.21.15.71:0/3353598665 conn(0x7fb66c06b8f0 msgr2=0x7fb66c0cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:42.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 shutdown_connections 2023-08-25T09:59:42.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:42.150+0000 7fb673342640 1 -- 172.21.15.71:0/3353598665 wait complete. 2023-08-25T09:59:42.170 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:44.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:44.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.395+0000 7f7a58b8b640 1 Processor -- start 2023-08-25T09:59:44.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.395+0000 7f7a58b8b640 1 -- start start 2023-08-25T09:59:44.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a58b8b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:44.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a58b8b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7a5415cc80 0x7f7a540cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:44.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a58b8b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5415d0d0 con 0x7f7a54152d80 2023-08-25T09:59:44.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a58b8b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a541591c0 con 0x7f7a5415cc80 2023-08-25T09:59:44.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a58b8b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a541594a0 con 0x7f7a540cacb0 2023-08-25T09:59:44.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a52d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7a5415cc80 0x7f7a540cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:44.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a52d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7a5415cc80 0x7f7a540cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37884/0 (socket says 172.21.15.71:37884) 2023-08-25T09:59:44.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.397+0000 7f7a52d9d640 1 -- 172.21.15.71:0/1593428237 learned_addr learned my addr 172.21.15.71:0/1593428237 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:44.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.398+0000 7f7a51d9b640 1 --2- 172.21.15.71:0/1593428237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:44.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.398+0000 7f7a52d9d640 1 -- 172.21.15.71:0/1593428237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 msgr2=0x7f7a5415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.398+0000 7f7a52d9d640 1 --2- 172.21.15.71:0/1593428237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.398+0000 7f7a52d9d640 1 -- 172.21.15.71:0/1593428237 >> v1:172.21.15.71:6789/0 conn(0x7f7a54152d80 legacy=0x7f7a540c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.398+0000 7f7a52d9d640 1 -- 172.21.15.71:0/1593428237 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a54159780 con 0x7f7a5415cc80 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.399+0000 7f7a52d9d640 1 --2- 172.21.15.71:0/1593428237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a5415cc80 0x7f7a540cb140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009e50 tx=0x7f7a4c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db222404d05069b6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.399+0000 7f7a5159a640 1 -- 172.21.15.71:0/1593428237 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c02cb90 con 0x7f7a5415cc80 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a5159a640 1 -- 172.21.15.71:0/1593428237 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c02ccf0 con 0x7f7a5415cc80 2023-08-25T09:59:44.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a5159a640 1 -- 172.21.15.71:0/1593428237 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c034610 con 0x7f7a5415cc80 2023-08-25T09:59:44.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 -- 172.21.15.71:0/1593428237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a5415cc80 msgr2=0x7f7a540cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:44.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/1593428237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a5415cc80 0x7f7a540cb140 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009e50 tx=0x7f7a4c02bff0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 -- 172.21.15.71:0/1593428237 shutdown_connections 2023-08-25T09:59:44.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/1593428237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a5415cc80 0x7f7a540cb140 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/1593428237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 -- 172.21.15.71:0/1593428237 >> 172.21.15.71:0/1593428237 conn(0x7f7a5406b8f0 msgr2=0x7f7a540654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:44.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 -- 172.21.15.71:0/1593428237 shutdown_connections 2023-08-25T09:59:44.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.400+0000 7f7a58b8b640 1 -- 172.21.15.71:0/1593428237 wait complete. 2023-08-25T09:59:44.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.402+0000 7f7a58b8b640 1 Processor -- start 2023-08-25T09:59:44.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.403+0000 7f7a58b8b640 1 -- start start 2023-08-25T09:59:44.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a58b8b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:44.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a58b8b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 0x7f7a541f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:44.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a58b8b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a5415cc80 0x7f7a541f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:44.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a58b8b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54156a10 con 0x7f7a54152d80 2023-08-25T09:59:44.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a58b8b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54144560 con 0x7f7a5415cc80 2023-08-25T09:59:44.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a58b8b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54143c70 con 0x7f7a540cacb0 2023-08-25T09:59:44.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a51d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 0x7f7a541f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:44.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a51d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 0x7f7a541f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37892/0 (socket says 172.21.15.71:37892) 2023-08-25T09:59:44.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.404+0000 7f7a51d9b640 1 -- 172.21.15.71:0/2461516194 learned_addr learned my addr 172.21.15.71:0/2461516194 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:44.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a5259c640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:44.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a51d9b640 1 -- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 msgr2=0x7f7a5415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:44.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a51d9b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a52d9d640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a5415cc80 0x7f7a541f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a51d9b640 1 -- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a5415cc80 msgr2=0x7f7a541f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a51d9b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a5415cc80 0x7f7a541f2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a51d9b640 1 -- 172.21.15.71:0/2461516194 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c009b30 con 0x7f7a54152d80 2023-08-25T09:59:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a5259c640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a52d9d640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a5415cc80 0x7f7a541f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a51d9b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 0x7f7a541f1ae0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800d5f0 tx=0x7f7a4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:44.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480125f0 con 0x7f7a54152d80 2023-08-25T09:59:44.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a48012750 con 0x7f7a54152d80 2023-08-25T09:59:44.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48012b00 con 0x7f7a54152d80 2023-08-25T09:59:44.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.405+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a54156c00 con 0x7f7a54152d80 2023-08-25T09:59:44.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.406+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a54143f10 con 0x7f7a54152d80 2023-08-25T09:59:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.407+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a540b1a60 con 0x7f7a54152d80 2023-08-25T09:59:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.408+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7a48012c60 con 0x7f7a54152d80 2023-08-25T09:59:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.409+0000 7f7a42ffd640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7a3406a050 0x7f7a3406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.409+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f7a48088f70 con 0x7f7a54152d80 2023-08-25T09:59:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.409+0000 7f7a5259c640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7a3406a050 0x7f7a3406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:44.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.410+0000 7f7a5259c640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7a3406a050 0x7f7a3406c510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000f60 tx=0x7f7a44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.414+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7a48054100 con 0x7f7a54152d80 2023-08-25T09:59:44.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.683+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a54069160 con 0x7f7a54152d80 2023-08-25T09:59:44.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.684+0000 7f7a42ffd640 1 -- 172.21.15.71:0/2461516194 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f7a48057720 con 0x7f7a54152d80 2023-08-25T09:59:44.686 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:44.686 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:44.687 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:44.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7a3406a050 msgr2=0x7f7a3406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:44.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7a3406a050 0x7f7a3406c510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000f60 tx=0x7f7a44009040 comp rx=0 tx=0).stop 2023-08-25T09:59:44.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 msgr2=0x7f7a541f1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:44.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 0x7f7a541f1ae0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800d5f0 tx=0x7f7a4800dab0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 shutdown_connections 2023-08-25T09:59:44.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a5415cc80 0x7f7a541f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7a3406a050 0x7f7a3406c510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7a54152d80 0x7f7a541f1ae0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 --2- 172.21.15.71:0/2461516194 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a540cacb0 0x7f7a5415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:44.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 >> 172.21.15.71:0/2461516194 conn(0x7f7a5406b8f0 msgr2=0x7f7a540cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:44.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.687+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 shutdown_connections 2023-08-25T09:59:44.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:44.688+0000 7f7a58b8b640 1 -- 172.21.15.71:0/2461516194 wait complete. 2023-08-25T09:59:44.709 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:46.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:46.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.932+0000 7f01ecc2c640 1 Processor -- start 2023-08-25T09:59:46.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.932+0000 7f01ecc2c640 1 -- start start 2023-08-25T09:59:46.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.934+0000 7f01ecc2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e80cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:46.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.934+0000 7f01ecc2c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01e8158d60 0x7f01e815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:46.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.934+0000 7f01ecc2c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e815c6c0 con 0x7f01e80cad40 2023-08-25T09:59:46.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.934+0000 7f01ecc2c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e81592c0 con 0x7f01e8158d60 2023-08-25T09:59:46.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.934+0000 7f01ecc2c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e81595a0 con 0x7f01e8152fc0 2023-08-25T09:59:46.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.935+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e80cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:46.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.935+0000 7f01e6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01e8158d60 0x7f01e815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:46.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.935+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e80cbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:59530/0 (socket says 172.21.15.71:59530) 2023-08-25T09:59:46.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.935+0000 7f01e77fe640 1 -- 172.21.15.71:0/1022397913 learned_addr learned my addr 172.21.15.71:0/1022397913 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:46.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.936+0000 7f01e6ffd640 1 -- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 msgr2=0x7f01e80cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:46.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.936+0000 7f01e6ffd640 1 --2- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e80cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:46.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.936+0000 7f01e6ffd640 1 -- 172.21.15.71:0/1022397913 >> v1:172.21.15.164:6789/0 conn(0x7f01e80cad40 legacy=0x7f01e80c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:46.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.936+0000 7f01e6ffd640 1 -- 172.21.15.71:0/1022397913 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8159880 con 0x7f01e8158d60 2023-08-25T09:59:46.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.937+0000 7f01e6ffd640 1 --2- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e815cb50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0097a0 tx=0x7f01dc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73f81ca97c66d1f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:46.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.938+0000 7f01e67fc640 1 -- 172.21.15.71:0/1022397913 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc02d040 con 0x7f01e8158d60 2023-08-25T09:59:46.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.938+0000 7f01e67fc640 1 -- 172.21.15.71:0/1022397913 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc031810 con 0x7f01e8158d60 2023-08-25T09:59:46.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.938+0000 7f01e67fc640 1 -- 172.21.15.71:0/1022397913 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc031ba0 con 0x7f01e8158d60 2023-08-25T09:59:46.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.938+0000 7f01ecc2c640 1 -- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 msgr2=0x7f01e815cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:46.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.938+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e815cb50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0097a0 tx=0x7f01dc02b8b0 comp rx=0 tx=0).stop 2023-08-25T09:59:46.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.939+0000 7f01ecc2c640 1 -- 172.21.15.71:0/1022397913 shutdown_connections 2023-08-25T09:59:46.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.939+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e815cb50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:46.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.939+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/1022397913 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e80cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:46.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.939+0000 7f01ecc2c640 1 -- 172.21.15.71:0/1022397913 >> 172.21.15.71:0/1022397913 conn(0x7f01e806a3a0 msgr2=0x7f01e8070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:46.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.939+0000 7f01ecc2c640 1 -- 172.21.15.71:0/1022397913 shutdown_connections 2023-08-25T09:59:46.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.939+0000 7f01ecc2c640 1 -- 172.21.15.71:0/1022397913 wait complete. 2023-08-25T09:59:46.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.940+0000 7f01ecc2c640 1 Processor -- start 2023-08-25T09:59:46.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.941+0000 7f01ecc2c640 1 -- start start 2023-08-25T09:59:46.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.941+0000 7f01ecc2c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 0x7f01e814a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:46.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01ecc2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e8156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:46.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01ecc2c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e813df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:46.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01ecc2c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e814d370 con 0x7f01e80cad40 2023-08-25T09:59:46.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01ecc2c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e814ae70 con 0x7f01e8158d60 2023-08-25T09:59:46.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01ecc2c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e814d970 con 0x7f01e8152fc0 2023-08-25T09:59:46.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 0x7f01e814a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:46.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 0x7f01e814a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37902/0 (socket says 172.21.15.71:37902) 2023-08-25T09:59:46.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 -- 172.21.15.71:0/2045873123 learned_addr learned my addr 172.21.15.71:0/2045873123 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:46.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e7fff640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e813df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:46.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e6ffd640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e8156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:46.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 -- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 msgr2=0x7f01e8156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:46.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e8156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:46.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 -- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 msgr2=0x7f01e813df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:46.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e813df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:46.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e77fe640 1 -- 172.21.15.71:0/2045873123 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc009400 con 0x7f01e80cad40 2023-08-25T09:59:46.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.942+0000 7f01e6ffd640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e8156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:46.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.943+0000 7f01e77fe640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 0x7f01e814a650 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009fd0 tx=0x7f01d800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:46.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.943+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800c940 con 0x7f01e80cad40 2023-08-25T09:59:46.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.943+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d800caa0 con 0x7f01e80cad40 2023-08-25T09:59:46.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.943+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800ce30 con 0x7f01e80cad40 2023-08-25T09:59:46.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.943+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e814ddf0 con 0x7f01e80cad40 2023-08-25T09:59:46.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.943+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e814e3b0 con 0x7f01e80cad40 2023-08-25T09:59:46.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.945+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e804e420 con 0x7f01e80cad40 2023-08-25T09:59:46.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.945+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f01d8013470 con 0x7f01e80cad40 2023-08-25T09:59:46.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.946+0000 7f01cffff640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f01d406a080 0x7f01d406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:46.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.947+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f01d8088b20 con 0x7f01e80cad40 2023-08-25T09:59:46.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.947+0000 7f01e6ffd640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f01d406a080 0x7f01d406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:46.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.948+0000 7f01e6ffd640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f01d406a080 0x7f01d406c540 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f01dc036000 tx=0x7f01dc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:46.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:46.953+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f01d8053de0 con 0x7f01e80cad40 2023-08-25T09:59:47.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.182+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01e814eb50 con 0x7f01e80cad40 2023-08-25T09:59:47.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.183+0000 7f01cffff640 1 -- 172.21.15.71:0/2045873123 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f01d8057400 con 0x7f01e80cad40 2023-08-25T09:59:47.186 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:47.186 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:47.186 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:47.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f01d406a080 msgr2=0x7f01d406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:47.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f01d406a080 0x7f01d406c540 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f01dc036000 tx=0x7f01dc008030 comp rx=0 tx=0).stop 2023-08-25T09:59:47.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 msgr2=0x7f01e814a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:47.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 0x7f01e814a650 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009fd0 tx=0x7f01d800ed40 comp rx=0 tx=0).stop 2023-08-25T09:59:47.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 shutdown_connections 2023-08-25T09:59:47.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e8158d60 0x7f01e813df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:47.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f01d406a080 0x7f01d406c540 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:47.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8152fc0 0x7f01e8156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:47.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 --2- 172.21.15.71:0/2045873123 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01e80cad40 0x7f01e814a650 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:47.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 >> 172.21.15.71:0/2045873123 conn(0x7f01e806a3a0 msgr2=0x7f01e8151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:47.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 shutdown_connections 2023-08-25T09:59:47.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:47.186+0000 7f01ecc2c640 1 -- 172.21.15.71:0/2045873123 wait complete. 2023-08-25T09:59:47.205 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T09:59:47.206 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T09:59:49.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:49.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.448+0000 7f958168e640 1 Processor -- start 2023-08-25T09:59:49.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.448+0000 7f958168e640 1 -- start start 2023-08-25T09:59:49.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f958168e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 0x7f957c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f958168e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f957c0c3a40 0x7f957c06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f958168e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957c0c1dc0 con 0x7f957c1509d0 2023-08-25T09:59:49.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f958168e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f957c157a80 con 0x7f957c0c3a40 2023-08-25T09:59:49.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f958168e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957c06cf70 con 0x7f957c0c3660 2023-08-25T09:59:49.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f957b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f957c1509d0 0x7f957c150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57846/0 (socket says 172.21.15.71:57846) 2023-08-25T09:59:49.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f957b7fe640 1 -- 172.21.15.71:0/2705888651 learned_addr learned my addr 172.21.15.71:0/2705888651 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:49.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f957affd640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 0x7f957c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f957a7fc640 1 --2- 172.21.15.71:0/2705888651 >> v2:172.21.15.164:3300/0 conn(0x7f957c0c3a40 0x7f957c06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.450+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984659476 0 0) 0x7f957c0c1dc0 con 0x7f957c1509d0 2023-08-25T09:59:49.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9564003520 con 0x7f957c1509d0 2023-08-25T09:59:49.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410745333 0 0) 0x7f9564003520 con 0x7f957c1509d0 2023-08-25T09:59:49.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 msgr2=0x7f957c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:49.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 0x7f957c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 >> v2:172.21.15.164:3300/0 conn(0x7f957c0c3a40 msgr2=0x7f957c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 --2- 172.21.15.71:0/2705888651 >> v2:172.21.15.164:3300/0 conn(0x7f957c0c3a40 0x7f957c06ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c156e00 con 0x7f957c1509d0 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f95740019f0 con 0x7f957c1509d0 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 >> v1:172.21.15.71:6789/0 conn(0x7f957c1509d0 legacy=0x7f957c150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95640039e0 0x7f9564003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95640047c0 0x7f9564004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 0x7f9564005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c157a80 con 0x7f95640047c0 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c06cf70 con 0x7f95640039e0 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956400b9d0 con 0x7f957c0c3660 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.451+0000 7f957a7fc640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95640047c0 0x7f9564004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957b7fe640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 0x7f9564005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957affd640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95640039e0 0x7f9564003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957a7fc640 1 -- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 msgr2=0x7f9564005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:49.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957a7fc640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f957c0c3660 0x7f9564005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957a7fc640 1 -- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95640039e0 msgr2=0x7f9564003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:49.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957a7fc640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95640039e0 0x7f9564003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.452+0000 7f957a7fc640 1 -- 172.21.15.71:0/2705888651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564003520 con 0x7f95640047c0 2023-08-25T09:59:49.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.453+0000 7f957a7fc640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95640047c0 0x7f9564004c40 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9570009ac0 tx=0x7f95640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4023ecc8efc7cf server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.453+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957002c710 con 0x7f95640047c0 2023-08-25T09:59:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957002c870 con 0x7f95640047c0 2023-08-25T09:59:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f9579ffb640 1 -- 172.21.15.71:0/2705888651 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957002cc00 con 0x7f95640047c0 2023-08-25T09:59:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f958168e640 1 -- 172.21.15.71:0/2705888651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95640047c0 msgr2=0x7f9564004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f958168e640 1 --2- 172.21.15.71:0/2705888651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95640047c0 0x7f9564004c40 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9570009ac0 tx=0x7f95640057f0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f957b7fe640 1 -- 172.21.15.71:0/2705888651 reap_dead start 2023-08-25T09:59:49.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f958168e640 1 -- 172.21.15.71:0/2705888651 shutdown_connections 2023-08-25T09:59:49.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f958168e640 1 -- 172.21.15.71:0/2705888651 >> 172.21.15.71:0/2705888651 conn(0x7f957c0654f0 msgr2=0x7f957c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:49.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f958168e640 1 -- 172.21.15.71:0/2705888651 shutdown_connections 2023-08-25T09:59:49.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.454+0000 7f958168e640 1 -- 172.21.15.71:0/2705888651 wait complete. 2023-08-25T09:59:49.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.456+0000 7f958168e640 1 Processor -- start 2023-08-25T09:59:49.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.457+0000 7f958168e640 1 -- start start 2023-08-25T09:59:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f958168e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95640047c0 0x7f957c14d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f958168e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f957c149fd0 0x7f957c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f958168e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 0x7f957c06e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f958168e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c14d990 con 0x7f957c158360 2023-08-25T09:59:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f958168e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c1ee360 con 0x7f957c149fd0 2023-08-25T09:59:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f958168e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957c065f60 con 0x7f95640047c0 2023-08-25T09:59:49.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f957b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 0x7f957c06e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f957b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 0x7f957c06e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37920/0 (socket says 172.21.15.71:37920) 2023-08-25T09:59:49.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.458+0000 7f957b7fe640 1 -- 172.21.15.71:0/3545667892 learned_addr learned my addr 172.21.15.71:0/3545667892 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:49.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957affd640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95640047c0 0x7f957c14d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957a7fc640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f957c149fd0 0x7f957c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957b7fe640 1 -- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95640047c0 msgr2=0x7f957c14d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957b7fe640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95640047c0 0x7f957c14d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957b7fe640 1 -- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f957c149fd0 msgr2=0x7f957c157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957b7fe640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f957c149fd0 0x7f957c157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957b7fe640 1 -- 172.21.15.71:0/3545667892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c15abc0 con 0x7f957c158360 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f957b7fe640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 0x7f957c06e6d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f957c0663f0 tx=0x7f957400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957400ec10 con 0x7f957c158360 2023-08-25T09:59:49.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957400ed90 con 0x7f957c158360 2023-08-25T09:59:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c14d720 con 0x7f957c158360 2023-08-25T09:59:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.459+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9574014670 con 0x7f957c158360 2023-08-25T09:59:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.460+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c1eda30 con 0x7f957c158360 2023-08-25T09:59:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.461+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c069160 con 0x7f957c158360 2023-08-25T09:59:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.462+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f95740147d0 con 0x7f957c158360 2023-08-25T09:59:49.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.463+0000 7f95637fe640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f956806a050 0x7f956806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:49.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.463+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f95740888e0 con 0x7f957c158360 2023-08-25T09:59:49.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.463+0000 7f957affd640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f956806a050 0x7f956806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:49.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.464+0000 7f957affd640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f956806a050 0x7f956806c510 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f957c0c2160 tx=0x7f956c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:49.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.468+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9574053a70 con 0x7f957c158360 2023-08-25T09:59:49.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.733+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f957c154680 con 0x7f957c158360 2023-08-25T09:59:49.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.735+0000 7f95637fe640 1 -- 172.21.15.71:0/3545667892 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f9574057090 con 0x7f957c158360 2023-08-25T09:59:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:49.738 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:49.738 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:49.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.737+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f956806a050 msgr2=0x7f956806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.737+0000 7f958168e640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f956806a050 0x7f956806c510 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f957c0c2160 tx=0x7f956c008040 comp rx=0 tx=0).stop 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.737+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 msgr2=0x7f957c06e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.737+0000 7f958168e640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 0x7f957c06e6d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f957c0663f0 tx=0x7f957400e530 comp rx=0 tx=0).stop 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 shutdown_connections 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f957c149fd0 0x7f957c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f956806a050 0x7f956806c510 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f957c158360 0x7f957c06e6d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 --2- 172.21.15.71:0/3545667892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95640047c0 0x7f957c14d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:49.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 >> 172.21.15.71:0/3545667892 conn(0x7f957c0654f0 msgr2=0x7f957c154ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:49.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 shutdown_connections 2023-08-25T09:59:49.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:49.738+0000 7f958168e640 1 -- 172.21.15.71:0/3545667892 wait complete. 2023-08-25T09:59:49.760 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:51.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:51.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.994+0000 7f0101121640 1 Processor -- start 2023-08-25T09:59:51.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.995+0000 7f0101121640 1 -- start start 2023-08-25T09:59:51.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f0101121640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f00fc06d330 0x7f00fc0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f0101121640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc0c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f0101121640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f00fc152980 0x7f00fc152db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f0101121640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc15c1f0 con 0x7f00fc06d330 2023-08-25T09:59:52.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f0101121640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc15c490 con 0x7f00fc152980 2023-08-25T09:59:52.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f0101121640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc15c770 con 0x7f00fc06d710 2023-08-25T09:59:52.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f00fbfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f00fc06d330 0x7f00fc0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f00fbfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f00fc06d330 0x7f00fc0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50670/0 (socket says 172.21.15.71:50670) 2023-08-25T09:59:52.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.996+0000 7f00fbfff640 1 -- 172.21.15.71:0/1345087691 learned_addr learned my addr 172.21.15.71:0/1345087691 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:52.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f00fb7fe640 1 --2- 172.21.15.71:0/1345087691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc0c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f0100920640 1 --2- 172.21.15.71:0/1345087691 >> v2:172.21.15.164:3300/0 conn(0x7f00fc152980 0x7f00fc152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f00fbfff640 1 -- 172.21.15.71:0/1345087691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 msgr2=0x7f00fc0c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:52.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f00fbfff640 1 --2- 172.21.15.71:0/1345087691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc0c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f00fbfff640 1 -- 172.21.15.71:0/1345087691 >> v2:172.21.15.164:3300/0 conn(0x7f00fc152980 msgr2=0x7f00fc152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:52.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f00fbfff640 1 --2- 172.21.15.71:0/1345087691 >> v2:172.21.15.164:3300/0 conn(0x7f00fc152980 0x7f00fc152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f00fbfff640 1 -- 172.21.15.71:0/1345087691 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc15ca50 con 0x7f00fc06d330 2023-08-25T09:59:52.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.997+0000 7f0100920640 1 --2- 172.21.15.71:0/1345087691 >> v2:172.21.15.164:3300/0 conn(0x7f00fc152980 0x7f00fc152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:52.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.998+0000 7f00fbfff640 1 --2- 172.21.15.71:0/1345087691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc06d330 0x7f00fc0c5fb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0adb20 tx=0x7f00ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782d20b454c99531 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:52.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.998+0000 7f00faffd640 1 -- 172.21.15.71:0/1345087691 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec030860 con 0x7f00fc06d330 2023-08-25T09:59:52.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f00faffd640 1 -- 172.21.15.71:0/1345087691 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec0309c0 con 0x7f00fc06d330 2023-08-25T09:59:52.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f00faffd640 1 -- 172.21.15.71:0/1345087691 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec030d50 con 0x7f00fc06d330 2023-08-25T09:59:52.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 -- 172.21.15.71:0/1345087691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc06d330 msgr2=0x7f00fc0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 --2- 172.21.15.71:0/1345087691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc06d330 0x7f00fc0c5fb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0adb20 tx=0x7f00ec02b9a0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 -- 172.21.15.71:0/1345087691 shutdown_connections 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 --2- 172.21.15.71:0/1345087691 >> v2:172.21.15.164:3300/0 conn(0x7f00fc152980 0x7f00fc152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 --2- 172.21.15.71:0/1345087691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc0c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 --2- 172.21.15.71:0/1345087691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc06d330 0x7f00fc0c5fb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 -- 172.21.15.71:0/1345087691 >> 172.21.15.71:0/1345087691 conn(0x7f00fc0cb4b0 msgr2=0x7f00fc0cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:52.008 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 -- 172.21.15.71:0/1345087691 shutdown_connections 2023-08-25T09:59:52.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:51.999+0000 7f0101121640 1 -- 172.21.15.71:0/1345087691 wait complete. 2023-08-25T09:59:52.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.001+0000 7f0101121640 1 Processor -- start 2023-08-25T09:59:52.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.001+0000 7f0101121640 1 -- start start 2023-08-25T09:59:52.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0101121640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00fc06d330 0x7f00fc14ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0101121640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0101121640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 0x7f00fc14b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0101121640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1eeb50 con 0x7f00fc152980 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0101121640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1edfd0 con 0x7f00fc06d330 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0101121640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1ee2b0 con 0x7f00fc06d710 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0100920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 0x7f00fc14b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.002+0000 7f0100920640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 0x7f00fc14b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50676/0 (socket says 172.21.15.71:50676) 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 -- 172.21.15.71:0/774386234 learned_addr learned my addr 172.21.15.71:0/774386234 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:52.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f00fb7fe640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f00fbfff640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00fc06d330 0x7f00fc14ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 -- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 msgr2=0x7f00fc14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 -- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00fc06d330 msgr2=0x7f00fc14ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00fc06d330 0x7f00fc14ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 -- 172.21.15.71:0/774386234 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec0094d0 con 0x7f00fc152980 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.003+0000 7f0100920640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 0x7f00fc14b8d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f00f400cd50 tx=0x7f00f400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:52.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.004+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f400f680 con 0x7f00fc152980 2023-08-25T09:59:52.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.004+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f400f800 con 0x7f00fc152980 2023-08-25T09:59:52.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.004+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f400fbb0 con 0x7f00fc152980 2023-08-25T09:59:52.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.004+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc1ee590 con 0x7f00fc152980 2023-08-25T09:59:52.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.004+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc14b3d0 con 0x7f00fc152980 2023-08-25T09:59:52.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.005+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc14b230 con 0x7f00fc152980 2023-08-25T09:59:52.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.006+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f00f400b040 con 0x7f00fc152980 2023-08-25T09:59:52.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.007+0000 7f00f8ff9640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f00e806a050 0x7f00e806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:52.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.007+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f00f40893a0 con 0x7f00fc152980 2023-08-25T09:59:52.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.008+0000 7f00fbfff640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f00e806a050 0x7f00e806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:52.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.008+0000 7f00fbfff640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f00e806a050 0x7f00e806c510 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f00ec036000 tx=0x7f00ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:52.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.012+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f00fc14b230 con 0x7f00fc152980 2023-08-25T09:59:52.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.257+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00fc14b230 con 0x7f00fc152980 2023-08-25T09:59:52.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.258+0000 7f00f8ff9640 1 -- 172.21.15.71:0/774386234 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f00fc14b230 con 0x7f00fc152980 2023-08-25T09:59:52.261 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:52.261 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:52.262 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:52.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f00e806a050 msgr2=0x7f00e806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:52.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f00e806a050 0x7f00e806c510 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f00ec036000 tx=0x7f00ec008030 comp rx=0 tx=0).stop 2023-08-25T09:59:52.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 msgr2=0x7f00fc14b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:52.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 0x7f00fc14b8d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f00f400cd50 tx=0x7f00f400a650 comp rx=0 tx=0).stop 2023-08-25T09:59:52.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 shutdown_connections 2023-08-25T09:59:52.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f00e806a050 0x7f00e806c510 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f00fc152980 0x7f00fc14b8d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f00fc06d710 0x7f00fc14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 --2- 172.21.15.71:0/774386234 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00fc06d330 0x7f00fc14ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 >> 172.21.15.71:0/774386234 conn(0x7f00fc0cb4b0 msgr2=0x7f00fc1509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.260+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 shutdown_connections 2023-08-25T09:59:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:52.261+0000 7f0101121640 1 -- 172.21.15.71:0/774386234 wait complete. 2023-08-25T09:59:52.283 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:54.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:54.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.504+0000 7f025b0aa640 1 Processor -- start 2023-08-25T09:59:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.505+0000 7f025b0aa640 1 -- start start 2023-08-25T09:59:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f025b0aa640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f02540cb570 0x7f02540ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f025b0aa640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02540b1af0 con 0x7f025415cd00 2023-08-25T09:59:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f025b0aa640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02540b0fb0 con 0x7f02540cb190 2023-08-25T09:59:54.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f025b0aa640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02540c98f0 con 0x7f02540cb570 2023-08-25T09:59:54.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f025a8a9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f025415cd00 0x7f0254158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35844/0 (socket says 172.21.15.71:35844) 2023-08-25T09:59:54.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f025a8a9640 1 -- 172.21.15.71:0/932606764 learned_addr learned my addr 172.21.15.71:0/932606764 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T09:59:54.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.507+0000 7f02598a7640 1 --2- 172.21.15.71:0/932606764 >> v2:172.21.15.164:3300/0 conn(0x7f02540cb570 0x7f02540ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025981048 0 0) 0x7f02540b1af0 con 0x7f025415cd00 2023-08-25T09:59:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0240003520 con 0x7f025415cd00 2023-08-25T09:59:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368125227 0 0) 0x7f0240003520 con 0x7f025415cd00 2023-08-25T09:59:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 >> v2:172.21.15.164:3300/0 conn(0x7f02540cb570 msgr2=0x7f02540ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 --2- 172.21.15.71:0/932606764 >> v2:172.21.15.164:3300/0 conn(0x7f02540cb570 0x7f02540ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 >> v1:172.21.15.164:6789/0 conn(0x7f02540cb190 legacy=0x7f02540c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025415d130 con 0x7f025415cd00 2023-08-25T09:59:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f024c0019f0 con 0x7f025415cd00 2023-08-25T09:59:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 >> v1:172.21.15.71:6789/0 conn(0x7f025415cd00 legacy=0x7f0254158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02400039e0 0x7f0240003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02400047c0 0x7f0240004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0240005180 0x7f024000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02540c98f0 con 0x7f02400047c0 2023-08-25T09:59:54.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024000c090 con 0x7f0240005180 2023-08-25T09:59:54.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.508+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024000c250 con 0x7f02400039e0 2023-08-25T09:59:54.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f02598a7640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02400047c0 0x7f0240004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f025a0a8640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02400039e0 0x7f0240003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f02598a7640 1 -- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02400039e0 msgr2=0x7f0240003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f025a8a9640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0240005180 0x7f024000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f02598a7640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02400039e0 0x7f0240003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f02598a7640 1 -- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0240005180 msgr2=0x7f024000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f02598a7640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0240005180 0x7f024000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f02598a7640 1 -- 172.21.15.71:0/932606764 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240003520 con 0x7f02400047c0 2023-08-25T09:59:54.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f025a8a9640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0240005180 0x7f024000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T09:59:54.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.509+0000 7f025a8a9640 1 -- 172.21.15.71:0/932606764 reap_dead start 2023-08-25T09:59:54.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.511+0000 7f02598a7640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02400047c0 0x7f0240004c40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0244009a90 tx=0x7f02400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd58366f5f5d9a24 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:54.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.511+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024402c680 con 0x7f02400047c0 2023-08-25T09:59:54.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.511+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024402c7e0 con 0x7f02400047c0 2023-08-25T09:59:54.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.511+0000 7f02590a6640 1 -- 172.21.15.71:0/932606764 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024402cb70 con 0x7f02400047c0 2023-08-25T09:59:54.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.511+0000 7f025b0aa640 1 -- 172.21.15.71:0/932606764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02400047c0 msgr2=0x7f0240004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.512+0000 7f025b0aa640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02400047c0 0x7f0240004c40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0244009a90 tx=0x7f02400057f0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.512+0000 7f025b0aa640 1 -- 172.21.15.71:0/932606764 shutdown_connections 2023-08-25T09:59:54.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.512+0000 7f025b0aa640 1 --2- 172.21.15.71:0/932606764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02400047c0 0x7f0240004c40 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.512+0000 7f025b0aa640 1 -- 172.21.15.71:0/932606764 >> 172.21.15.71:0/932606764 conn(0x7f025406ae20 msgr2=0x7f025406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:54.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.512+0000 7f025b0aa640 1 -- 172.21.15.71:0/932606764 shutdown_connections 2023-08-25T09:59:54.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.512+0000 7f025b0aa640 1 -- 172.21.15.71:0/932606764 wait complete. 2023-08-25T09:59:54.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.514+0000 7f025b0aa640 1 Processor -- start 2023-08-25T09:59:54.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.514+0000 7f025b0aa640 1 -- start start 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.515+0000 7f025b0aa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02400047c0 0x7f0254154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.515+0000 7f025b0aa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 0x7f02541f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.515+0000 7f025b0aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02541584a0 0x7f02541f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.515+0000 7f025b0aa640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02541506d0 con 0x7f02540cc730 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.515+0000 7f025b0aa640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02540c8130 con 0x7f02400047c0 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.515+0000 7f025b0aa640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0254158010 con 0x7f02541584a0 2023-08-25T09:59:54.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 0x7f02541f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 0x7f02541f19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50698/0 (socket says 172.21.15.71:50698) 2023-08-25T09:59:54.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 -- 172.21.15.71:0/272024663 learned_addr learned my addr 172.21.15.71:0/272024663 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:54.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f025a0a8640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02400047c0 0x7f0254154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f025a8a9640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02541584a0 0x7f02541f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 -- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02541584a0 msgr2=0x7f02541f1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02541584a0 0x7f02541f1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 -- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02400047c0 msgr2=0x7f0254154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02400047c0 0x7f0254154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 -- 172.21.15.71:0/272024663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0254157d10 con 0x7f02540cc730 2023-08-25T09:59:54.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.516+0000 7f02598a7640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 0x7f02541f19c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0244037040 tx=0x7f0244004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:54.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.517+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244003810 con 0x7f02540cc730 2023-08-25T09:59:54.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.517+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0244003970 con 0x7f02540cc730 2023-08-25T09:59:54.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.517+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244003d00 con 0x7f02540cc730 2023-08-25T09:59:54.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.517+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02541569e0 con 0x7f02540cc730 2023-08-25T09:59:54.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.517+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02540c8380 con 0x7f02540cc730 2023-08-25T09:59:54.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.518+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025404e420 con 0x7f02540cc730 2023-08-25T09:59:54.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.519+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0244040450 con 0x7f02540cc730 2023-08-25T09:59:54.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.520+0000 7f024affd640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:54.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.520+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f02440aaa70 con 0x7f02540cc730 2023-08-25T09:59:54.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.520+0000 7f025a0a8640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:54.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.521+0000 7f025a0a8640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0250007fd0 tx=0x7f0250009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:54.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.524+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0244075c00 con 0x7f02540cc730 2023-08-25T09:59:54.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.806+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0254158150 con 0x7f02540cc730 2023-08-25T09:59:54.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.807+0000 7f024affd640 1 -- 172.21.15.71:0/272024663 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f0244003e60 con 0x7f02540cc730 2023-08-25T09:59:54.810 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:54.810 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:54.810 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:54.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 msgr2=0x7f023806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0250007fd0 tx=0x7f0250009040 comp rx=0 tx=0).stop 2023-08-25T09:59:54.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 msgr2=0x7f02541f19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:54.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 0x7f02541f19c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0244037040 tx=0x7f0244004730 comp rx=0 tx=0).stop 2023-08-25T09:59:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 shutdown_connections 2023-08-25T09:59:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02541584a0 0x7f02541f1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02540cc730 0x7f02541f19c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 --2- 172.21.15.71:0/272024663 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02400047c0 0x7f0254154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:54.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 >> 172.21.15.71:0/272024663 conn(0x7f025406ae20 msgr2=0x7f02540cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:54.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 shutdown_connections 2023-08-25T09:59:54.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:54.810+0000 7f025b0aa640 1 -- 172.21.15.71:0/272024663 wait complete. 2023-08-25T09:59:54.832 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:55.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:59:55.159 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T09:59:56.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:57.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.054+0000 7f73f67b5640 1 Processor -- start 2023-08-25T09:59:57.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.054+0000 7f73f67b5640 1 -- start start 2023-08-25T09:59:57.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.056+0000 7f73f67b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73f0152d80 0x7f73f00c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:57.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.056+0000 7f73f67b5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f73f00cacb0 0x7f73f00c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:57.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.056+0000 7f73f67b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f01591c0 con 0x7f73f015c740 2023-08-25T09:59:57.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.056+0000 7f73f67b5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f01594a0 con 0x7f73f00cacb0 2023-08-25T09:59:57.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.056+0000 7f73f67b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f0159780 con 0x7f73f0152d80 2023-08-25T09:59:57.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.056+0000 7f73e7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f73f00cacb0 0x7f73f00c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:57.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73e7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f73f00cacb0 0x7f73f00c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50714/0 (socket says 172.21.15.71:50714) 2023-08-25T09:59:57.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73e7fff640 1 -- 172.21.15.71:0/3825333166 learned_addr learned my addr 172.21.15.71:0/3825333166 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:57.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73effff640 1 --2- 172.21.15.71:0/3825333166 >> v2:172.21.15.164:3300/0 conn(0x7f73f0152d80 0x7f73f00c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:57.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73e7fff640 1 -- 172.21.15.71:0/3825333166 >> v2:172.21.15.164:3300/0 conn(0x7f73f0152d80 msgr2=0x7f73f00c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:57.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73e7fff640 1 --2- 172.21.15.71:0/3825333166 >> v2:172.21.15.164:3300/0 conn(0x7f73f0152d80 0x7f73f00c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73e7fff640 1 -- 172.21.15.71:0/3825333166 >> v1:172.21.15.164:6789/0 conn(0x7f73f015c740 legacy=0x7f73f015cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:57.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.057+0000 7f73e7fff640 1 -- 172.21.15.71:0/3825333166 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0159a60 con 0x7f73f00cacb0 2023-08-25T09:59:57.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.058+0000 7f73e7fff640 1 --2- 172.21.15.71:0/3825333166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f00c98a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f73f006bfa0 tx=0x7f73d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd4a155fa075bb6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:57.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73ef7fe640 1 -- 172.21.15.71:0/3825333166 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8030860 con 0x7f73f00cacb0 2023-08-25T09:59:57.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73ef7fe640 1 -- 172.21.15.71:0/3825333166 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d80309c0 con 0x7f73f00cacb0 2023-08-25T09:59:57.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73ef7fe640 1 -- 172.21.15.71:0/3825333166 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8030d50 con 0x7f73f00cacb0 2023-08-25T09:59:57.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 -- 172.21.15.71:0/3825333166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 msgr2=0x7f73f00c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:57.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 --2- 172.21.15.71:0/3825333166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f00c98a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f73f006bfa0 tx=0x7f73d802b9a0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 -- 172.21.15.71:0/3825333166 shutdown_connections 2023-08-25T09:59:57.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 --2- 172.21.15.71:0/3825333166 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f00c98a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 --2- 172.21.15.71:0/3825333166 >> v2:172.21.15.164:3300/0 conn(0x7f73f0152d80 0x7f73f00c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 -- 172.21.15.71:0/3825333166 >> 172.21.15.71:0/3825333166 conn(0x7f73f006b8f0 msgr2=0x7f73f00654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:57.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 -- 172.21.15.71:0/3825333166 shutdown_connections 2023-08-25T09:59:57.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.059+0000 7f73f67b5640 1 -- 172.21.15.71:0/3825333166 wait complete. 2023-08-25T09:59:57.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.061+0000 7f73f67b5640 1 Processor -- start 2023-08-25T09:59:57.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.062+0000 7f73f67b5640 1 -- start start 2023-08-25T09:59:57.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73f67b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f01f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:57.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73f67b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73f0152d80 0x7f73f00cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:57.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73f67b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73f015c740 0x7f73f0151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:57.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73f67b5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0146060 con 0x7f73f00cacb0 2023-08-25T09:59:57.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73f67b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f01f2f50 con 0x7f73f0152d80 2023-08-25T09:59:57.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73f67b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0150b80 con 0x7f73f015c740 2023-08-25T09:59:57.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f01f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:57.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f01f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50722/0 (socket says 172.21.15.71:50722) 2023-08-25T09:59:57.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 -- 172.21.15.71:0/2859401671 learned_addr learned my addr 172.21.15.71:0/2859401671 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:57.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 -- 172.21.15.71:0/2859401671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73f015c740 msgr2=0x7f73f0151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:57.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73f015c740 0x7f73f0151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 -- 172.21.15.71:0/2859401671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73f0152d80 msgr2=0x7f73f00cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:57.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73f0152d80 0x7f73f00cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.063+0000 7f73effff640 1 -- 172.21.15.71:0/2859401671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d80094d0 con 0x7f73f00cacb0 2023-08-25T09:59:57.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.064+0000 7f73effff640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f01f2a10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f73e000d830 tx=0x7f73e000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:57.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.064+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e000b660 con 0x7f73f00cacb0 2023-08-25T09:59:57.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.064+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e000b7c0 con 0x7f73f00cacb0 2023-08-25T09:59:57.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.064+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f01eef40 con 0x7f73f00cacb0 2023-08-25T09:59:57.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.064+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e000bb30 con 0x7f73f00cacb0 2023-08-25T09:59:57.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.064+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f01ef500 con 0x7f73f00cacb0 2023-08-25T09:59:57.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.066+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f00b1a60 con 0x7f73f00cacb0 2023-08-25T09:59:57.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.066+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f73e001e460 con 0x7f73f00cacb0 2023-08-25T09:59:57.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.067+0000 7f73ed7fa640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73d006a050 0x7f73d006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:57.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.068+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f73e0089230 con 0x7f73f00cacb0 2023-08-25T09:59:57.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.068+0000 7f73e7fff640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73d006a050 0x7f73d006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:57.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.068+0000 7f73e7fff640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73d006a050 0x7f73d006c510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f73d8036000 tx=0x7f73d803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:57.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.073+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f73e00543c0 con 0x7f73f00cacb0 2023-08-25T09:59:57.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.337+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73f0156a00 con 0x7f73f00cacb0 2023-08-25T09:59:57.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.338+0000 7f73ed7fa640 1 -- 172.21.15.71:0/2859401671 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f73e00579e0 con 0x7f73f00cacb0 2023-08-25T09:59:57.341 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:57.341 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:57.341 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:57.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73d006a050 msgr2=0x7f73d006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:57.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73d006a050 0x7f73d006c510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f73d8036000 tx=0x7f73d803b040 comp rx=0 tx=0).stop 2023-08-25T09:59:57.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 msgr2=0x7f73f01f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:57.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f01f2a10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f73e000d830 tx=0x7f73e000dcf0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.343 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 shutdown_connections 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73f0152d80 0x7f73f00cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f73d006a050 0x7f73d006c510 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73f015c740 0x7f73f0151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 --2- 172.21.15.71:0/2859401671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f73f00cacb0 0x7f73f01f2a10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 >> 172.21.15.71:0/2859401671 conn(0x7f73f006b8f0 msgr2=0x7f73f00c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 shutdown_connections 2023-08-25T09:59:57.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:57.340+0000 7f73f67b5640 1 -- 172.21.15.71:0/2859401671 wait complete. 2023-08-25T09:59:57.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T09:59:59.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T09:59:59.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.593+0000 7f69bf14f640 1 Processor -- start 2023-08-25T09:59:59.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.594+0000 7f69bf14f640 1 -- start start 2023-08-25T09:59:59.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69bf14f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69b81542d0 0x7f69b81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:59.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69bf14f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b8069850 0x7f69b80c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:59.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69bf14f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b8157bf0 con 0x7f69b80c75c0 2023-08-25T09:59:59.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69bf14f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80c4b60 con 0x7f69b81542d0 2023-08-25T09:59:59.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69bf14f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b806a130 con 0x7f69b8069850 2023-08-25T09:59:59.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69be14d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69b81542d0 0x7f69b81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:59.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69be14d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f69b81542d0 0x7f69b81576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50738/0 (socket says 172.21.15.71:50738) 2023-08-25T09:59:59.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.596+0000 7f69be14d640 1 -- 172.21.15.71:0/1501102448 learned_addr learned my addr 172.21.15.71:0/1501102448 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:59.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.597+0000 7f69bd94c640 1 --2- 172.21.15.71:0/1501102448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b8069850 0x7f69b80c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:59.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.597+0000 7f69be14d640 1 -- 172.21.15.71:0/1501102448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b8069850 msgr2=0x7f69b80c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T09:59:59.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.597+0000 7f69be14d640 1 --2- 172.21.15.71:0/1501102448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b8069850 0x7f69b80c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.597+0000 7f69be14d640 1 -- 172.21.15.71:0/1501102448 >> v1:172.21.15.164:6789/0 conn(0x7f69b80c75c0 legacy=0x7f69b80c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:59.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.597+0000 7f69be14d640 1 -- 172.21.15.71:0/1501102448 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b81546b0 con 0x7f69b81542d0 2023-08-25T09:59:59.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.598+0000 7f69be14d640 1 --2- 172.21.15.71:0/1501102448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b81542d0 0x7f69b81576b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009870 tx=0x7f69b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ac65a44fdc5f3d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:59.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.598+0000 7f69bd14b640 1 -- 172.21.15.71:0/1501102448 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4030840 con 0x7f69b81542d0 2023-08-25T09:59:59.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.598+0000 7f69bd14b640 1 -- 172.21.15.71:0/1501102448 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b40309a0 con 0x7f69b81542d0 2023-08-25T09:59:59.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bd14b640 1 -- 172.21.15.71:0/1501102448 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4030d30 con 0x7f69b81542d0 2023-08-25T09:59:59.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 -- 172.21.15.71:0/1501102448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b81542d0 msgr2=0x7f69b81576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:59.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 --2- 172.21.15.71:0/1501102448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b81542d0 0x7f69b81576b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009870 tx=0x7f69b402b980 comp rx=0 tx=0).stop 2023-08-25T09:59:59.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 -- 172.21.15.71:0/1501102448 shutdown_connections 2023-08-25T09:59:59.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 --2- 172.21.15.71:0/1501102448 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b8069850 0x7f69b80c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 --2- 172.21.15.71:0/1501102448 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b81542d0 0x7f69b81576b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 -- 172.21.15.71:0/1501102448 >> 172.21.15.71:0/1501102448 conn(0x7f69b806c280 msgr2=0x7f69b806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:59.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 -- 172.21.15.71:0/1501102448 shutdown_connections 2023-08-25T09:59:59.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.599+0000 7f69bf14f640 1 -- 172.21.15.71:0/1501102448 wait complete. 2023-08-25T09:59:59.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.601+0000 7f69bf14f640 1 Processor -- start 2023-08-25T09:59:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.601+0000 7f69bf14f640 1 -- start start 2023-08-25T09:59:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bf14f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b8069850 0x7f69b8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bf14f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 0x7f69b8150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bf14f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b81542d0 0x7f69b8155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bf14f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b81f1b20 con 0x7f69b80c75c0 2023-08-25T09:59:59.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bf14f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8154bd0 con 0x7f69b8069850 2023-08-25T09:59:59.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bf14f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8156df0 con 0x7f69b81542d0 2023-08-25T09:59:59.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.602+0000 7f69bd94c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 0x7f69b8150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:59.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 0x7f69b8150990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50742/0 (socket says 172.21.15.71:50742) 2023-08-25T09:59:59.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 -- 172.21.15.71:0/2342403236 learned_addr learned my addr 172.21.15.71:0/2342403236 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T09:59:59.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69be94e640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b81542d0 0x7f69b8155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:59.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69be14d640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b8069850 0x7f69b8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:59.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 -- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b81542d0 msgr2=0x7f69b8155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:59.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b81542d0 0x7f69b8155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 -- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b8069850 msgr2=0x7f69b8156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:59.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b8069850 0x7f69b8156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 -- 172.21.15.71:0/2342403236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40094d0 con 0x7f69b80c75c0 2023-08-25T09:59:59.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.603+0000 7f69bd94c640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 0x7f69b8150990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f69a800d800 tx=0x7f69a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:59.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.604+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a800b660 con 0x7f69b80c75c0 2023-08-25T09:59:59.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.604+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a800b7e0 con 0x7f69b80c75c0 2023-08-25T09:59:59.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.604+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a800bb90 con 0x7f69b80c75c0 2023-08-25T09:59:59.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.604+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b8157090 con 0x7f69b80c75c0 2023-08-25T09:59:59.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.604+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b8157650 con 0x7f69b80c75c0 2023-08-25T09:59:59.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.605+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b8154d80 con 0x7f69b80c75c0 2023-08-25T09:59:59.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.606+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f69a800bcf0 con 0x7f69b80c75c0 2023-08-25T09:59:59.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.607+0000 7f69aeffd640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f699c06a050 0x7f699c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T09:59:59.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.607+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f69a80890a0 con 0x7f69b80c75c0 2023-08-25T09:59:59.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.607+0000 7f69be14d640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f699c06a050 0x7f699c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T09:59:59.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.608+0000 7f69be14d640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f699c06a050 0x7f699c06c510 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f69b4036000 tx=0x7f69b403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T09:59:59.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.612+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f69b8154d80 con 0x7f69b80c75c0 2023-08-25T09:59:59.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.877+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69b8154d80 con 0x7f69b80c75c0 2023-08-25T09:59:59.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.878+0000 7f69aeffd640 1 -- 172.21.15.71:0/2342403236 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f69b8154d80 con 0x7f69b80c75c0 2023-08-25T09:59:59.881 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T09:59:59.882 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T09:59:59.882 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T09:59:59.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f699c06a050 msgr2=0x7f699c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:59.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f699c06a050 0x7f699c06c510 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f69b4036000 tx=0x7f69b403b040 comp rx=0 tx=0).stop 2023-08-25T09:59:59.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 msgr2=0x7f69b8150990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T09:59:59.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 0x7f69b8150990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f69a800d800 tx=0x7f69a800dcc0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 shutdown_connections 2023-08-25T09:59:59.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f699c06a050 0x7f699c06c510 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b81542d0 0x7f69b8155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69b80c75c0 0x7f69b8150990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 --2- 172.21.15.71:0/2342403236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b8069850 0x7f69b8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T09:59:59.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 >> 172.21.15.71:0/2342403236 conn(0x7f69b806c280 msgr2=0x7f69b806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T09:59:59.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 shutdown_connections 2023-08-25T09:59:59.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T09:59:59.881+0000 7f69bf14f640 1 -- 172.21.15.71:0/2342403236 wait complete. 2023-08-25T09:59:59.904 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T09:59:59.905 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:00:01.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:02.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.136+0000 7f7235b2f640 1 Processor -- start 2023-08-25T10:00:02.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.137+0000 7f7235b2f640 1 -- start start 2023-08-25T10:00:02.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.138+0000 7f7235b2f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 0x7f72300c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:02.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f7235b2f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f723015ce50 0x7f72300c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:02.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f7235b2f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72301591c0 con 0x7f72300cacb0 2023-08-25T10:00:02.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f7235b2f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72301594a0 con 0x7f723015ce50 2023-08-25T10:00:02.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f7235b2f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7230159780 con 0x7f7230152d80 2023-08-25T10:00:02.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f722f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 0x7f72300c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:02.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f722f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 0x7f72300c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:32844/0 (socket says 172.21.15.71:32844) 2023-08-25T10:00:02.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.139+0000 7f722f7fe640 1 -- 172.21.15.71:0/319920970 learned_addr learned my addr 172.21.15.71:0/319920970 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:02.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.140+0000 7f722ffff640 1 --2- 172.21.15.71:0/319920970 >> v2:172.21.15.164:3300/0 conn(0x7f723015ce50 0x7f72300c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:02.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.140+0000 7f722f7fe640 1 -- 172.21.15.71:0/319920970 >> v2:172.21.15.164:3300/0 conn(0x7f723015ce50 msgr2=0x7f72300c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:02.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.140+0000 7f722f7fe640 1 --2- 172.21.15.71:0/319920970 >> v2:172.21.15.164:3300/0 conn(0x7f723015ce50 0x7f72300c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.140+0000 7f722f7fe640 1 -- 172.21.15.71:0/319920970 >> v1:172.21.15.164:6789/0 conn(0x7f72300cacb0 legacy=0x7f723015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:02.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.140+0000 7f722f7fe640 1 -- 172.21.15.71:0/319920970 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230159a60 con 0x7f7230152d80 2023-08-25T10:00:02.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.142+0000 7f722f7fe640 1 --2- 172.21.15.71:0/319920970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 0x7f72300c9360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7220009870 tx=0x7f722002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0bf6c026b9df73f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:02.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.142+0000 7f722e7fc640 1 -- 172.21.15.71:0/319920970 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722002f840 con 0x7f7230152d80 2023-08-25T10:00:02.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.142+0000 7f722e7fc640 1 -- 172.21.15.71:0/319920970 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722002f9a0 con 0x7f7230152d80 2023-08-25T10:00:02.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.142+0000 7f722e7fc640 1 -- 172.21.15.71:0/319920970 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722002fd30 con 0x7f7230152d80 2023-08-25T10:00:02.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 -- 172.21.15.71:0/319920970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 msgr2=0x7f72300c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:02.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 --2- 172.21.15.71:0/319920970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 0x7f72300c9360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7220009870 tx=0x7f722002ba20 comp rx=0 tx=0).stop 2023-08-25T10:00:02.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 -- 172.21.15.71:0/319920970 shutdown_connections 2023-08-25T10:00:02.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 --2- 172.21.15.71:0/319920970 >> v2:172.21.15.164:3300/0 conn(0x7f723015ce50 0x7f72300c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 --2- 172.21.15.71:0/319920970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7230152d80 0x7f72300c9360 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 -- 172.21.15.71:0/319920970 >> 172.21.15.71:0/319920970 conn(0x7f723006b8f0 msgr2=0x7f72300654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:02.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 -- 172.21.15.71:0/319920970 shutdown_connections 2023-08-25T10:00:02.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.143+0000 7f7235b2f640 1 -- 172.21.15.71:0/319920970 wait complete. 2023-08-25T10:00:02.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.145+0000 7f7235b2f640 1 Processor -- start 2023-08-25T10:00:02.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.145+0000 7f7235b2f640 1 -- start start 2023-08-25T10:00:02.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.146+0000 7f7235b2f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300cacb0 0x7f72301eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:02.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.146+0000 7f7235b2f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 0x7f72300c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:02.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.146+0000 7f7235b2f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723015ce50 0x7f72300c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:02.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.146+0000 7f7235b2f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72301f2540 con 0x7f7230152d80 2023-08-25T10:00:02.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.146+0000 7f7235b2f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72301f2a30 con 0x7f723015ce50 2023-08-25T10:00:02.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.146+0000 7f7235b2f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72301ee760 con 0x7f72300cacb0 2023-08-25T10:00:02.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 0x7f72300c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:02.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 0x7f72300c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35276/0 (socket says 172.21.15.71:35276) 2023-08-25T10:00:02.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 -- 172.21.15.71:0/75855674 learned_addr learned my addr 172.21.15.71:0/75855674 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:02.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722f7fe640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300cacb0 0x7f72301eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:02.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722ffff640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723015ce50 0x7f72300c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:02.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 -- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300cacb0 msgr2=0x7f72301eef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:02.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300cacb0 0x7f72301eef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 -- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723015ce50 msgr2=0x7f72300c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:02.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723015ce50 0x7f72300c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.147+0000 7f722effd640 1 -- 172.21.15.71:0/75855674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200094d0 con 0x7f7230152d80 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.148+0000 7f722effd640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 0x7f72300c3eb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f722400c400 tx=0x7f722400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.148+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72240052d0 con 0x7f7230152d80 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.148+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7224005430 con 0x7f7230152d80 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.148+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72301521d0 con 0x7f7230152d80 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.148+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7224003830 con 0x7f7230152d80 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.148+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72301eea00 con 0x7f7230152d80 2023-08-25T10:00:02.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.149+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72300b1a60 con 0x7f7230152d80 2023-08-25T10:00:02.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.150+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7224003990 con 0x7f7230152d80 2023-08-25T10:00:02.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.152+0000 7f7217fff640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a050 0x7f721c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:02.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.152+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f72240880b0 con 0x7f7230152d80 2023-08-25T10:00:02.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.152+0000 7f722f7fe640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a050 0x7f721c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:02.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.153+0000 7f722f7fe640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a050 0x7f721c06c510 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7220036000 tx=0x7f7220008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:02.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.156+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7224053250 con 0x7f7230152d80 2023-08-25T10:00:02.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.414+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72301ee5c0 con 0x7f7230152d80 2023-08-25T10:00:02.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.415+0000 7f7217fff640 1 -- 172.21.15.71:0/75855674 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f72301ee5c0 con 0x7f7230152d80 2023-08-25T10:00:02.418 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:02.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:02.418 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a050 msgr2=0x7f721c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a050 0x7f721c06c510 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7220036000 tx=0x7f7220008030 comp rx=0 tx=0).stop 2023-08-25T10:00:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 msgr2=0x7f72300c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 0x7f72300c3eb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f722400c400 tx=0x7f722400c8c0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 shutdown_connections 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f723015ce50 0x7f72300c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f721c06a050 0x7f721c06c510 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7230152d80 0x7f72300c3eb0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 --2- 172.21.15.71:0/75855674 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72300cacb0 0x7f72301eef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 >> 172.21.15.71:0/75855674 conn(0x7f723006b8f0 msgr2=0x7f7230156ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 shutdown_connections 2023-08-25T10:00:02.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:02.418+0000 7f7235b2f640 1 -- 172.21.15.71:0/75855674 wait complete. 2023-08-25T10:00:02.440 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:04.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:04.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.682+0000 7f728a9d2640 1 Processor -- start 2023-08-25T10:00:04.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.682+0000 7f728a9d2640 1 -- start start 2023-08-25T10:00:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.684+0000 7f728a9d2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb190 0x7f72840ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.684+0000 7f728a9d2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f728415cd00 0x7f7284158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.684+0000 7f728a9d2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72840b1af0 con 0x7f72840cb570 2023-08-25T10:00:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.684+0000 7f728a9d2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72840c98f0 con 0x7f728415cd00 2023-08-25T10:00:04.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.684+0000 7f728a9d2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728415d190 con 0x7f72840cb190 2023-08-25T10:00:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.684+0000 7f728a1d1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f728415cd00 0x7f7284158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f728a1d1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f728415cd00 0x7f7284158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35284/0 (socket says 172.21.15.71:35284) 2023-08-25T10:00:04.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f728a1d1640 1 -- 172.21.15.71:0/2608891552 learned_addr learned my addr 172.21.15.71:0/2608891552 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f72899d0640 1 --2- 172.21.15.71:0/2608891552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb190 0x7f72840ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f728a1d1640 1 -- 172.21.15.71:0/2608891552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb190 msgr2=0x7f72840ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f728a1d1640 1 --2- 172.21.15.71:0/2608891552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb190 0x7f72840ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f728a1d1640 1 -- 172.21.15.71:0/2608891552 >> v1:172.21.15.164:6789/0 conn(0x7f72840cb570 legacy=0x7f72840c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.685+0000 7f728a1d1640 1 -- 172.21.15.71:0/2608891552 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72840b0fb0 con 0x7f728415cd00 2023-08-25T10:00:04.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.686+0000 7f728a1d1640 1 --2- 172.21.15.71:0/2608891552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f7284158d80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f727c009e50 tx=0x7f727c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cbf2f3e2481ef9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:04.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f72889ce640 1 -- 172.21.15.71:0/2608891552 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c02cb00 con 0x7f728415cd00 2023-08-25T10:00:04.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f72889ce640 1 -- 172.21.15.71:0/2608891552 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c02cc60 con 0x7f728415cd00 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f72889ce640 1 -- 172.21.15.71:0/2608891552 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c0343f0 con 0x7f728415cd00 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f728a9d2640 1 -- 172.21.15.71:0/2608891552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 msgr2=0x7f7284158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f728a9d2640 1 --2- 172.21.15.71:0/2608891552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f7284158d80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f727c009e50 tx=0x7f727c02bf60 comp rx=0 tx=0).stop 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f728a9d2640 1 -- 172.21.15.71:0/2608891552 shutdown_connections 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f728a9d2640 1 --2- 172.21.15.71:0/2608891552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f7284158d80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f728a9d2640 1 --2- 172.21.15.71:0/2608891552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb190 0x7f72840ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.687+0000 7f728a9d2640 1 -- 172.21.15.71:0/2608891552 >> 172.21.15.71:0/2608891552 conn(0x7f728406ae20 msgr2=0x7f728406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:04.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.688+0000 7f728a9d2640 1 -- 172.21.15.71:0/2608891552 shutdown_connections 2023-08-25T10:00:04.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.688+0000 7f728a9d2640 1 -- 172.21.15.71:0/2608891552 wait complete. 2023-08-25T10:00:04.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.690+0000 7f728a9d2640 1 Processor -- start 2023-08-25T10:00:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.690+0000 7f728a9d2640 1 -- start start 2023-08-25T10:00:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a9d2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72840cb190 0x7f7284159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a9d2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb570 0x7f72840ce850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a9d2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f72841ee820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a9d2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7284158560 con 0x7f728415cd00 2023-08-25T10:00:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a9d2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728414f250 con 0x7f72840cb190 2023-08-25T10:00:04.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a9d2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72841595b0 con 0x7f72840cb570 2023-08-25T10:00:04.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.691+0000 7f728a1d1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f72841ee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f72841ee820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35290/0 (socket says 172.21.15.71:35290) 2023-08-25T10:00:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 -- 172.21.15.71:0/1451736977 learned_addr learned my addr 172.21.15.71:0/1451736977 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f72899d0640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72840cb190 0x7f7284159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f72891cf640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb570 0x7f72840ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 -- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb570 msgr2=0x7f72840ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:04.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb570 0x7f72840ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 -- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72840cb190 msgr2=0x7f7284159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72840cb190 0x7f7284159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 -- 172.21.15.71:0/1451736977 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c009b30 con 0x7f728415cd00 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.692+0000 7f728a1d1640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f72841ee820 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f727c036040 tx=0x7f727c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.693+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c009510 con 0x7f728415cd00 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.693+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727c03e450 con 0x7f728415cd00 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.693+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72841ee320 con 0x7f728415cd00 2023-08-25T10:00:04.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.693+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727c03e800 con 0x7f728415cd00 2023-08-25T10:00:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.693+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72841f2340 con 0x7f728415cd00 2023-08-25T10:00:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.694+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72841ee540 con 0x7f728415cd00 2023-08-25T10:00:04.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.695+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f727c03e960 con 0x7f728415cd00 2023-08-25T10:00:04.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.696+0000 7f727a7fc640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f726806a050 0x7f726806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:04.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.696+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f727c0ab8a0 con 0x7f728415cd00 2023-08-25T10:00:04.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.697+0000 7f72899d0640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f726806a050 0x7f726806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:04.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.697+0000 7f72899d0640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f726806a050 0x7f726806c510 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7280006fd0 tx=0x7f7280008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:04.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.700+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f72841ee540 con 0x7f728415cd00 2023-08-25T10:00:04.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.967+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72841ee540 con 0x7f728415cd00 2023-08-25T10:00:04.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.968+0000 7f727a7fc640 1 -- 172.21.15.71:0/1451736977 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f72841ee540 con 0x7f728415cd00 2023-08-25T10:00:04.970 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:04.970 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:04.971 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:04.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.970+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f726806a050 msgr2=0x7f726806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:04.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.970+0000 7f728a9d2640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f726806a050 0x7f726806c510 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7280006fd0 tx=0x7f7280008040 comp rx=0 tx=0).stop 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.970+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 msgr2=0x7f72841ee820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.970+0000 7f728a9d2640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f72841ee820 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f727c036040 tx=0x7f727c004a40 comp rx=0 tx=0).stop 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 shutdown_connections 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f726806a050 0x7f726806c510 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f728415cd00 0x7f72841ee820 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72840cb570 0x7f72840ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 --2- 172.21.15.71:0/1451736977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72840cb190 0x7f7284159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:04.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 >> 172.21.15.71:0/1451736977 conn(0x7f728406ae20 msgr2=0x7f728415a840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:04.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 shutdown_connections 2023-08-25T10:00:04.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:04.971+0000 7f728a9d2640 1 -- 172.21.15.71:0/1451736977 wait complete. 2023-08-25T10:00:04.993 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:06.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:07.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.224+0000 7fc6532b0640 1 Processor -- start 2023-08-25T10:00:07.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.224+0000 7fc6532b0640 1 -- start start 2023-08-25T10:00:07.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.225+0000 7fc6532b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0cb190 0x7fc64c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:07.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.225+0000 7fc6532b0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc64c0cb570 0x7fc64c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:07.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc6532b0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0b1af0 con 0x7fc64c0c7f10 2023-08-25T10:00:07.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc6532b0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c0c98f0 con 0x7fc64c0cb570 2023-08-25T10:00:07.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc6532b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0b0fb0 con 0x7fc64c0cb190 2023-08-25T10:00:07.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc651aad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc64c0cb570 0x7fc64c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc651aad640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc64c0cb570 0x7fc64c15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35306/0 (socket says 172.21.15.71:35306) 2023-08-25T10:00:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc651aad640 1 -- 172.21.15.71:0/2633766532 learned_addr learned my addr 172.21.15.71:0/2633766532 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc652aaf640 1 --1- 172.21.15.71:0/2633766532 >> v1:172.21.15.71:6789/0 conn(0x7fc64c0c7f10 0x7fc64c158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41872/0 (socket says 172.21.15.71:41872) 2023-08-25T10:00:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc6522ae640 1 --2- 172.21.15.71:0/2633766532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0cb190 0x7fc64c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc651aad640 1 -- 172.21.15.71:0/2633766532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0cb190 msgr2=0x7fc64c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:07.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.226+0000 7fc651aad640 1 --2- 172.21.15.71:0/2633766532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0cb190 0x7fc64c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.227+0000 7fc651aad640 1 -- 172.21.15.71:0/2633766532 >> v1:172.21.15.71:6789/0 conn(0x7fc64c0c7f10 legacy=0x7fc64c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:07.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.227+0000 7fc651aad640 1 -- 172.21.15.71:0/2633766532 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c0b1530 con 0x7fc64c0cb570 2023-08-25T10:00:07.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.228+0000 7fc651aad640 1 --2- 172.21.15.71:0/2633766532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c15cd00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009870 tx=0x7fc63c02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c58608bcafb4aa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:07.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.228+0000 7fc6512ac640 1 -- 172.21.15.71:0/2633766532 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63c02f830 con 0x7fc64c0cb570 2023-08-25T10:00:07.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.228+0000 7fc6512ac640 1 -- 172.21.15.71:0/2633766532 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63c02f990 con 0x7fc64c0cb570 2023-08-25T10:00:07.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.228+0000 7fc6512ac640 1 -- 172.21.15.71:0/2633766532 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63c02fd20 con 0x7fc64c0cb570 2023-08-25T10:00:07.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 -- 172.21.15.71:0/2633766532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 msgr2=0x7fc64c15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:07.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 --2- 172.21.15.71:0/2633766532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c15cd00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009870 tx=0x7fc63c02ba10 comp rx=0 tx=0).stop 2023-08-25T10:00:07.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 -- 172.21.15.71:0/2633766532 shutdown_connections 2023-08-25T10:00:07.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 --2- 172.21.15.71:0/2633766532 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c15cd00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 --2- 172.21.15.71:0/2633766532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0cb190 0x7fc64c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 -- 172.21.15.71:0/2633766532 >> 172.21.15.71:0/2633766532 conn(0x7fc64c06ae20 msgr2=0x7fc64c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:07.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 -- 172.21.15.71:0/2633766532 shutdown_connections 2023-08-25T10:00:07.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.229+0000 7fc6532b0640 1 -- 172.21.15.71:0/2633766532 wait complete. 2023-08-25T10:00:07.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.231+0000 7fc6532b0640 1 Processor -- start 2023-08-25T10:00:07.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.231+0000 7fc6532b0640 1 -- start start 2023-08-25T10:00:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.232+0000 7fc6532b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0c7f10 0x7fc64c0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.232+0000 7fc6532b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc64c0cb190 0x7fc64c0b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.232+0000 7fc6532b0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c1498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.232+0000 7fc6532b0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c14aad0 con 0x7fc64c0cb570 2023-08-25T10:00:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.232+0000 7fc6532b0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c14b690 con 0x7fc64c0cb190 2023-08-25T10:00:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.232+0000 7fc6532b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c149410 con 0x7fc64c0c7f10 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c1498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c1498a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35310/0 (socket says 172.21.15.71:35310) 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 -- 172.21.15.71:0/1522819393 learned_addr learned my addr 172.21.15.71:0/1522819393 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc6522ae640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0c7f10 0x7fc64c0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc651aad640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc64c0cb190 0x7fc64c0b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 -- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0c7f10 msgr2=0x7fc64c0ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:07.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0c7f10 0x7fc64c0ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 -- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc64c0cb190 msgr2=0x7fc64c0b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc64c0cb190 0x7fc64c0b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.233+0000 7fc652aaf640 1 -- 172.21.15.71:0/1522819393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c0094d0 con 0x7fc64c0cb570 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.234+0000 7fc652aaf640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c1498a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c780 tx=0x7fc64400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.234+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400ce90 con 0x7fc64c0cb570 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.234+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64400a5e0 con 0x7fc64c0cb570 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.234+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64400a950 con 0x7fc64c0cb570 2023-08-25T10:00:07.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.234+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c1f14f0 con 0x7fc64c0cb570 2023-08-25T10:00:07.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.234+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c1f1a90 con 0x7fc64c0cb570 2023-08-25T10:00:07.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.235+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c04e420 con 0x7fc64c0cb570 2023-08-25T10:00:07.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.236+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc64400aab0 con 0x7fc64c0cb570 2023-08-25T10:00:07.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.237+0000 7fc642ffd640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc63006a050 0x7fc63006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:07.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.237+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc644088260 con 0x7fc64c0cb570 2023-08-25T10:00:07.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.238+0000 7fc6522ae640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc63006a050 0x7fc63006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:07.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.238+0000 7fc6522ae640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc63006a050 0x7fc63006c510 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc648006fd0 tx=0x7fc648008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:07.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.244+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc644053410 con 0x7fc64c0cb570 2023-08-25T10:00:07.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.498+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc64c149550 con 0x7fc64c0cb570 2023-08-25T10:00:07.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.499+0000 7fc642ffd640 1 -- 172.21.15.71:0/1522819393 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fc644017090 con 0x7fc64c0cb570 2023-08-25T10:00:07.501 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:07.501 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:07.502 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:07.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc63006a050 msgr2=0x7fc63006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:07.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc63006a050 0x7fc63006c510 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc648006fd0 tx=0x7fc648008040 comp rx=0 tx=0).stop 2023-08-25T10:00:07.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 msgr2=0x7fc64c1498a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c1498a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c780 tx=0x7fc64400cc40 comp rx=0 tx=0).stop 2023-08-25T10:00:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 shutdown_connections 2023-08-25T10:00:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc64c0cb190 0x7fc64c0b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc63006a050 0x7fc63006c510 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc64c0cb570 0x7fc64c1498a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 --2- 172.21.15.71:0/1522819393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc64c0c7f10 0x7fc64c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 >> 172.21.15.71:0/1522819393 conn(0x7fc64c06ae20 msgr2=0x7fc64c0c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:07.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.502+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 shutdown_connections 2023-08-25T10:00:07.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:07.503+0000 7fc6532b0640 1 -- 172.21.15.71:0/1522819393 wait complete. 2023-08-25T10:00:07.523 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:09.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:09.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.755+0000 7f1da0a8a640 1 Processor -- start 2023-08-25T10:00:09.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.755+0000 7f1da0a8a640 1 -- start start 2023-08-25T10:00:09.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.757+0000 7f1da0a8a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1d9c1530b0 0x7f1d9c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.757+0000 7f1da0a8a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1d9c071960 0x7f1d9c155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.757+0000 7f1da0a8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c06ab10 0x7f1d9c06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.757+0000 7f1da0a8a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c06b650 con 0x7f1d9c071960 2023-08-25T10:00:09.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.757+0000 7f1da0a8a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9c0c18a0 con 0x7f1d9c1530b0 2023-08-25T10:00:09.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.757+0000 7f1da0a8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c159ed0 con 0x7f1d9c06ab10 2023-08-25T10:00:09.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1d9c071960 0x7f1d9c155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:09.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1d9c071960 0x7f1d9c155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35312/0 (socket says 172.21.15.71:35312) 2023-08-25T10:00:09.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 -- 172.21.15.71:0/3542272701 learned_addr learned my addr 172.21.15.71:0/3542272701 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:09.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9b7fe640 1 --2- 172.21.15.71:0/3542272701 >> v2:172.21.15.164:3300/0 conn(0x7f1d9c1530b0 0x7f1d9c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:09.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 -- 172.21.15.71:0/3542272701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c06ab10 msgr2=0x7f1d9c06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:09.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 --2- 172.21.15.71:0/3542272701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c06ab10 0x7f1d9c06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 -- 172.21.15.71:0/3542272701 >> v2:172.21.15.164:3300/0 conn(0x7f1d9c1530b0 msgr2=0x7f1d9c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:09.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 --2- 172.21.15.71:0/3542272701 >> v2:172.21.15.164:3300/0 conn(0x7f1d9c1530b0 0x7f1d9c155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.758+0000 7f1d9affd640 1 -- 172.21.15.71:0/3542272701 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c0c7b70 con 0x7f1d9c071960 2023-08-25T10:00:09.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.759+0000 7f1d9affd640 1 --2- 172.21.15.71:0/3542272701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c155f40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0cc270 tx=0x7f1d9002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87c443eeadbc83b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:09.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.760+0000 7f1d9a7fc640 1 -- 172.21.15.71:0/3542272701 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d900317c0 con 0x7f1d9c071960 2023-08-25T10:00:09.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.760+0000 7f1d9a7fc640 1 -- 172.21.15.71:0/3542272701 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d90031920 con 0x7f1d9c071960 2023-08-25T10:00:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.760+0000 7f1d9a7fc640 1 -- 172.21.15.71:0/3542272701 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90031cb0 con 0x7f1d9c071960 2023-08-25T10:00:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.760+0000 7f1da0a8a640 1 -- 172.21.15.71:0/3542272701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 msgr2=0x7f1d9c155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.760+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/3542272701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c155f40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0cc270 tx=0x7f1d9002b860 comp rx=0 tx=0).stop 2023-08-25T10:00:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 -- 172.21.15.71:0/3542272701 shutdown_connections 2023-08-25T10:00:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/3542272701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c06ab10 0x7f1d9c06bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/3542272701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c155f40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/3542272701 >> v2:172.21.15.164:3300/0 conn(0x7f1d9c1530b0 0x7f1d9c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 -- 172.21.15.71:0/3542272701 >> 172.21.15.71:0/3542272701 conn(0x7f1d9c0cb4b0 msgr2=0x7f1d9c0cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:09.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 -- 172.21.15.71:0/3542272701 shutdown_connections 2023-08-25T10:00:09.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.761+0000 7f1da0a8a640 1 -- 172.21.15.71:0/3542272701 wait complete. 2023-08-25T10:00:09.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.762+0000 7f1da0a8a640 1 Processor -- start 2023-08-25T10:00:09.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.763+0000 7f1da0a8a640 1 -- start start 2023-08-25T10:00:09.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1da0a8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d9c06ab10 0x7f1d9c161750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1da0a8a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c1eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1da0a8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c1530b0 0x7f1d9c0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1da0a8a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1600f0 con 0x7f1d9c071960 2023-08-25T10:00:09.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1da0a8a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1f18c0 con 0x7f1d9c06ab10 2023-08-25T10:00:09.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1da0a8a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c149d70 con 0x7f1d9c1530b0 2023-08-25T10:00:09.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1d9affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c1eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:09.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1d9affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c1eda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35320/0 (socket says 172.21.15.71:35320) 2023-08-25T10:00:09.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1d9affd640 1 -- 172.21.15.71:0/2737954976 learned_addr learned my addr 172.21.15.71:0/2737954976 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:09.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1d9bfff640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c1530b0 0x7f1d9c0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:09.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.764+0000 7f1d9b7fe640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d9c06ab10 0x7f1d9c161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:09.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d9affd640 1 -- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c1530b0 msgr2=0x7f1d9c0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:09.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d9affd640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c1530b0 0x7f1d9c0c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d9affd640 1 -- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d9c06ab10 msgr2=0x7f1d9c161750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:09.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d9affd640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d9c06ab10 0x7f1d9c161750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:09.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d9affd640 1 -- 172.21.15.71:0/2737954976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d900094d0 con 0x7f1d9c071960 2023-08-25T10:00:09.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d9affd640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c1eda70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1d9002b8a0 tx=0x7f1d900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:09.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90032960 con 0x7f1d9c071960 2023-08-25T10:00:09.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d90032ae0 con 0x7f1d9c071960 2023-08-25T10:00:09.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.765+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90032e90 con 0x7f1d9c071960 2023-08-25T10:00:09.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.766+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c161c90 con 0x7f1d9c071960 2023-08-25T10:00:09.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.766+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c1eea40 con 0x7f1d9c071960 2023-08-25T10:00:09.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.767+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c1f1a00 con 0x7f1d9c071960 2023-08-25T10:00:09.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.768+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1d9004e450 con 0x7f1d9c071960 2023-08-25T10:00:09.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.769+0000 7f1d83fff640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d8806a050 0x7f1d8806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:09.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.769+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f1d900aaf40 con 0x7f1d9c071960 2023-08-25T10:00:09.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.769+0000 7f1d9b7fe640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d8806a050 0x7f1d8806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:09.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.770+0000 7f1d9b7fe640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d8806a050 0x7f1d8806c510 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c006fd0 tx=0x7f1d8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:09.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:09.774+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1d9c1f1a00 con 0x7f1d9c071960 2023-08-25T10:00:10.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.034+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d9c1f1a00 con 0x7f1d9c071960 2023-08-25T10:00:10.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.036+0000 7f1d83fff640 1 -- 172.21.15.71:0/2737954976 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f1d9c1f1a00 con 0x7f1d9c071960 2023-08-25T10:00:10.038 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:10.038 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:10.038 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:10.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d8806a050 msgr2=0x7f1d8806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:10.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d8806a050 0x7f1d8806c510 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c006fd0 tx=0x7f1d8c008040 comp rx=0 tx=0).stop 2023-08-25T10:00:10.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 msgr2=0x7f1d9c1eda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:10.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c1eda70 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1d9002b8a0 tx=0x7f1d900046c0 comp rx=0 tx=0).stop 2023-08-25T10:00:10.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 shutdown_connections 2023-08-25T10:00:10.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1d8806a050 0x7f1d8806c510 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:10.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1d9c1530b0 0x7f1d9c0c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:10.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1d9c071960 0x7f1d9c1eda70 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:10.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 --2- 172.21.15.71:0/2737954976 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1d9c06ab10 0x7f1d9c161750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:10.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 >> 172.21.15.71:0/2737954976 conn(0x7f1d9c0cb4b0 msgr2=0x7f1d9c0cb890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:10.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 shutdown_connections 2023-08-25T10:00:10.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:10.039+0000 7f1da0a8a640 1 -- 172.21.15.71:0/2737954976 wait complete. 2023-08-25T10:00:10.065 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:12.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:12.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.293+0000 7fb6cf59e640 1 Processor -- start 2023-08-25T10:00:12.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.293+0000 7fb6cf59e640 1 -- start start 2023-08-25T10:00:12.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.295+0000 7fb6cf59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb6d00cb570 0x7fb6d00ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.295+0000 7fb6cf59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 0x7fb6d0158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.295+0000 7fb6cf59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d00b1af0 con 0x7fb6d00cb190 2023-08-25T10:00:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.295+0000 7fb6cf59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d00c98f0 con 0x7fb6d00cb570 2023-08-25T10:00:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.295+0000 7fb6cf59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d015d170 con 0x7fb6d015cd00 2023-08-25T10:00:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.296+0000 7fb6ce59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb6d00cb190 0x7fb6d00c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50606/0 (socket says 172.21.15.71:50606) 2023-08-25T10:00:12.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.296+0000 7fb6ce59c640 1 -- 172.21.15.71:0/2778329521 learned_addr learned my addr 172.21.15.71:0/2778329521 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:12.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.296+0000 7fb6ced9d640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 0x7fb6d0158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.296+0000 7fb6cdd9b640 1 --2- 172.21.15.71:0/2778329521 >> v2:172.21.15.164:3300/0 conn(0x7fb6d00cb570 0x7fb6d00ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.296+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273830327 0 0) 0x7fb6d00b1af0 con 0x7fb6d00cb190 2023-08-25T10:00:12.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.296+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b4003520 con 0x7fb6d00cb190 2023-08-25T10:00:12.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264661139 0 0) 0x7fb6b4003520 con 0x7fb6d00cb190 2023-08-25T10:00:12.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 msgr2=0x7fb6d0158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:12.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 0x7fb6d0158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 >> v2:172.21.15.164:3300/0 conn(0x7fb6d00cb570 msgr2=0x7fb6d00ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 --2- 172.21.15.71:0/2778329521 >> v2:172.21.15.164:3300/0 conn(0x7fb6d00cb570 0x7fb6d00ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d00b0fb0 con 0x7fb6d00cb190 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fb6c0001670 con 0x7fb6d00cb190 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 >> v1:172.21.15.71:6789/0 conn(0x7fb6d00cb190 legacy=0x7fb6d00c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 0x7fb6b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6b4004d00 0x7fb6b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d00c98f0 con 0x7fb6b40039e0 2023-08-25T10:00:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d015d170 con 0x7fb6b4004d00 2023-08-25T10:00:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b400ba00 con 0x7fb6d015cd00 2023-08-25T10:00:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.297+0000 7fb6cdd9b640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 0x7fb6b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ce59c640 1 -- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 msgr2=0x7fb6b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d015cd00 0x7fb6b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ce59c640 1 -- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6b4004d00 msgr2=0x7fb6b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ced9d640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6b4004d00 0x7fb6b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6b4004d00 0x7fb6b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ce59c640 1 -- 172.21.15.71:0/2778329521 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b4003520 con 0x7fb6b40039e0 2023-08-25T10:00:12.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.298+0000 7fb6ced9d640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6b4004d00 0x7fb6b4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:00:12.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.299+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6b4003e00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb6c000c240 tx=0x7fb6b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946fbafe14fe6b39 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.299+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c002cb00 con 0x7fb6b40039e0 2023-08-25T10:00:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.299+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c002cc60 con 0x7fb6b40039e0 2023-08-25T10:00:12.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.299+0000 7fb6cd59a640 1 -- 172.21.15.71:0/2778329521 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c0005af0 con 0x7fb6b40039e0 2023-08-25T10:00:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6cf59e640 1 -- 172.21.15.71:0/2778329521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 msgr2=0x7fb6b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/2778329521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6b4003e00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fb6c000c240 tx=0x7fb6b40057f0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6ced9d640 1 -- 172.21.15.71:0/2778329521 reap_dead start 2023-08-25T10:00:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6cf59e640 1 -- 172.21.15.71:0/2778329521 shutdown_connections 2023-08-25T10:00:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6cf59e640 1 -- 172.21.15.71:0/2778329521 >> 172.21.15.71:0/2778329521 conn(0x7fb6d006ae20 msgr2=0x7fb6d006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6cf59e640 1 -- 172.21.15.71:0/2778329521 shutdown_connections 2023-08-25T10:00:12.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.300+0000 7fb6cf59e640 1 -- 172.21.15.71:0/2778329521 wait complete. 2023-08-25T10:00:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.302+0000 7fb6cf59e640 1 Processor -- start 2023-08-25T10:00:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.302+0000 7fb6cf59e640 1 -- start start 2023-08-25T10:00:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6cf59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6d0157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6cf59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d015baf0 0x7fb6d0157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6cf59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d0155d40 0x7fb6d01561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6cf59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01538f0 con 0x7fb6b40039e0 2023-08-25T10:00:12.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6cf59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d0159c60 con 0x7fb6d015baf0 2023-08-25T10:00:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6cf59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d01566e0 con 0x7fb6d0155d40 2023-08-25T10:00:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6ce59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6d0157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6ce59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6d0157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35372/0 (socket says 172.21.15.71:35372) 2023-08-25T10:00:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.303+0000 7fb6ce59c640 1 -- 172.21.15.71:0/1170587984 learned_addr learned my addr 172.21.15.71:0/1170587984 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ced9d640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d0155d40 0x7fb6d01561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6cdd9b640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d015baf0 0x7fb6d0157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ce59c640 1 -- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d0155d40 msgr2=0x7fb6d01561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d0155d40 0x7fb6d01561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ce59c640 1 -- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d015baf0 msgr2=0x7fb6d0157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:12.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d015baf0 0x7fb6d0157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ce59c640 1 -- 172.21.15.71:0/1170587984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0156d70 con 0x7fb6b40039e0 2023-08-25T10:00:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6ce59c640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6d0157790 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb6c002c4a0 tx=0x7fb6c0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c0039070 con 0x7fb6b40039e0 2023-08-25T10:00:12.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6c0004bf0 con 0x7fb6b40039e0 2023-08-25T10:00:12.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.304+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c003c040 con 0x7fb6b40039e0 2023-08-25T10:00:12.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.305+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d0157030 con 0x7fb6b40039e0 2023-08-25T10:00:12.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.305+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d01598f0 con 0x7fb6b40039e0 2023-08-25T10:00:12.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.306+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d0159a80 con 0x7fb6b40039e0 2023-08-25T10:00:12.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.307+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb6c0032070 con 0x7fb6b40039e0 2023-08-25T10:00:12.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.308+0000 7fb6c6ffd640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb6b806a050 0x7fb6b806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:12.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.308+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb6c00a9a70 con 0x7fb6b40039e0 2023-08-25T10:00:12.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.308+0000 7fb6cdd9b640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb6b806a050 0x7fb6b806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:12.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.309+0000 7fb6cdd9b640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb6b806a050 0x7fb6b806c510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0152d00 tx=0x7fb6bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:12.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.313+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb6c0074d10 con 0x7fb6b40039e0 2023-08-25T10:00:12.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.581+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6d004e420 con 0x7fb6b40039e0 2023-08-25T10:00:12.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.582+0000 7fb6c6ffd640 1 -- 172.21.15.71:0/1170587984 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fb6c0078330 con 0x7fb6b40039e0 2023-08-25T10:00:12.584 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:12.584 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:12.585 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:12.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb6b806a050 msgr2=0x7fb6b806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:12.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb6b806a050 0x7fb6b806c510 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0152d00 tx=0x7fb6bc008040 comp rx=0 tx=0).stop 2023-08-25T10:00:12.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 msgr2=0x7fb6d0157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:12.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6d0157790 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fb6c002c4a0 tx=0x7fb6c0001a00 comp rx=0 tx=0).stop 2023-08-25T10:00:12.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 shutdown_connections 2023-08-25T10:00:12.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d015baf0 0x7fb6d0157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb6b806a050 0x7fb6b806c510 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d0155d40 0x7fb6d01561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 --2- 172.21.15.71:0/1170587984 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6b40039e0 0x7fb6d0157790 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:12.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.584+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 >> 172.21.15.71:0/1170587984 conn(0x7fb6d006ae20 msgr2=0x7fb6d00b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:12.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.585+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 shutdown_connections 2023-08-25T10:00:12.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:12.585+0000 7fb6cf59e640 1 -- 172.21.15.71:0/1170587984 wait complete. 2023-08-25T10:00:12.605 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:00:12.606 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:00:14.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:14.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.834+0000 7f9321381640 1 Processor -- start 2023-08-25T10:00:14.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.835+0000 7f9321381640 1 -- start start 2023-08-25T10:00:14.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f9321381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb570 0x7f931c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:14.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f9321381640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f931c15cd00 0x7f931c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:14.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f9321381640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c0b1af0 con 0x7f931c0cb190 2023-08-25T10:00:14.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f9321381640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0c98f0 con 0x7f931c15cd00 2023-08-25T10:00:14.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f9321381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c15d190 con 0x7f931c0cb570 2023-08-25T10:00:14.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f9320b80640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f931c15cd00 0x7f931c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:14.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f931bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f931c0cb190 0x7f931c0c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50612/0 (socket says 172.21.15.71:50612) 2023-08-25T10:00:14.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.836+0000 7f931bfff640 1 -- 172.21.15.71:0/1018565518 learned_addr learned my addr 172.21.15.71:0/1018565518 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.837+0000 7f931b7fe640 1 --2- 172.21.15.71:0/1018565518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb570 0x7f931c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.837+0000 7f9320b80640 1 -- 172.21.15.71:0/1018565518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb570 msgr2=0x7f931c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.837+0000 7f9320b80640 1 --2- 172.21.15.71:0/1018565518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb570 0x7f931c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.837+0000 7f9320b80640 1 -- 172.21.15.71:0/1018565518 >> v1:172.21.15.71:6789/0 conn(0x7f931c0cb190 legacy=0x7f931c0c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:14.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.837+0000 7f9320b80640 1 -- 172.21.15.71:0/1018565518 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0b0fb0 con 0x7f931c15cd00 2023-08-25T10:00:14.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.837+0000 7f931affd640 1 -- 172.21.15.71:0/1018565518 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726108691 0 0) 0x7f931c0b1af0 con 0x7f931c0cb190 2023-08-25T10:00:14.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.838+0000 7f9320b80640 1 --2- 172.21.15.71:0/1018565518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c158d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9314009d90 tx=0x7f931402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab99550810d86985 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:14.841 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.838+0000 7f931affd640 1 -- 172.21.15.71:0/1018565518 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931402cce0 con 0x7f931c15cd00 2023-08-25T10:00:14.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.838+0000 7f931affd640 1 -- 172.21.15.71:0/1018565518 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931402ce40 con 0x7f931c15cd00 2023-08-25T10:00:14.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.838+0000 7f931affd640 1 -- 172.21.15.71:0/1018565518 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314034660 con 0x7f931c15cd00 2023-08-25T10:00:14.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 -- 172.21.15.71:0/1018565518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 msgr2=0x7f931c158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:14.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 --2- 172.21.15.71:0/1018565518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c158d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f9314009d90 tx=0x7f931402bf30 comp rx=0 tx=0).stop 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 -- 172.21.15.71:0/1018565518 shutdown_connections 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 --2- 172.21.15.71:0/1018565518 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c158d80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 --2- 172.21.15.71:0/1018565518 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb570 0x7f931c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 -- 172.21.15.71:0/1018565518 >> 172.21.15.71:0/1018565518 conn(0x7f931c06ae20 msgr2=0x7f931c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 -- 172.21.15.71:0/1018565518 shutdown_connections 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.839+0000 7f9321381640 1 -- 172.21.15.71:0/1018565518 wait complete. 2023-08-25T10:00:14.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.841+0000 7f9321381640 1 Processor -- start 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.841+0000 7f9321381640 1 -- start start 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.842+0000 7f9321381640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb190 0x7f931c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.842+0000 7f9321381640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f931c0cb570 0x7f931c14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.842+0000 7f9321381640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.842+0000 7f9321381640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c157760 con 0x7f931c15cd00 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.842+0000 7f9321381640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c14e220 con 0x7f931c0cb570 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.842+0000 7f9321381640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c153b70 con 0x7f931c0cb190 2023-08-25T10:00:14.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:14.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c14dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35396/0 (socket says 172.21.15.71:35396) 2023-08-25T10:00:14.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 -- 172.21.15.71:0/2348235516 learned_addr learned my addr 172.21.15.71:0/2348235516 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:14.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f931bfff640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb190 0x7f931c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:14.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f931b7fe640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f931c0cb570 0x7f931c14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:14.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 -- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb190 msgr2=0x7f931c158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:14.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb190 0x7f931c158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:14.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 -- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f931c0cb570 msgr2=0x7f931c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:14.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f931c0cb570 0x7f931c14d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:14.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.843+0000 7f9320b80640 1 -- 172.21.15.71:0/2348235516 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314009a70 con 0x7f931c15cd00 2023-08-25T10:00:14.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.844+0000 7f9320b80640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c14dce0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9314009d90 tx=0x7f9314004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:14.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.844+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931403d600 con 0x7f931c15cd00 2023-08-25T10:00:14.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.844+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931403d780 con 0x7f931c15cd00 2023-08-25T10:00:14.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.844+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931403db30 con 0x7f931c15cd00 2023-08-25T10:00:14.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.844+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c153890 con 0x7f931c15cd00 2023-08-25T10:00:14.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.844+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0ce1f0 con 0x7f931c15cd00 2023-08-25T10:00:14.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.845+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c04e420 con 0x7f931c15cd00 2023-08-25T10:00:14.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.846+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f931403b040 con 0x7f931c15cd00 2023-08-25T10:00:14.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.848+0000 7f9318ff9640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f930806a050 0x7f930806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:14.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.848+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f93140ab0e0 con 0x7f931c15cd00 2023-08-25T10:00:14.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.848+0000 7f931bfff640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f930806a050 0x7f930806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:14.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.849+0000 7f931bfff640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f930806a050 0x7f930806c510 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f930c007fd0 tx=0x7f930c007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:14.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:14.853+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9314076290 con 0x7f931c15cd00 2023-08-25T10:00:15.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.127+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f931c0c9df0 con 0x7f931c15cd00 2023-08-25T10:00:15.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.129+0000 7f9318ff9640 1 -- 172.21.15.71:0/2348235516 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f931404e090 con 0x7f931c15cd00 2023-08-25T10:00:15.131 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:15.131 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:15.131 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:15.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f930806a050 msgr2=0x7f930806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:15.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f930806a050 0x7f930806c510 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f930c007fd0 tx=0x7f930c007de0 comp rx=0 tx=0).stop 2023-08-25T10:00:15.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 msgr2=0x7f931c14dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:15.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c14dce0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9314009d90 tx=0x7f9314004a40 comp rx=0 tx=0).stop 2023-08-25T10:00:15.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 shutdown_connections 2023-08-25T10:00:15.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f931c0cb570 0x7f931c14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:15.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f930806a050 0x7f930806c510 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:15.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f931c15cd00 0x7f931c14dce0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:15.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 --2- 172.21.15.71:0/2348235516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f931c0cb190 0x7f931c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:15.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 >> 172.21.15.71:0/2348235516 conn(0x7f931c06ae20 msgr2=0x7f931c151370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:15.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 shutdown_connections 2023-08-25T10:00:15.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:15.132+0000 7f9321381640 1 -- 172.21.15.71:0/2348235516 wait complete. 2023-08-25T10:00:15.152 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:17.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:17.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.383+0000 7fb57327f640 1 Processor -- start 2023-08-25T10:00:17.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.383+0000 7fb57327f640 1 -- start start 2023-08-25T10:00:17.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.384+0000 7fb57327f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb56c1530d0 0x7fb56c15c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:17.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57327f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c1534b0 con 0x7fb56c0c6370 2023-08-25T10:00:17.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57327f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0c34e0 con 0x7fb56c071c30 2023-08-25T10:00:17.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57327f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c0c3c50 con 0x7fb56c1530d0 2023-08-25T10:00:17.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57227d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb56c1530d0 0x7fb56c15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:17.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57227d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb56c1530d0 0x7fb56c15c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35412/0 (socket says 172.21.15.71:35412) 2023-08-25T10:00:17.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57227d640 1 -- 172.21.15.71:0/1722792978 learned_addr learned my addr 172.21.15.71:0/1722792978 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:17.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb572a7e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb56c0c6370 0x7fb56c163570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50626/0 (socket says 172.21.15.71:50626) 2023-08-25T10:00:17.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57227d640 1 -- 172.21.15.71:0/1722792978 >> v1:172.21.15.164:6789/0 conn(0x7fb56c071c30 legacy=0x7fb56c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:17.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57227d640 1 -- 172.21.15.71:0/1722792978 >> v1:172.21.15.71:6789/0 conn(0x7fb56c0c6370 legacy=0x7fb56c163570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:17.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.385+0000 7fb57227d640 1 -- 172.21.15.71:0/1722792978 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0c6c80 con 0x7fb56c1530d0 2023-08-25T10:00:17.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.387+0000 7fb57227d640 1 --2- 172.21.15.71:0/1722792978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c1530d0 0x7fb56c15c240 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb568009e50 tx=0x7fb56802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81953bfcd2d7dcf3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:17.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.387+0000 7fb57127b640 1 -- 172.21.15.71:0/1722792978 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56802cb10 con 0x7fb56c1530d0 2023-08-25T10:00:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.387+0000 7fb57127b640 1 -- 172.21.15.71:0/1722792978 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56802cc70 con 0x7fb56c1530d0 2023-08-25T10:00:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.387+0000 7fb57127b640 1 -- 172.21.15.71:0/1722792978 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5680343f0 con 0x7fb56c1530d0 2023-08-25T10:00:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 -- 172.21.15.71:0/1722792978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c1530d0 msgr2=0x7fb56c15c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 --2- 172.21.15.71:0/1722792978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c1530d0 0x7fb56c15c240 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb568009e50 tx=0x7fb56802bf70 comp rx=0 tx=0).stop 2023-08-25T10:00:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 -- 172.21.15.71:0/1722792978 shutdown_connections 2023-08-25T10:00:17.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 --2- 172.21.15.71:0/1722792978 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c1530d0 0x7fb56c15c240 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 -- 172.21.15.71:0/1722792978 >> 172.21.15.71:0/1722792978 conn(0x7fb56c0cab90 msgr2=0x7fb56c0cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:17.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 -- 172.21.15.71:0/1722792978 shutdown_connections 2023-08-25T10:00:17.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.388+0000 7fb57327f640 1 -- 172.21.15.71:0/1722792978 wait complete. 2023-08-25T10:00:17.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.390+0000 7fb57327f640 1 Processor -- start 2023-08-25T10:00:17.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.390+0000 7fb57327f640 1 -- start start 2023-08-25T10:00:17.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.391+0000 7fb57327f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb56c071c30 0x7fb56c148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:17.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.391+0000 7fb57327f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 0x7fb56c144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:17.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.391+0000 7fb57327f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb56c1530d0 0x7fb56c145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:17.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.391+0000 7fb57327f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c0c5a20 con 0x7fb56c0c6370 2023-08-25T10:00:17.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.391+0000 7fb57327f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c070c10 con 0x7fb56c071c30 2023-08-25T10:00:17.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.391+0000 7fb57327f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c1f23a0 con 0x7fb56c1530d0 2023-08-25T10:00:17.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 0x7fb56c144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:17.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 0x7fb56c144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35418/0 (socket says 172.21.15.71:35418) 2023-08-25T10:00:17.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 -- 172.21.15.71:0/1419438534 learned_addr learned my addr 172.21.15.71:0/1419438534 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:17.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb57227d640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb56c071c30 0x7fb56c148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:17.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb572a7e640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb56c1530d0 0x7fb56c145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:17.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 -- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb56c1530d0 msgr2=0x7fb56c145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:17.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb56c1530d0 0x7fb56c145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 -- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb56c071c30 msgr2=0x7fb56c148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb56c071c30 0x7fb56c148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 -- 172.21.15.71:0/1419438534 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568009b30 con 0x7fb56c0c6370 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.392+0000 7fb571a7c640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 0x7fb56c144f00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c460 tx=0x7fb55c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.393+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c012830 con 0x7fb56c0c6370 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.393+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0129d0 con 0x7fb56c0c6370 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.393+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c1e2020 con 0x7fb56c0c6370 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.393+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c012d80 con 0x7fb56c0c6370 2023-08-25T10:00:17.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.393+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c1e25e0 con 0x7fb56c0c6370 2023-08-25T10:00:17.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.394+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c04e420 con 0x7fb56c0c6370 2023-08-25T10:00:17.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.395+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb55c01a3b0 con 0x7fb56c0c6370 2023-08-25T10:00:17.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.396+0000 7fb562ffd640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb55006a050 0x7fb55006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:17.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.396+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb55c0889c0 con 0x7fb56c0c6370 2023-08-25T10:00:17.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.397+0000 7fb57227d640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb55006a050 0x7fb55006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:17.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.397+0000 7fb57227d640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb55006a050 0x7fb55006c510 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb568035000 tx=0x7fb568030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:17.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.402+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb55c053be0 con 0x7fb56c0c6370 2023-08-25T10:00:17.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.666+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb56c0c5d90 con 0x7fb56c0c6370 2023-08-25T10:00:17.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.667+0000 7fb562ffd640 1 -- 172.21.15.71:0/1419438534 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fb55c057200 con 0x7fb56c0c6370 2023-08-25T10:00:17.669 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:17.669 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:17.670 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:17.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb55006a050 msgr2=0x7fb55006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:17.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb55006a050 0x7fb55006c510 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb568035000 tx=0x7fb568030000 comp rx=0 tx=0).stop 2023-08-25T10:00:17.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 msgr2=0x7fb56c144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:17.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 0x7fb56c144f00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c460 tx=0x7fb55c00c920 comp rx=0 tx=0).stop 2023-08-25T10:00:17.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 shutdown_connections 2023-08-25T10:00:17.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb55006a050 0x7fb55006c510 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb56c1530d0 0x7fb56c145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb56c0c6370 0x7fb56c144f00 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 --2- 172.21.15.71:0/1419438534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb56c071c30 0x7fb56c148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:17.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 >> 172.21.15.71:0/1419438534 conn(0x7fb56c0cab90 msgr2=0x7fb56c0ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:17.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 shutdown_connections 2023-08-25T10:00:17.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:17.670+0000 7fb57327f640 1 -- 172.21.15.71:0/1419438534 wait complete. 2023-08-25T10:00:17.690 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:19.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:19.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.925+0000 7faa1ba0d640 1 Processor -- start 2023-08-25T10:00:19.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.925+0000 7faa1ba0d640 1 -- start start 2023-08-25T10:00:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.927+0000 7faa1ba0d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa14152d80 0x7faa140c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa1ba0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1415ce50 0x7faa140c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa1ba0d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa141591c0 con 0x7faa140cacb0 2023-08-25T10:00:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa1ba0d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa141594a0 con 0x7faa14152d80 2023-08-25T10:00:19.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa1ba0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa14159780 con 0x7faa1415ce50 2023-08-25T10:00:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa197a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa14152d80 0x7faa140c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa18fa8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7faa140cacb0 0x7faa1415c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50636/0 (socket says 172.21.15.71:50636) 2023-08-25T10:00:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa197a9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7faa14152d80 0x7faa140c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35430/0 (socket says 172.21.15.71:35430) 2023-08-25T10:00:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa18fa8640 1 -- 172.21.15.71:0/1049727390 learned_addr learned my addr 172.21.15.71:0/1049727390 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:19.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.928+0000 7faa19faa640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1415ce50 0x7faa140c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.929+0000 7faa197a9640 1 -- 172.21.15.71:0/1049727390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1415ce50 msgr2=0x7faa140c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.929+0000 7faa197a9640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1415ce50 0x7faa140c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.929+0000 7faa197a9640 1 -- 172.21.15.71:0/1049727390 >> v1:172.21.15.71:6789/0 conn(0x7faa140cacb0 legacy=0x7faa1415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.929+0000 7faa197a9640 1 -- 172.21.15.71:0/1049727390 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14159a60 con 0x7faa14152d80 2023-08-25T10:00:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.929+0000 7faa0bfff640 1 -- 172.21.15.71:0/1049727390 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813124672 0 0) 0x7faa141591c0 con 0x7faa140cacb0 2023-08-25T10:00:19.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.929+0000 7faa19faa640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1415ce50 0x7faa140c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:00:19.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.930+0000 7faa197a9640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa14152d80 0x7faa140c9360 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7faa100097a0 tx=0x7faa1002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86897d1f3efb6c3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.930+0000 7faa0bfff640 1 -- 172.21.15.71:0/1049727390 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10030800 con 0x7faa14152d80 2023-08-25T10:00:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.930+0000 7faa0bfff640 1 -- 172.21.15.71:0/1049727390 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10030960 con 0x7faa14152d80 2023-08-25T10:00:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.930+0000 7faa0bfff640 1 -- 172.21.15.71:0/1049727390 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10030cf0 con 0x7faa14152d80 2023-08-25T10:00:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 -- 172.21.15.71:0/1049727390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa14152d80 msgr2=0x7faa140c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa14152d80 0x7faa140c9360 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7faa100097a0 tx=0x7faa1002b940 comp rx=0 tx=0).stop 2023-08-25T10:00:19.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 -- 172.21.15.71:0/1049727390 shutdown_connections 2023-08-25T10:00:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1415ce50 0x7faa140c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/1049727390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa14152d80 0x7faa140c9360 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 -- 172.21.15.71:0/1049727390 >> 172.21.15.71:0/1049727390 conn(0x7faa1406b8f0 msgr2=0x7faa140654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 -- 172.21.15.71:0/1049727390 shutdown_connections 2023-08-25T10:00:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.931+0000 7faa1ba0d640 1 -- 172.21.15.71:0/1049727390 wait complete. 2023-08-25T10:00:19.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.933+0000 7faa1ba0d640 1 Processor -- start 2023-08-25T10:00:19.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.933+0000 7faa1ba0d640 1 -- start start 2023-08-25T10:00:19.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa1ba0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140cacb0 0x7faa140c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:19.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa1ba0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa14152d80 0x7faa1415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:19.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa1ba0d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 0x7faa140ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:19.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa1ba0d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1415a6c0 con 0x7faa1415ce50 2023-08-25T10:00:19.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa1ba0d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1415a140 con 0x7faa14152d80 2023-08-25T10:00:19.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa1ba0d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa140ceeb0 con 0x7faa140cacb0 2023-08-25T10:00:19.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa19faa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 0x7faa140ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:19.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa19faa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 0x7faa140ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35438/0 (socket says 172.21.15.71:35438) 2023-08-25T10:00:19.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.934+0000 7faa19faa640 1 -- 172.21.15.71:0/390515785 learned_addr learned my addr 172.21.15.71:0/390515785 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:19.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa197a9640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140cacb0 0x7faa140c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:19.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa18fa8640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa14152d80 0x7faa1415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:19.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa19faa640 1 -- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140cacb0 msgr2=0x7faa140c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:19.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa19faa640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140cacb0 0x7faa140c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:19.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa19faa640 1 -- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa14152d80 msgr2=0x7faa1415aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:19.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa19faa640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa14152d80 0x7faa1415aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:19.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa19faa640 1 -- 172.21.15.71:0/390515785 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009400 con 0x7faa1415ce50 2023-08-25T10:00:19.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa197a9640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140cacb0 0x7faa140c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:00:19.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa18fa8640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa14152d80 0x7faa1415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:00:19.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa19faa640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 0x7faa140ce130 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7faa0c007df0 tx=0x7faa0c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:19.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c003990 con 0x7faa1415ce50 2023-08-25T10:00:19.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c00f040 con 0x7faa1415ce50 2023-08-25T10:00:19.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c003d30 con 0x7faa1415ce50 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.935+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa140cf190 con 0x7faa1415ce50 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.936+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa14144410 con 0x7faa1415ce50 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.937+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa140b1a60 con 0x7faa1415ce50 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.938+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7faa0c004030 con 0x7faa1415ce50 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.939+0000 7faa09ffb640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa9f806a050 0x7fa9f806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.939+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7faa0c088d40 con 0x7faa1415ce50 2023-08-25T10:00:19.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.939+0000 7faa197a9640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa9f806a050 0x7fa9f806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:19.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.940+0000 7faa197a9640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa9f806a050 0x7fa9f806c510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa10036000 tx=0x7faa1003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:19.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:19.944+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7faa0c053f70 con 0x7faa1415ce50 2023-08-25T10:00:20.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.212+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa1415abc0 con 0x7faa1415ce50 2023-08-25T10:00:20.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.213+0000 7faa09ffb640 1 -- 172.21.15.71:0/390515785 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7faa1415abc0 con 0x7faa1415ce50 2023-08-25T10:00:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:20.215 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:20.216 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:20.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa9f806a050 msgr2=0x7fa9f806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:20.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa9f806a050 0x7fa9f806c510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa10036000 tx=0x7faa1003b040 comp rx=0 tx=0).stop 2023-08-25T10:00:20.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 msgr2=0x7faa140ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:20.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 0x7faa140ce130 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7faa0c007df0 tx=0x7faa0c00d4a0 comp rx=0 tx=0).stop 2023-08-25T10:00:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 shutdown_connections 2023-08-25T10:00:20.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa14152d80 0x7faa1415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa9f806a050 0x7fa9f806c510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1415ce50 0x7faa140ce130 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 --2- 172.21.15.71:0/390515785 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140cacb0 0x7faa140c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:20.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 >> 172.21.15.71:0/390515785 conn(0x7faa1406b8f0 msgr2=0x7faa140c4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 shutdown_connections 2023-08-25T10:00:20.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:20.216+0000 7faa1ba0d640 1 -- 172.21.15.71:0/390515785 wait complete. 2023-08-25T10:00:20.236 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:22.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:22.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.455+0000 7fbe5276b640 1 Processor -- start 2023-08-25T10:00:22.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.455+0000 7fbe5276b640 1 -- start start 2023-08-25T10:00:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe5276b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbe4c06b180 0x7fbe4c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe5276b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0c2980 con 0x7fbe4c06bd20 2023-08-25T10:00:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe5276b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0c2c60 con 0x7fbe4c06b940 2023-08-25T10:00:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe5276b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4c15a2d0 con 0x7fbe4c06b180 2023-08-25T10:00:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe50d08640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbe4c06b180 0x7fbe4c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:22.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe50d08640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbe4c06b180 0x7fbe4c15adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59228/0 (socket says 172.21.15.71:59228) 2023-08-25T10:00:22.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.457+0000 7fbe50d08640 1 -- 172.21.15.71:0/3597322228 learned_addr learned my addr 172.21.15.71:0/3597322228 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:22.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.458+0000 7fbe50d08640 1 -- 172.21.15.71:0/3597322228 >> v1:172.21.15.164:6789/0 conn(0x7fbe4c06b940 legacy=0x7fbe4c06a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.458+0000 7fbe50d08640 1 -- 172.21.15.71:0/3597322228 >> v1:172.21.15.71:6789/0 conn(0x7fbe4c06bd20 legacy=0x7fbe4c150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.458+0000 7fbe50d08640 1 -- 172.21.15.71:0/3597322228 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c15a5b0 con 0x7fbe4c06b180 2023-08-25T10:00:22.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.459+0000 7fbe50d08640 1 --2- 172.21.15.71:0/3597322228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b180 0x7fbe4c15adb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009e50 tx=0x7fbe4402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05d8e399ad5ef57 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:22.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.459+0000 7fbe4affd640 1 -- 172.21.15.71:0/3597322228 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4402cbb0 con 0x7fbe4c06b180 2023-08-25T10:00:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe4affd640 1 -- 172.21.15.71:0/3597322228 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4402cd10 con 0x7fbe4c06b180 2023-08-25T10:00:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe4affd640 1 -- 172.21.15.71:0/3597322228 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe44034610 con 0x7fbe4c06b180 2023-08-25T10:00:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe5276b640 1 -- 172.21.15.71:0/3597322228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b180 msgr2=0x7fbe4c15adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe5276b640 1 --2- 172.21.15.71:0/3597322228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b180 0x7fbe4c15adb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009e50 tx=0x7fbe4402c010 comp rx=0 tx=0).stop 2023-08-25T10:00:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe5276b640 1 -- 172.21.15.71:0/3597322228 shutdown_connections 2023-08-25T10:00:22.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe5276b640 1 --2- 172.21.15.71:0/3597322228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b180 0x7fbe4c15adb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe5276b640 1 -- 172.21.15.71:0/3597322228 >> 172.21.15.71:0/3597322228 conn(0x7fbe4c0654f0 msgr2=0x7fbe4c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.460+0000 7fbe5276b640 1 -- 172.21.15.71:0/3597322228 shutdown_connections 2023-08-25T10:00:22.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.461+0000 7fbe5276b640 1 -- 172.21.15.71:0/3597322228 wait complete. 2023-08-25T10:00:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.463+0000 7fbe5276b640 1 Processor -- start 2023-08-25T10:00:22.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.463+0000 7fbe5276b640 1 -- start start 2023-08-25T10:00:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.464+0000 7fbe5276b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe4c06b180 0x7fbe4c1eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:22.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe5276b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 0x7fbe4c154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:22.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe5276b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe4c06bd20 0x7fbe4c1442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:22.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe5276b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c1ee740 con 0x7fbe4c06b940 2023-08-25T10:00:22.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe5276b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c156580 con 0x7fbe4c06bd20 2023-08-25T10:00:22.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe5276b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4c1562e0 con 0x7fbe4c06b180 2023-08-25T10:00:22.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 0x7fbe4c154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:22.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 0x7fbe4c154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59232/0 (socket says 172.21.15.71:59232) 2023-08-25T10:00:22.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 -- 172.21.15.71:0/1703416141 learned_addr learned my addr 172.21.15.71:0/1703416141 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:22.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4bfff640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe4c06b180 0x7fbe4c1eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:22.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe50d08640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe4c06bd20 0x7fbe4c1442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:22.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 -- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe4c06b180 msgr2=0x7fbe4c1eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe4c06b180 0x7fbe4c1eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 -- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe4c06bd20 msgr2=0x7fbe4c1442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe4c06bd20 0x7fbe4c1442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.465+0000 7fbe4b7fe640 1 -- 172.21.15.71:0/1703416141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44009b30 con 0x7fbe4c06b940 2023-08-25T10:00:22.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.466+0000 7fbe4b7fe640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 0x7fbe4c154f90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c450 tx=0x7fbe4000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:22.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.466+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400052d0 con 0x7fbe4c06b940 2023-08-25T10:00:22.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.466+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe40005430 con 0x7fbe4c06b940 2023-08-25T10:00:22.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.466+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40012870 con 0x7fbe4c06b940 2023-08-25T10:00:22.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.466+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c1ee410 con 0x7fbe4c06b940 2023-08-25T10:00:22.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.466+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c06ae10 con 0x7fbe4c06b940 2023-08-25T10:00:22.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.468+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c1ee270 con 0x7fbe4c06b940 2023-08-25T10:00:22.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.468+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbe400129d0 con 0x7fbe4c06b940 2023-08-25T10:00:22.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.469+0000 7fbe48ff9640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbe3806a050 0x7fbe3806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:22.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.470+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fbe40088640 con 0x7fbe4c06b940 2023-08-25T10:00:22.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.470+0000 7fbe4bfff640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbe3806a050 0x7fbe3806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:22.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.470+0000 7fbe4bfff640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbe3806a050 0x7fbe3806c510 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c000f60 tx=0x7fbe3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:22.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.474+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbe4c1ee270 con 0x7fbe4c06b940 2023-08-25T10:00:22.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.734+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe4c1ee270 con 0x7fbe4c06b940 2023-08-25T10:00:22.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.735+0000 7fbe48ff9640 1 -- 172.21.15.71:0/1703416141 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fbe4c1ee270 con 0x7fbe4c06b940 2023-08-25T10:00:22.737 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:22.737 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:22.737 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:22.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbe3806a050 msgr2=0x7fbe3806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbe3806a050 0x7fbe3806c510 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c000f60 tx=0x7fbe3c009040 comp rx=0 tx=0).stop 2023-08-25T10:00:22.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 msgr2=0x7fbe4c154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:22.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 0x7fbe4c154f90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c450 tx=0x7fbe4000c910 comp rx=0 tx=0).stop 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 shutdown_connections 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe4c06bd20 0x7fbe4c1442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbe3806a050 0x7fbe3806c510 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe4c06b940 0x7fbe4c154f90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 --2- 172.21.15.71:0/1703416141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbe4c06b180 0x7fbe4c1eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.737+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 >> 172.21.15.71:0/1703416141 conn(0x7fbe4c0654f0 msgr2=0x7fbe4c150180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:22.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.738+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 shutdown_connections 2023-08-25T10:00:22.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:22.738+0000 7fbe5276b640 1 -- 172.21.15.71:0/1703416141 wait complete. 2023-08-25T10:00:22.758 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:24.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:24.991 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.990+0000 7fd2509a5640 1 Processor -- start 2023-08-25T10:00:24.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.990+0000 7fd2509a5640 1 -- start start 2023-08-25T10:00:24.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd2509a5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:24.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd2509a5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd24c069850 0x7fd24c0c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:24.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd2509a5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24c157bf0 con 0x7fd24c0c75c0 2023-08-25T10:00:24.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd2509a5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24c0c4b60 con 0x7fd24c069850 2023-08-25T10:00:24.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd2509a5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c06a130 con 0x7fd24c1542d0 2023-08-25T10:00:24.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd24a59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd24c069850 0x7fd24c0c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:24.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd24a59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd24c069850 0x7fd24c0c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59240/0 (socket says 172.21.15.71:59240) 2023-08-25T10:00:24.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd24a59c640 1 -- 172.21.15.71:0/2122280946 learned_addr learned my addr 172.21.15.71:0/2122280946 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:24.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.992+0000 7fd24ad9d640 1 --2- 172.21.15.71:0/2122280946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:24.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.993+0000 7fd24a59c640 1 -- 172.21.15.71:0/2122280946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 msgr2=0x7fd24c1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:24.996 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.993+0000 7fd24a59c640 1 --2- 172.21.15.71:0/2122280946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.993+0000 7fd24a59c640 1 -- 172.21.15.71:0/2122280946 >> v1:172.21.15.164:6789/0 conn(0x7fd24c0c75c0 legacy=0x7fd24c0c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.993+0000 7fd24a59c640 1 -- 172.21.15.71:0/2122280946 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c1546b0 con 0x7fd24c069850 2023-08-25T10:00:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.994+0000 7fd24a59c640 1 --2- 172.21.15.71:0/2122280946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c0c7080 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd240009870 tx=0x7fd24002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cc9ec46066da94 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd249d9b640 1 -- 172.21.15.71:0/2122280946 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24002f840 con 0x7fd24c069850 2023-08-25T10:00:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd249d9b640 1 -- 172.21.15.71:0/2122280946 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24002f9a0 con 0x7fd24c069850 2023-08-25T10:00:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd249d9b640 1 -- 172.21.15.71:0/2122280946 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24002fd30 con 0x7fd24c069850 2023-08-25T10:00:24.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd2509a5640 1 -- 172.21.15.71:0/2122280946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 msgr2=0x7fd24c0c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:24.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd2509a5640 1 --2- 172.21.15.71:0/2122280946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c0c7080 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd240009870 tx=0x7fd24002ba20 comp rx=0 tx=0).stop 2023-08-25T10:00:24.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd2509a5640 1 -- 172.21.15.71:0/2122280946 shutdown_connections 2023-08-25T10:00:24.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd2509a5640 1 --2- 172.21.15.71:0/2122280946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c0c7080 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:24.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd2509a5640 1 --2- 172.21.15.71:0/2122280946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:24.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.995+0000 7fd2509a5640 1 -- 172.21.15.71:0/2122280946 >> 172.21.15.71:0/2122280946 conn(0x7fd24c06c280 msgr2=0x7fd24c06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:24.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.996+0000 7fd2509a5640 1 -- 172.21.15.71:0/2122280946 shutdown_connections 2023-08-25T10:00:24.999 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.996+0000 7fd2509a5640 1 -- 172.21.15.71:0/2122280946 wait complete. 2023-08-25T10:00:25.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.997+0000 7fd2509a5640 1 Processor -- start 2023-08-25T10:00:25.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.997+0000 7fd2509a5640 1 -- start start 2023-08-25T10:00:25.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.998+0000 7fd2509a5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:25.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.998+0000 7fd2509a5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd24c0c75c0 0x7fd24c14f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:25.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.998+0000 7fd2509a5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:25.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.998+0000 7fd2509a5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c152d30 con 0x7fd24c069850 2023-08-25T10:00:25.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.998+0000 7fd2509a5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c152120 con 0x7fd24c0c75c0 2023-08-25T10:00:25.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.998+0000 7fd2509a5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c152400 con 0x7fd24c1542d0 2023-08-25T10:00:25.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:25.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59242/0 (socket says 172.21.15.71:59242) 2023-08-25T10:00:25.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 -- 172.21.15.71:0/1410102366 learned_addr learned my addr 172.21.15.71:0/1410102366 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 -- 172.21.15.71:0/1410102366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 msgr2=0x7fd24c1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 -- 172.21.15.71:0/1410102366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd24c0c75c0 msgr2=0x7fd24c14f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd24c0c75c0 0x7fd24c14f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 -- 172.21.15.71:0/1410102366 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400094d0 con 0x7fd24c069850 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:24.999+0000 7fd24ad9d640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c149a50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00d5d0 tx=0x7fd23c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:25.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.000+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0125f0 con 0x7fd24c069850 2023-08-25T10:00:25.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.000+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c012770 con 0x7fd24c069850 2023-08-25T10:00:25.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.000+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c012b20 con 0x7fd24c069850 2023-08-25T10:00:25.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.000+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24c1526e0 con 0x7fd24c069850 2023-08-25T10:00:25.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.000+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24c0c9b00 con 0x7fd24c069850 2023-08-25T10:00:25.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.001+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c04e420 con 0x7fd24c069850 2023-08-25T10:00:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.003+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd23c012c80 con 0x7fd24c069850 2023-08-25T10:00:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.004+0000 7fd2337fe640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23806a050 0x7fd23806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:25.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.004+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fd23c089120 con 0x7fd24c069850 2023-08-25T10:00:25.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.004+0000 7fd24a59c640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23806a050 0x7fd23806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:25.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.005+0000 7fd24a59c640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23806a050 0x7fd23806c510 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd240036000 tx=0x7fd240039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:25.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.009+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd23c054220 con 0x7fd24c069850 2023-08-25T10:00:25.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.278+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd24c069f10 con 0x7fd24c069850 2023-08-25T10:00:25.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.279+0000 7fd2337fe640 1 -- 172.21.15.71:0/1410102366 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7fd23c057840 con 0x7fd24c069850 2023-08-25T10:00:25.281 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:25.282 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:25.282 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:25.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23806a050 msgr2=0x7fd23806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:25.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23806a050 0x7fd23806c510 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd240036000 tx=0x7fd240039040 comp rx=0 tx=0).stop 2023-08-25T10:00:25.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 msgr2=0x7fd24c149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:25.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c149a50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00d5d0 tx=0x7fd23c00da90 comp rx=0 tx=0).stop 2023-08-25T10:00:25.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 shutdown_connections 2023-08-25T10:00:25.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd24c0c75c0 0x7fd24c14f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:25.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23806a050 0x7fd23806c510 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:25.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24c1542d0 0x7fd24c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:25.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 --2- 172.21.15.71:0/1410102366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd24c069850 0x7fd24c149a50 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:25.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 >> 172.21.15.71:0/1410102366 conn(0x7fd24c06c280 msgr2=0x7fd24c06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:25.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 shutdown_connections 2023-08-25T10:00:25.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:25.281+0000 7fd2509a5640 1 -- 172.21.15.71:0/1410102366 wait complete. 2023-08-25T10:00:25.303 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:00:25.304 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 79, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T09:59:08.987163+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:00:25.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:00:25.318 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:00:27.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:27.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.534+0000 7f03894c5640 1 Processor -- start 2023-08-25T10:00:27.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.534+0000 7f03894c5640 1 -- start start 2023-08-25T10:00:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.536+0000 7f03894c5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 0x7f038406a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.536+0000 7f03894c5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03840c4510 0x7f03840cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.536+0000 7f03894c5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03841590c0 con 0x7f03840cec00 2023-08-25T10:00:27.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.536+0000 7f03894c5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0384159380 con 0x7f03840c4510 2023-08-25T10:00:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.536+0000 7f03894c5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0384159660 con 0x7f0384153310 2023-08-25T10:00:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.537+0000 7f03827fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f03840cec00 0x7f03840c3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49894/0 (socket says 172.21.15.71:49894) 2023-08-25T10:00:27.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.537+0000 7f03827fc640 1 -- 172.21.15.71:0/2074921276 learned_addr learned my addr 172.21.15.71:0/2074921276 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:27.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.537+0000 7f0382ffd640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 0x7f038406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.537+0000 7f03837fe640 1 --2- 172.21.15.71:0/2074921276 >> v2:172.21.15.164:3300/0 conn(0x7f03840c4510 0x7f03840cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.537+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323128243 0 0) 0x7f03841590c0 con 0x7f03840cec00 2023-08-25T10:00:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.537+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f036c003520 con 0x7f03840cec00 2023-08-25T10:00:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452067214 0 0) 0x7f036c003520 con 0x7f03840cec00 2023-08-25T10:00:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 msgr2=0x7f038406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:27.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 0x7f038406a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 >> v2:172.21.15.164:3300/0 conn(0x7f03840c4510 msgr2=0x7f03840cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 --2- 172.21.15.71:0/2074921276 >> v2:172.21.15.164:3300/0 conn(0x7f03840c4510 0x7f03840cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0384159940 con 0x7f03840cec00 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0378001670 con 0x7f03840cec00 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 >> v1:172.21.15.71:6789/0 conn(0x7f03840cec00 legacy=0x7f03840c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 0x7f036c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f036c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f036c004d00 0x7f036c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0384159380 con 0x7f036c003f60 2023-08-25T10:00:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0384159660 con 0x7f036c004d00 2023-08-25T10:00:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c00baf0 con 0x7f0384153310 2023-08-25T10:00:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f03837fe640 1 --2- 172.21.15.71:0/2074921276 >> v2:172.21.15.164:3300/0 conn(0x7f03840c4510 0x7f03840cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:00:27.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f03827fc640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f036c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.538+0000 7f0382ffd640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 0x7f036c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.539+0000 7f03837fe640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f036c004d00 0x7f036c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.539+0000 7f03827fc640 1 -- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 msgr2=0x7f036c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:27.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.539+0000 7f03827fc640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0384153310 0x7f036c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.539+0000 7f03827fc640 1 -- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f036c004d00 msgr2=0x7f036c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:27.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.539+0000 7f03827fc640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f036c004d00 0x7f036c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.539+0000 7f03827fc640 1 -- 172.21.15.71:0/2074921276 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c003520 con 0x7f036c003f60 2023-08-25T10:00:27.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.540+0000 7f03827fc640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f036c0047c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f037800c240 tx=0x7f036c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7a4e49c56da207 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:27.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.540+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037802cb00 con 0x7f036c003f60 2023-08-25T10:00:27.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037802cc60 con 0x7f036c003f60 2023-08-25T10:00:27.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f0381ffb640 1 -- 172.21.15.71:0/2074921276 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378005af0 con 0x7f036c003f60 2023-08-25T10:00:27.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03894c5640 1 -- 172.21.15.71:0/2074921276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 msgr2=0x7f036c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:27.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03894c5640 1 --2- 172.21.15.71:0/2074921276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f036c0047c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f037800c240 tx=0x7f036c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03837fe640 1 -- 172.21.15.71:0/2074921276 reap_dead start 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03894c5640 1 -- 172.21.15.71:0/2074921276 shutdown_connections 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03894c5640 1 -- 172.21.15.71:0/2074921276 >> 172.21.15.71:0/2074921276 conn(0x7f0384065910 msgr2=0x7f0384065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03894c5640 1 -- 172.21.15.71:0/2074921276 shutdown_connections 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.541+0000 7f03894c5640 1 -- 172.21.15.71:0/2074921276 wait complete. 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.543+0000 7f03894c5640 1 Processor -- start 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.543+0000 7f03894c5640 1 -- start start 2023-08-25T10:00:27.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f03894c5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f0384149350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f03894c5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03840c61a0 0x7f03841573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f03894c5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03841e41b0 0x7f03841578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f03894c5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03841e4ae0 con 0x7f036c003f60 2023-08-25T10:00:27.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f03894c5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03840c7b60 con 0x7f03840c61a0 2023-08-25T10:00:27.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f03894c5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03841e3ff0 con 0x7f03841e41b0 2023-08-25T10:00:27.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f0382ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f0384149350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f0382ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f0384149350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59262/0 (socket says 172.21.15.71:59262) 2023-08-25T10:00:27.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.544+0000 7f0382ffd640 1 -- 172.21.15.71:0/1509326580 learned_addr learned my addr 172.21.15.71:0/1509326580 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:27.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f0382ffd640 1 -- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03841e41b0 msgr2=0x7f03841578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:27.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f03837fe640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03841e41b0 0x7f03841578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f03827fc640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03840c61a0 0x7f03841573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f0382ffd640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03841e41b0 0x7f03841578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f0382ffd640 1 -- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03840c61a0 msgr2=0x7f03841573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f0382ffd640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03840c61a0 0x7f03841573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f0382ffd640 1 -- 172.21.15.71:0/1509326580 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03841e4820 con 0x7f036c003f60 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f0382ffd640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f0384149350 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0374011600 tx=0x7f0374011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400f8c0 con 0x7f036c003f60 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037400fa20 con 0x7f036c003f60 2023-08-25T10:00:27.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.545+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03840b11a0 con 0x7f036c003f60 2023-08-25T10:00:27.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.546+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400fd90 con 0x7f036c003f60 2023-08-25T10:00:27.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.546+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03840c7780 con 0x7f036c003f60 2023-08-25T10:00:27.556 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.547+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0384069160 con 0x7f036c003f60 2023-08-25T10:00:27.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.548+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0374015460 con 0x7f036c003f60 2023-08-25T10:00:27.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.549+0000 7f036b7fe640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f037006a050 0x7f037006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:27.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.549+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f03740893d0 con 0x7f036c003f60 2023-08-25T10:00:27.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.549+0000 7f03827fc640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f037006a050 0x7f037006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:27.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.550+0000 7f03827fc640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f037006a050 0x7f037006c510 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0378034000 tx=0x7f0378007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:27.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.554+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f03740544d0 con 0x7f036c003f60 2023-08-25T10:00:27.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.807+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03840ce4c0 con 0x7f036c003f60 2023-08-25T10:00:27.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.807+0000 7f036b7fe640 1 -- 172.21.15.71:0/1509326580 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7f0374057af0 con 0x7f036c003f60 2023-08-25T10:00:27.809 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:27.810 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:27.810 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:27.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.810+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f037006a050 msgr2=0x7f037006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.810+0000 7f03894c5640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f037006a050 0x7f037006c510 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0378034000 tx=0x7f0378007000 comp rx=0 tx=0).stop 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.810+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 msgr2=0x7f0384149350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.810+0000 7f03894c5640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f0384149350 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0374011600 tx=0x7f0374011ac0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 shutdown_connections 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03840c61a0 0x7f03841573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f037006a050 0x7f037006c510 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03841e41b0 0x7f03841578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 --2- 172.21.15.71:0/1509326580 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f036c003f60 0x7f0384149350 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:27.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 >> 172.21.15.71:0/1509326580 conn(0x7f0384065910 msgr2=0x7f0384151800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:27.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 shutdown_connections 2023-08-25T10:00:27.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:27.811+0000 7f03894c5640 1 -- 172.21.15.71:0/1509326580 wait complete. 2023-08-25T10:00:27.831 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:29.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:30.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.067+0000 7fecb8a7e640 1 Processor -- start 2023-08-25T10:00:30.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.067+0000 7fecb8a7e640 1 -- start start 2023-08-25T10:00:30.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb8a7e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb4159490 0x7fecb40ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb8a7e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40b1af0 con 0x7fecb40cb570 2023-08-25T10:00:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb8a7e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40b0fb0 con 0x7fecb40cb190 2023-08-25T10:00:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb8a7e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40c98f0 con 0x7fecb4159490 2023-08-25T10:00:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb2ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fecb40cb570 0x7fecb4158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51220/0 (socket says 172.21.15.71:51220) 2023-08-25T10:00:30.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb2ffd640 1 -- 172.21.15.71:0/1850825118 learned_addr learned my addr 172.21.15.71:0/1850825118 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb3fff640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb4159490 0x7fecb40ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.069+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623995128 0 0) 0x7fecb40b1af0 con 0x7fecb40cb570 2023-08-25T10:00:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c003520 con 0x7fecb40cb570 2023-08-25T10:00:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1232844062 0 0) 0x7fec9c003520 con 0x7fecb40cb570 2023-08-25T10:00:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb4159490 msgr2=0x7fecb40ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:30.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb4159490 0x7fecb40ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 >> v1:172.21.15.164:6789/0 conn(0x7fecb40cb190 legacy=0x7fecb40c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb40b1530 con 0x7fecb40cb570 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7feca8001690 con 0x7fecb40cb570 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 >> v1:172.21.15.71:6789/0 conn(0x7fecb40cb570 legacy=0x7fecb4158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9c003a10 0x7fec9c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c004370 0x7fec9c004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec9c005040 0x7fec9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb40c98f0 con 0x7fec9c005040 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c00bf50 con 0x7fec9c004370 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.070+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c00c110 con 0x7fec9c003a10 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb3fff640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec9c005040 0x7fec9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb37fe640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9c003a10 0x7fec9c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb2ffd640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c004370 0x7fec9c004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb3fff640 1 -- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9c003a10 msgr2=0x7fec9c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:30.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb3fff640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9c003a10 0x7fec9c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb3fff640 1 -- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c004370 msgr2=0x7fec9c004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb3fff640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c004370 0x7fec9c004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.071+0000 7fecb3fff640 1 -- 172.21.15.71:0/1850825118 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c003520 con 0x7fec9c005040 2023-08-25T10:00:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.072+0000 7fecb3fff640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec9c005040 0x7fec9c00b980 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fecac009f90 tx=0x7fec9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b1dbd808ead568 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac02cbd0 con 0x7fec9c005040 2023-08-25T10:00:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecac00f040 con 0x7fec9c005040 2023-08-25T10:00:30.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb27fc640 1 -- 172.21.15.71:0/1850825118 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac034440 con 0x7fec9c005040 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb8a7e640 1 -- 172.21.15.71:0/1850825118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec9c005040 msgr2=0x7fec9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/1850825118 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec9c005040 0x7fec9c00b980 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fecac009f90 tx=0x7fec9c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb3fff640 1 -- 172.21.15.71:0/1850825118 reap_dead start 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb8a7e640 1 -- 172.21.15.71:0/1850825118 shutdown_connections 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.073+0000 7fecb8a7e640 1 -- 172.21.15.71:0/1850825118 >> 172.21.15.71:0/1850825118 conn(0x7fecb406ae20 msgr2=0x7fecb406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.074+0000 7fecb8a7e640 1 -- 172.21.15.71:0/1850825118 shutdown_connections 2023-08-25T10:00:30.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.074+0000 7fecb8a7e640 1 -- 172.21.15.71:0/1850825118 wait complete. 2023-08-25T10:00:30.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.075+0000 7fecb8a7e640 1 Processor -- start 2023-08-25T10:00:30.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.076+0000 7fecb8a7e640 1 -- start start 2023-08-25T10:00:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.076+0000 7fecb8a7e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c005040 0x7fecb4158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.076+0000 7fecb8a7e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 0x7fecb41f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb8a7e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb41536f0 0x7fecb414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb8a7e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb414f190 con 0x7fecb41563a0 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb8a7e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb4159050 con 0x7fec9c005040 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb8a7e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb4158590 con 0x7fecb41536f0 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 0x7fecb41f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 0x7fecb41f0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38484/0 (socket says 172.21.15.71:38484) 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 -- 172.21.15.71:0/2891016420 learned_addr learned my addr 172.21.15.71:0/2891016420 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb37fe640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c005040 0x7fecb4158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 -- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb41536f0 msgr2=0x7fecb414ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb41536f0 0x7fecb414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 -- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c005040 msgr2=0x7fecb4158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c005040 0x7fecb4158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.077+0000 7fecb2ffd640 1 -- 172.21.15.71:0/2891016420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb415b980 con 0x7fecb41563a0 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.078+0000 7fecb2ffd640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 0x7fecb41f0ea0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7feca800e760 tx=0x7feca800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.078+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800c640 con 0x7fecb41563a0 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.078+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca800c7c0 con 0x7fecb41563a0 2023-08-25T10:00:30.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.078+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca800cb70 con 0x7fecb41563a0 2023-08-25T10:00:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.078+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb4159280 con 0x7fecb41563a0 2023-08-25T10:00:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.078+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb4159840 con 0x7fecb41563a0 2023-08-25T10:00:30.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.080+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb4072710 con 0x7fecb41563a0 2023-08-25T10:00:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.081+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7feca800ccd0 con 0x7fecb41563a0 2023-08-25T10:00:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.082+0000 7fec9bfff640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feca006a050 0x7feca006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.082+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7feca8088900 con 0x7fecb41563a0 2023-08-25T10:00:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.082+0000 7fecb37fe640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feca006a050 0x7feca006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:30.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.083+0000 7fecb37fe640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feca006a050 0x7feca006c510 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feca40035f0 tx=0x7feca4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:30.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.087+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7feca8053a90 con 0x7fecb41563a0 2023-08-25T10:00:30.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.313+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecb404e420 con 0x7fecb41563a0 2023-08-25T10:00:30.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.314+0000 7fec9bfff640 1 -- 172.21.15.71:0/2891016420 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+6555 (secure 0 0 0) 0x7feca80570b0 con 0x7fecb41563a0 2023-08-25T10:00:30.316 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:30.316 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T09:59:08.987163+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:30.316 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 79 2023-08-25T10:00:30.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feca006a050 msgr2=0x7feca006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feca006a050 0x7feca006c510 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7feca40035f0 tx=0x7feca4009040 comp rx=0 tx=0).stop 2023-08-25T10:00:30.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 msgr2=0x7fecb41f0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:30.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 0x7fecb41f0ea0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7feca800e760 tx=0x7feca800ec20 comp rx=0 tx=0).stop 2023-08-25T10:00:30.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 shutdown_connections 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feca006a050 0x7feca006c510 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecb41536f0 0x7fecb414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecb41563a0 0x7fecb41f0ea0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 --2- 172.21.15.71:0/2891016420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec9c005040 0x7fecb4158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 >> 172.21.15.71:0/2891016420 conn(0x7fecb406ae20 msgr2=0x7fecb4152840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 shutdown_connections 2023-08-25T10:00:30.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:30.316+0000 7fecb8a7e640 1 -- 172.21.15.71:0/2891016420 wait complete. 2023-08-25T10:00:30.338 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:32.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:32.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.564+0000 7f82694c2640 1 Processor -- start 2023-08-25T10:00:32.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.565+0000 7f82694c2640 1 -- start start 2023-08-25T10:00:32.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82694c2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f82640c72f0 0x7f82640acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82694c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f82640c76d0 0x7f8264152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82694c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640ad2f0 0x7f82641532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82694c2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82640adf70 con 0x7f82640c72f0 2023-08-25T10:00:32.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82694c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82640c1060 con 0x7f82640c76d0 2023-08-25T10:00:32.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82694c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82640af730 con 0x7f82640ad2f0 2023-08-25T10:00:32.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f8263fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f82640c72f0 0x7f82640acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f8263fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f82640c72f0 0x7f82640acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38490/0 (socket says 172.21.15.71:38490) 2023-08-25T10:00:32.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f8263fff640 1 -- 172.21.15.71:0/3844882845 learned_addr learned my addr 172.21.15.71:0/3844882845 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:32.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.567+0000 7f82637fe640 1 --2- 172.21.15.71:0/3844882845 >> v2:172.21.15.164:3300/0 conn(0x7f82640c76d0 0x7f8264152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.568+0000 7f8268cc1640 1 --2- 172.21.15.71:0/3844882845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640ad2f0 0x7f82641532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.568+0000 7f8263fff640 1 -- 172.21.15.71:0/3844882845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640ad2f0 msgr2=0x7f82641532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:32.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.568+0000 7f8263fff640 1 --2- 172.21.15.71:0/3844882845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640ad2f0 0x7f82641532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.568+0000 7f8263fff640 1 -- 172.21.15.71:0/3844882845 >> v2:172.21.15.164:3300/0 conn(0x7f82640c76d0 msgr2=0x7f8264152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.568+0000 7f8263fff640 1 --2- 172.21.15.71:0/3844882845 >> v2:172.21.15.164:3300/0 conn(0x7f82640c76d0 0x7f8264152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.568+0000 7f8263fff640 1 -- 172.21.15.71:0/3844882845 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826406b4a0 con 0x7f82640c72f0 2023-08-25T10:00:32.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.569+0000 7f8263fff640 1 --2- 172.21.15.71:0/3844882845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640c72f0 0x7f82640acdb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f82640c7b80 tx=0x7f825402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ca4ecf3cd3bc0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.569+0000 7f8262ffd640 1 -- 172.21.15.71:0/3844882845 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254030860 con 0x7f82640c72f0 2023-08-25T10:00:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.569+0000 7f8262ffd640 1 -- 172.21.15.71:0/3844882845 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82540309c0 con 0x7f82640c72f0 2023-08-25T10:00:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.569+0000 7f8262ffd640 1 -- 172.21.15.71:0/3844882845 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254030d50 con 0x7f82640c72f0 2023-08-25T10:00:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 -- 172.21.15.71:0/3844882845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640c72f0 msgr2=0x7f82640acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:32.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 --2- 172.21.15.71:0/3844882845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640c72f0 0x7f82640acdb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f82640c7b80 tx=0x7f825402b9a0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 -- 172.21.15.71:0/3844882845 shutdown_connections 2023-08-25T10:00:32.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 --2- 172.21.15.71:0/3844882845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640ad2f0 0x7f82641532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 --2- 172.21.15.71:0/3844882845 >> v2:172.21.15.164:3300/0 conn(0x7f82640c76d0 0x7f8264152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 --2- 172.21.15.71:0/3844882845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640c72f0 0x7f82640acdb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 -- 172.21.15.71:0/3844882845 >> 172.21.15.71:0/3844882845 conn(0x7f826406c280 msgr2=0x7f826406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:32.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 -- 172.21.15.71:0/3844882845 shutdown_connections 2023-08-25T10:00:32.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.570+0000 7f82694c2640 1 -- 172.21.15.71:0/3844882845 wait complete. 2023-08-25T10:00:32.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.572+0000 7f82694c2640 1 Processor -- start 2023-08-25T10:00:32.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.572+0000 7f82694c2640 1 -- start start 2023-08-25T10:00:32.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82694c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 0x7f826414edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82694c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640c72f0 0x7f8264154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82694c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82640c76d0 0x7f8264155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82694c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826414f2f0 con 0x7f82640ad2f0 2023-08-25T10:00:32.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82694c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82641e95c0 con 0x7f82640c76d0 2023-08-25T10:00:32.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82694c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82640aba10 con 0x7f82640c72f0 2023-08-25T10:00:32.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 0x7f826414edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 0x7f826414edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38500/0 (socket says 172.21.15.71:38500) 2023-08-25T10:00:32.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 -- 172.21.15.71:0/3359311177 learned_addr learned my addr 172.21.15.71:0/3359311177 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:32.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f82637fe640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640c72f0 0x7f8264154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8268cc1640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82640c76d0 0x7f8264155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 -- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640c72f0 msgr2=0x7f8264154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:32.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640c72f0 0x7f8264154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 -- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82640c76d0 msgr2=0x7f8264155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:32.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.574+0000 7f8263fff640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82640c76d0 0x7f8264155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f8263fff640 1 -- 172.21.15.71:0/3359311177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540094d0 con 0x7f82640ad2f0 2023-08-25T10:00:32.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f8263fff640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 0x7f826414edb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f825402b9e0 tx=0x7f8254004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:32.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254039d60 con 0x7f82640ad2f0 2023-08-25T10:00:32.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254043400 con 0x7f82640ad2f0 2023-08-25T10:00:32.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82540437b0 con 0x7f82640ad2f0 2023-08-25T10:00:32.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82640abcf0 con 0x7f82640ad2f0 2023-08-25T10:00:32.587 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.575+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82640ac2b0 con 0x7f82640ad2f0 2023-08-25T10:00:32.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.577+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82640ac470 con 0x7f82640ad2f0 2023-08-25T10:00:32.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.578+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8254043910 con 0x7f82640ad2f0 2023-08-25T10:00:32.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.579+0000 7f8260ff9640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f825006a050 0x7f825006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:32.588 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.579+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f825403e070 con 0x7f82640ad2f0 2023-08-25T10:00:32.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.579+0000 7f82637fe640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f825006a050 0x7f825006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:32.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.580+0000 7f82637fe640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f825006a050 0x7f825006c510 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8258006fd0 tx=0x7f8258008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:32.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.583+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f82640ac470 con 0x7f82640ad2f0 2023-08-25T10:00:32.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.826+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82640ac470 con 0x7f82640ad2f0 2023-08-25T10:00:32.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.827+0000 7f8260ff9640 1 -- 172.21.15.71:0/3359311177 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f82640ac470 con 0x7f82640ad2f0 2023-08-25T10:00:32.831 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:32.831 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:31.782271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:32.832 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 80 2023-08-25T10:00:32.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f825006a050 msgr2=0x7f825006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:32.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f825006a050 0x7f825006c510 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8258006fd0 tx=0x7f8258008040 comp rx=0 tx=0).stop 2023-08-25T10:00:32.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 msgr2=0x7f826414edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:32.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 0x7f826414edb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f825402b9e0 tx=0x7f8254004730 comp rx=0 tx=0).stop 2023-08-25T10:00:32.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 shutdown_connections 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82640c76d0 0x7f8264155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f825006a050 0x7f825006c510 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f82640c72f0 0x7f8264154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 --2- 172.21.15.71:0/3359311177 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f82640ad2f0 0x7f826414edb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 >> 172.21.15.71:0/3359311177 conn(0x7f826406c280 msgr2=0x7f8264150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 shutdown_connections 2023-08-25T10:00:32.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:32.829+0000 7f82694c2640 1 -- 172.21.15.71:0/3359311177 wait complete. 2023-08-25T10:00:32.849 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:34.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:35.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.080+0000 7f8a06c45640 1 Processor -- start 2023-08-25T10:00:35.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.081+0000 7f8a06c45640 1 -- start start 2023-08-25T10:00:35.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.083+0000 7f8a06c45640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8a00153200 0x7f8a0015ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:35.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.083+0000 7f8a06c45640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f8a0006dae0 0x7f8a00069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:35.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.083+0000 7f8a06c45640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a001599c0 con 0x7f8a00152e20 2023-08-25T10:00:35.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.083+0000 7f8a06c45640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a000c5f00 con 0x7f8a0006dae0 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.083+0000 7f8a06c45640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a00159c60 con 0x7f8a00153200 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a049e1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8a00152e20 0x7f8a000c60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:51222/0 (socket says 172.21.15.71:51222) 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a049e1640 1 -- 172.21.15.71:0/3798778817 learned_addr learned my addr 172.21.15.71:0/3798778817 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a051e2640 1 --2- 172.21.15.71:0/3798778817 >> v2:172.21.15.71:3300/0 conn(0x7f8a0006dae0 0x7f8a00069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f89fffff640 1 --2- 172.21.15.71:0/3798778817 >> v2:172.21.15.164:3300/0 conn(0x7f8a00153200 0x7f8a0015ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f89ff7fe640 1 -- 172.21.15.71:0/3798778817 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987683886 0 0) 0x7f8a001599c0 con 0x7f8a00152e20 2023-08-25T10:00:35.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f89ff7fe640 1 -- 172.21.15.71:0/3798778817 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f0003520 con 0x7f8a00152e20 2023-08-25T10:00:35.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a051e2640 1 -- 172.21.15.71:0/3798778817 >> v2:172.21.15.164:3300/0 conn(0x7f8a00153200 msgr2=0x7f8a0015ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:35.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a051e2640 1 --2- 172.21.15.71:0/3798778817 >> v2:172.21.15.164:3300/0 conn(0x7f8a00153200 0x7f8a0015ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a051e2640 1 -- 172.21.15.71:0/3798778817 >> v1:172.21.15.71:6789/0 conn(0x7f8a00152e20 legacy=0x7f8a000c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:35.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f8a051e2640 1 -- 172.21.15.71:0/3798778817 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0015c410 con 0x7f8a0006dae0 2023-08-25T10:00:35.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.084+0000 7f89fffff640 1 --2- 172.21.15.71:0/3798778817 >> v2:172.21.15.164:3300/0 conn(0x7f8a00153200 0x7f8a0015ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:00:35.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.085+0000 7f8a051e2640 1 --2- 172.21.15.71:0/3798778817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a0006dae0 0x7f8a00069850 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f8a00071dc0 tx=0x7f89f802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60dccfe5fe387616 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:35.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.086+0000 7f89ff7fe640 1 -- 172.21.15.71:0/3798778817 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f802cd00 con 0x7f8a0006dae0 2023-08-25T10:00:35.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.086+0000 7f89ff7fe640 1 -- 172.21.15.71:0/3798778817 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f802ce60 con 0x7f8a0006dae0 2023-08-25T10:00:35.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.086+0000 7f89ff7fe640 1 -- 172.21.15.71:0/3798778817 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8034660 con 0x7f8a0006dae0 2023-08-25T10:00:35.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.086+0000 7f8a06c45640 1 -- 172.21.15.71:0/3798778817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a0006dae0 msgr2=0x7f8a00069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:35.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.086+0000 7f8a06c45640 1 --2- 172.21.15.71:0/3798778817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a0006dae0 0x7f8a00069850 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f8a00071dc0 tx=0x7f89f802bf50 comp rx=0 tx=0).stop 2023-08-25T10:00:35.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.087+0000 7f8a06c45640 1 -- 172.21.15.71:0/3798778817 shutdown_connections 2023-08-25T10:00:35.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.087+0000 7f8a06c45640 1 --2- 172.21.15.71:0/3798778817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a0006dae0 0x7f8a00069850 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.087+0000 7f8a06c45640 1 --2- 172.21.15.71:0/3798778817 >> v2:172.21.15.164:3300/0 conn(0x7f8a00153200 0x7f8a0015ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.087+0000 7f8a06c45640 1 -- 172.21.15.71:0/3798778817 >> 172.21.15.71:0/3798778817 conn(0x7f8a000654f0 msgr2=0x7f8a00065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:35.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.087+0000 7f8a06c45640 1 -- 172.21.15.71:0/3798778817 shutdown_connections 2023-08-25T10:00:35.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.087+0000 7f8a06c45640 1 -- 172.21.15.71:0/3798778817 wait complete. 2023-08-25T10:00:35.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.088+0000 7f8a06c45640 1 Processor -- start 2023-08-25T10:00:35.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.089+0000 7f8a06c45640 1 -- start start 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.089+0000 7f8a06c45640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a0006dae0 0x7f8a001f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a06c45640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a00152e20 0x7f8a000c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a06c45640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 0x7f8a0014f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a06c45640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a00153fd0 con 0x7f8a00153200 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a06c45640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a001f20e0 con 0x7f8a0006dae0 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a06c45640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a00153670 con 0x7f8a00152e20 2023-08-25T10:00:35.095 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 0x7f8a0014f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 0x7f8a0014f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38510/0 (socket says 172.21.15.71:38510) 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 -- 172.21.15.71:0/1407232350 learned_addr learned my addr 172.21.15.71:0/1407232350 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 -- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a00152e20 msgr2=0x7f8a000c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a049e1640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a0006dae0 0x7f8a001f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a00152e20 0x7f8a000c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 -- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a0006dae0 msgr2=0x7f8a001f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:35.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a0006dae0 0x7f8a001f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.090+0000 7f8a051e2640 1 -- 172.21.15.71:0/1407232350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f8009a70 con 0x7f8a00153200 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.091+0000 7f8a051e2640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 0x7f8a0014f000 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f89f802c560 tx=0x7f89f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.091+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f803d600 con 0x7f8a00153200 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.091+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f803d780 con 0x7f8a00153200 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.091+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f803db30 con 0x7f8a00153200 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.091+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a000c3cc0 con 0x7f8a00153200 2023-08-25T10:00:35.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.091+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a001590b0 con 0x7f8a00153200 2023-08-25T10:00:35.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.093+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a00154110 con 0x7f8a00153200 2023-08-25T10:00:35.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.094+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f89f803b040 con 0x7f8a00153200 2023-08-25T10:00:35.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.095+0000 7f89fd7fa640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89e406a020 0x7f89e406c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:35.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.095+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f89f80ab080 con 0x7f8a00153200 2023-08-25T10:00:35.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.095+0000 7f8a049e1640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89e406a020 0x7f89e406c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:35.099 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.096+0000 7f8a049e1640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89e406a020 0x7f89e406c4e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f89f40011e0 tx=0x7f89f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:35.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.100+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f89f80dd990 con 0x7f8a00153200 2023-08-25T10:00:35.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.377+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a00069160 con 0x7f8a00153200 2023-08-25T10:00:35.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.378+0000 7f89fd7fa640 1 -- 172.21.15.71:0/1407232350 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f89f804e020 con 0x7f8a00153200 2023-08-25T10:00:35.381 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:35.381 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:31.782271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:35.381 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 80 2023-08-25T10:00:35.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.380+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89e406a020 msgr2=0x7f89e406c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:35.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.380+0000 7f8a06c45640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89e406a020 0x7f89e406c4e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f89f40011e0 tx=0x7f89f4009040 comp rx=0 tx=0).stop 2023-08-25T10:00:35.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.380+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 msgr2=0x7f8a0014f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:35.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.380+0000 7f8a06c45640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 0x7f8a0014f000 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f89f802c560 tx=0x7f89f8004a40 comp rx=0 tx=0).stop 2023-08-25T10:00:35.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 shutdown_connections 2023-08-25T10:00:35.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89e406a020 0x7f89e406c4e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a00153200 0x7f8a0014f000 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a00152e20 0x7f8a000c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 --2- 172.21.15.71:0/1407232350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a0006dae0 0x7f8a001f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:35.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 >> 172.21.15.71:0/1407232350 conn(0x7f8a000654f0 msgr2=0x7f8a000c60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:35.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 shutdown_connections 2023-08-25T10:00:35.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:35.381+0000 7f8a06c45640 1 -- 172.21.15.71:0/1407232350 wait complete. 2023-08-25T10:00:35.402 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:37.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:37.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.618+0000 7fefcb907640 1 Processor -- start 2023-08-25T10:00:37.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.618+0000 7fefcb907640 1 -- start start 2023-08-25T10:00:37.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefcb907640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fefc41531e0 0x7fefc406a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:37.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefcb907640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fefc40c9e00 0x7fefc40ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:37.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefcb907640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc40b1830 con 0x7fefc40ceff0 2023-08-25T10:00:37.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefcb907640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc40cf400 con 0x7fefc41531e0 2023-08-25T10:00:37.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefcb907640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc406aaa0 con 0x7fefc40c9e00 2023-08-25T10:00:37.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefc96a3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fefc41531e0 0x7fefc406a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:37.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.620+0000 7fefc96a3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fefc41531e0 0x7fefc406a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38520/0 (socket says 172.21.15.71:38520) 2023-08-25T10:00:37.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc96a3640 1 -- 172.21.15.71:0/638271077 learned_addr learned my addr 172.21.15.71:0/638271077 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:37.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc8ea2640 1 --2- 172.21.15.71:0/638271077 >> v2:172.21.15.164:3300/0 conn(0x7fefc40c9e00 0x7fefc40ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:37.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc96a3640 1 -- 172.21.15.71:0/638271077 >> v2:172.21.15.164:3300/0 conn(0x7fefc40c9e00 msgr2=0x7fefc40ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:37.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc96a3640 1 --2- 172.21.15.71:0/638271077 >> v2:172.21.15.164:3300/0 conn(0x7fefc40c9e00 0x7fefc40ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc96a3640 1 -- 172.21.15.71:0/638271077 >> v1:172.21.15.164:6789/0 conn(0x7fefc40ceff0 legacy=0x7fefc40cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:37.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc96a3640 1 -- 172.21.15.71:0/638271077 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4159100 con 0x7fefc41531e0 2023-08-25T10:00:37.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.621+0000 7fefc8ea2640 1 --2- 172.21.15.71:0/638271077 >> v2:172.21.15.164:3300/0 conn(0x7fefc40c9e00 0x7fefc40ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:00:37.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.623+0000 7fefc96a3640 1 --2- 172.21.15.71:0/638271077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc41531e0 0x7fefc406a560 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fefc40cc5b0 tx=0x7fefc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fa261327ce1401 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:37.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.623+0000 7fefbbfff640 1 -- 172.21.15.71:0/638271077 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc0031870 con 0x7fefc41531e0 2023-08-25T10:00:37.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.623+0000 7fefbbfff640 1 -- 172.21.15.71:0/638271077 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc00319d0 con 0x7fefc41531e0 2023-08-25T10:00:37.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.623+0000 7fefbbfff640 1 -- 172.21.15.71:0/638271077 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc0031d60 con 0x7fefc41531e0 2023-08-25T10:00:37.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 -- 172.21.15.71:0/638271077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc41531e0 msgr2=0x7fefc406a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:37.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 --2- 172.21.15.71:0/638271077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc41531e0 0x7fefc406a560 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fefc40cc5b0 tx=0x7fefc002b910 comp rx=0 tx=0).stop 2023-08-25T10:00:37.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 -- 172.21.15.71:0/638271077 shutdown_connections 2023-08-25T10:00:37.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 --2- 172.21.15.71:0/638271077 >> v2:172.21.15.164:3300/0 conn(0x7fefc40c9e00 0x7fefc40ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 --2- 172.21.15.71:0/638271077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc41531e0 0x7fefc406a560 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 -- 172.21.15.71:0/638271077 >> 172.21.15.71:0/638271077 conn(0x7fefc406b240 msgr2=0x7fefc406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:37.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 -- 172.21.15.71:0/638271077 shutdown_connections 2023-08-25T10:00:37.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.624+0000 7fefcb907640 1 -- 172.21.15.71:0/638271077 wait complete. 2023-08-25T10:00:37.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.626+0000 7fefcb907640 1 Processor -- start 2023-08-25T10:00:37.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.626+0000 7fefcb907640 1 -- start start 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefcb907640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 0x7fefc40b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefcb907640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc40ceff0 0x7fefc415b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefcb907640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc41531e0 0x7fefc4146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefcb907640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc415ca20 con 0x7fefc40c9e00 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefcb907640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc41ef360 con 0x7fefc40ceff0 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefcb907640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc415cc70 con 0x7fefc41531e0 2023-08-25T10:00:37.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefc96a3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 0x7fefc40b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:37.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefc96a3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 0x7fefc40b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38528/0 (socket says 172.21.15.71:38528) 2023-08-25T10:00:37.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.627+0000 7fefc96a3640 1 -- 172.21.15.71:0/2209224971 learned_addr learned my addr 172.21.15.71:0/2209224971 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:37.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc8ea2640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc40ceff0 0x7fefc415b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:37.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc96a3640 1 -- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc41531e0 msgr2=0x7fefc4146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:37.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc96a3640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc41531e0 0x7fefc4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc96a3640 1 -- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc40ceff0 msgr2=0x7fefc415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:37.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc96a3640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc40ceff0 0x7fefc415b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc96a3640 1 -- 172.21.15.71:0/2209224971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc00094d0 con 0x7fefc40c9e00 2023-08-25T10:00:37.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.628+0000 7fefc96a3640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 0x7fefc40b3030 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fefc0000c00 tx=0x7fefc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:37.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.629+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc003abe0 con 0x7fefc40c9e00 2023-08-25T10:00:37.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.629+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc003ad40 con 0x7fefc40c9e00 2023-08-25T10:00:37.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.629+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc0044610 con 0x7fefc40c9e00 2023-08-25T10:00:37.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.629+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc415b1f0 con 0x7fefc40c9e00 2023-08-25T10:00:37.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.629+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc415cf50 con 0x7fefc40c9e00 2023-08-25T10:00:37.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.630+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc415b710 con 0x7fefc40c9e00 2023-08-25T10:00:37.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.632+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fefc004d450 con 0x7fefc40c9e00 2023-08-25T10:00:37.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.633+0000 7fefb9ffb640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fefa806a080 0x7fefa806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:37.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.633+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fefc002d040 con 0x7fefc40c9e00 2023-08-25T10:00:37.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.633+0000 7fefc8ea2640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fefa806a080 0x7fefa806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:37.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.634+0000 7fefc8ea2640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fefa806a080 0x7fefa806c540 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fefc415c760 tx=0x7fefb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:37.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.638+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fefc415b710 con 0x7fefc40c9e00 2023-08-25T10:00:37.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.899+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefc415b710 con 0x7fefc40c9e00 2023-08-25T10:00:37.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.900+0000 7fefb9ffb640 1 -- 172.21.15.71:0/2209224971 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6553 (secure 0 0 0) 0x7fefc415b710 con 0x7fefc40c9e00 2023-08-25T10:00:37.903 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:37.903 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:31.782271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:37.903 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 80 2023-08-25T10:00:37.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fefa806a080 msgr2=0x7fefa806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:37.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fefa806a080 0x7fefa806c540 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fefc415c760 tx=0x7fefb4008040 comp rx=0 tx=0).stop 2023-08-25T10:00:37.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 msgr2=0x7fefc40b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:37.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 0x7fefc40b3030 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fefc0000c00 tx=0x7fefc00046c0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 shutdown_connections 2023-08-25T10:00:37.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc40ceff0 0x7fefc415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fefa806a080 0x7fefa806c540 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc41531e0 0x7fefc4146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 --2- 172.21.15.71:0/2209224971 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fefc40c9e00 0x7fefc40b3030 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:37.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 >> 172.21.15.71:0/2209224971 conn(0x7fefc406b240 msgr2=0x7fefc40cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:37.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 shutdown_connections 2023-08-25T10:00:37.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:37.903+0000 7fefcb907640 1 -- 172.21.15.71:0/2209224971 wait complete. 2023-08-25T10:00:37.927 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:00:37.928 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 80, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:00:31.782271+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:00:39.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:40.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.175+0000 7f605494e640 1 Processor -- start 2023-08-25T10:00:40.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.175+0000 7f605494e640 1 -- start start 2023-08-25T10:00:40.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f605494e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60500cb190 0x7f60500ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f605494e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f605015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f605494e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f60500c7f10 0x7f60500c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f605494e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60500c98f0 con 0x7f60500cb190 2023-08-25T10:00:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f605494e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60500b1af0 con 0x7f60500c7f10 2023-08-25T10:00:40.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f605494e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60500b0fb0 con 0x7f60500cb570 2023-08-25T10:00:40.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f604ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60500cb190 0x7f60500ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f604ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f60500cb190 0x7f60500ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58226/0 (socket says 172.21.15.71:58226) 2023-08-25T10:00:40.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.177+0000 7f604ed9d640 1 -- 172.21.15.71:0/99461175 learned_addr learned my addr 172.21.15.71:0/99461175 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:40.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604e59c640 1 --2- 172.21.15.71:0/99461175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f605015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604f59e640 1 --2- 172.21.15.71:0/99461175 >> v2:172.21.15.164:3300/0 conn(0x7f60500c7f10 0x7f60500c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604ed9d640 1 -- 172.21.15.71:0/99461175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 msgr2=0x7f605015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:40.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604ed9d640 1 --2- 172.21.15.71:0/99461175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f605015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604ed9d640 1 -- 172.21.15.71:0/99461175 >> v2:172.21.15.164:3300/0 conn(0x7f60500c7f10 msgr2=0x7f60500c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:40.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604ed9d640 1 --2- 172.21.15.71:0/99461175 >> v2:172.21.15.164:3300/0 conn(0x7f60500c7f10 0x7f60500c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604ed9d640 1 -- 172.21.15.71:0/99461175 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60500b1530 con 0x7f60500cb190 2023-08-25T10:00:40.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.178+0000 7f604f59e640 1 --2- 172.21.15.71:0/99461175 >> v2:172.21.15.164:3300/0 conn(0x7f60500c7f10 0x7f60500c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:00:40.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.179+0000 7f604ed9d640 1 --2- 172.21.15.71:0/99461175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ca910 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f60500cae50 tx=0x7f604002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e414e890a132c10 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:40.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f604dd9b640 1 -- 172.21.15.71:0/99461175 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040030850 con 0x7f60500cb190 2023-08-25T10:00:40.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f604dd9b640 1 -- 172.21.15.71:0/99461175 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60400309b0 con 0x7f60500cb190 2023-08-25T10:00:40.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f604dd9b640 1 -- 172.21.15.71:0/99461175 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040030d40 con 0x7f60500cb190 2023-08-25T10:00:40.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 -- 172.21.15.71:0/99461175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 msgr2=0x7f60500ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:40.184 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 --2- 172.21.15.71:0/99461175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ca910 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f60500cae50 tx=0x7f604002b990 comp rx=0 tx=0).stop 2023-08-25T10:00:40.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 -- 172.21.15.71:0/99461175 shutdown_connections 2023-08-25T10:00:40.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 --2- 172.21.15.71:0/99461175 >> v2:172.21.15.164:3300/0 conn(0x7f60500c7f10 0x7f60500c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 --2- 172.21.15.71:0/99461175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f605015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 --2- 172.21.15.71:0/99461175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ca910 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 -- 172.21.15.71:0/99461175 >> 172.21.15.71:0/99461175 conn(0x7f605006ae20 msgr2=0x7f605006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:40.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 -- 172.21.15.71:0/99461175 shutdown_connections 2023-08-25T10:00:40.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.180+0000 7f605494e640 1 -- 172.21.15.71:0/99461175 wait complete. 2023-08-25T10:00:40.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.182+0000 7f605494e640 1 Processor -- start 2023-08-25T10:00:40.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.182+0000 7f605494e640 1 -- start start 2023-08-25T10:00:40.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.183+0000 7f605494e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60500c7f10 0x7f605014db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.183+0000 7f605494e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.183+0000 7f605494e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f60500cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.183+0000 7f605494e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60501f1510 con 0x7f60500cb190 2023-08-25T10:00:40.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.183+0000 7f605494e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60501ee0a0 con 0x7f60500c7f10 2023-08-25T10:00:40.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.183+0000 7f605494e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60501f1740 con 0x7f60500cb570 2023-08-25T10:00:40.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58228/0 (socket says 172.21.15.71:58228) 2023-08-25T10:00:40.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 -- 172.21.15.71:0/2966370025 learned_addr learned my addr 172.21.15.71:0/2966370025 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:40.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604ed9d640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60500c7f10 0x7f605014db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604f59e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f60500cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 -- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 msgr2=0x7f60500cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:40.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f60500cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 -- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60500c7f10 msgr2=0x7f605014db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60500c7f10 0x7f605014db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.184+0000 7f604e59c640 1 -- 172.21.15.71:0/2966370025 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400094d0 con 0x7f60500cb190 2023-08-25T10:00:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.185+0000 7f604e59c640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ce880 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f604400d800 tx=0x7f604400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:40.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.185+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400b6c0 con 0x7f60500cb190 2023-08-25T10:00:40.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.185+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604400b820 con 0x7f60500cb190 2023-08-25T10:00:40.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.185+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60501f1a20 con 0x7f60500cb190 2023-08-25T10:00:40.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.185+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400bbb0 con 0x7f60500cb190 2023-08-25T10:00:40.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.185+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60501f1fe0 con 0x7f60500cb190 2023-08-25T10:00:40.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.186+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60500b3cc0 con 0x7f60500cb190 2023-08-25T10:00:40.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.188+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f604400bd10 con 0x7f60500cb190 2023-08-25T10:00:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.189+0000 7f60377fe640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f603c06a050 0x7f603c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.189+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6044088fe0 con 0x7f60500cb190 2023-08-25T10:00:40.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.189+0000 7f604ed9d640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f603c06a050 0x7f603c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:40.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.190+0000 7f604ed9d640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f603c06a050 0x7f603c06c510 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6040036000 tx=0x7f6040008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:40.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.194+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6044054140 con 0x7f60500cb190 2023-08-25T10:00:40.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.433+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60501ee210 con 0x7f60500cb190 2023-08-25T10:00:40.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.434+0000 7f60377fe640 1 -- 172.21.15.71:0/2966370025 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+6553 (secure 0 0 0) 0x7f6044057760 con 0x7f60500cb190 2023-08-25T10:00:40.437 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:40.438 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:31.782271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:40.438 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 80 2023-08-25T10:00:40.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f603c06a050 msgr2=0x7f603c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:40.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f603c06a050 0x7f603c06c510 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6040036000 tx=0x7f6040008030 comp rx=0 tx=0).stop 2023-08-25T10:00:40.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 msgr2=0x7f60500ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:40.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ce880 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f604400d800 tx=0x7f604400dcc0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 shutdown_connections 2023-08-25T10:00:40.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f603c06a050 0x7f603c06c510 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60500cb570 0x7f60500cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60500cb190 0x7f60500ce880 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 --2- 172.21.15.71:0/2966370025 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60500c7f10 0x7f605014db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:40.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 >> 172.21.15.71:0/2966370025 conn(0x7f605006ae20 msgr2=0x7f60501515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:40.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.436+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 shutdown_connections 2023-08-25T10:00:40.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:40.437+0000 7f605494e640 1 -- 172.21.15.71:0/2966370025 wait complete. 2023-08-25T10:00:40.457 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:42.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:42.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.698+0000 7f151759e640 1 Processor -- start 2023-08-25T10:00:42.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.699+0000 7f151759e640 1 -- start start 2023-08-25T10:00:42.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.701+0000 7f151759e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f15180af420 0x7f1518069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.701+0000 7f151759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.701+0000 7f151759e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15180c0690 con 0x7f151806b350 2023-08-25T10:00:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.701+0000 7f151759e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1518161690 con 0x7f15180af420 2023-08-25T10:00:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.701+0000 7f151759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15180bf830 con 0x7f1518160ce0 2023-08-25T10:00:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f15180af420 0x7f1518069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f15180af420 0x7f1518069850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58240/0 (socket says 172.21.15.71:58240) 2023-08-25T10:00:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 -- 172.21.15.71:0/211356447 learned_addr learned my addr 172.21.15.71:0/211356447 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f150dd9b640 1 --1- 172.21.15.71:0/211356447 >> v1:172.21.15.164:6789/0 conn(0x7f151806b350 0x7f15180c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:53946/0 (socket says 172.21.15.71:53946) 2023-08-25T10:00:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f1516d9d640 1 --2- 172.21.15.71:0/211356447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 -- 172.21.15.71:0/211356447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 msgr2=0x7f1518161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:42.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 --2- 172.21.15.71:0/211356447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518161150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:42.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 -- 172.21.15.71:0/211356447 >> v1:172.21.15.164:6789/0 conn(0x7f151806b350 legacy=0x7f15180c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:42.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.702+0000 7f151659c640 1 -- 172.21.15.71:0/211356447 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180bfb10 con 0x7f15180af420 2023-08-25T10:00:42.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.704+0000 7f151659c640 1 --2- 172.21.15.71:0/211356447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15180af420 0x7f1518069850 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1508009870 tx=0x7f150802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ec91fec71a756f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:42.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.704+0000 7f1515d9b640 1 -- 172.21.15.71:0/211356447 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508030840 con 0x7f15180af420 2023-08-25T10:00:42.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.704+0000 7f1515d9b640 1 -- 172.21.15.71:0/211356447 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15080309a0 con 0x7f15180af420 2023-08-25T10:00:42.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.704+0000 7f1515d9b640 1 -- 172.21.15.71:0/211356447 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508030d30 con 0x7f15180af420 2023-08-25T10:00:42.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 -- 172.21.15.71:0/211356447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15180af420 msgr2=0x7f1518069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:42.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 --2- 172.21.15.71:0/211356447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15180af420 0x7f1518069850 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1508009870 tx=0x7f150802b980 comp rx=0 tx=0).stop 2023-08-25T10:00:42.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 -- 172.21.15.71:0/211356447 shutdown_connections 2023-08-25T10:00:42.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 --2- 172.21.15.71:0/211356447 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518161150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:42.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 --2- 172.21.15.71:0/211356447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15180af420 0x7f1518069850 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:42.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 -- 172.21.15.71:0/211356447 >> 172.21.15.71:0/211356447 conn(0x7f151806c280 msgr2=0x7f151806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:42.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 -- 172.21.15.71:0/211356447 shutdown_connections 2023-08-25T10:00:42.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.705+0000 7f151759e640 1 -- 172.21.15.71:0/211356447 wait complete. 2023-08-25T10:00:42.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.708+0000 7f151759e640 1 Processor -- start 2023-08-25T10:00:42.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.708+0000 7f151759e640 1 -- start start 2023-08-25T10:00:42.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f151759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f151806b350 0x7f15180c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:42.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f151759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:42.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f151759e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 0x7f1518159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f151759e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151815ec40 con 0x7f15181f5d40 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f151759e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151815ee90 con 0x7f151806b350 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f151759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151815f170 con 0x7f1518160ce0 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f1516d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 0x7f1518159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f1516d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 0x7f1518159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58252/0 (socket says 172.21.15.71:58252) 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.709+0000 7f1516d9d640 1 -- 172.21.15.71:0/3698443205 learned_addr learned my addr 172.21.15.71:0/3698443205 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f150dd9b640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:42.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f151659c640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f151806b350 0x7f15180c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:42.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f1516d9d640 1 -- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 msgr2=0x7f1518160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:42.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f1516d9d640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:42.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f1516d9d640 1 -- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f151806b350 msgr2=0x7f15180c14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:42.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f1516d9d640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f151806b350 0x7f15180c14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f1516d9d640 1 -- 172.21.15.71:0/3698443205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15080094d0 con 0x7f15181f5d40 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.710+0000 7f150dd9b640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.711+0000 7f1516d9d640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 0x7f1518159f20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f151000efc0 tx=0x7f151000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.711+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510009ba0 con 0x7f15181f5d40 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.711+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151000f040 con 0x7f15181f5d40 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.711+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100146c0 con 0x7f15181f5d40 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.711+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151815f450 con 0x7f15181f5d40 2023-08-25T10:00:42.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.711+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151815fa10 con 0x7f15181f5d40 2023-08-25T10:00:42.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.713+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151804e420 con 0x7f15181f5d40 2023-08-25T10:00:42.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.714+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1510014820 con 0x7f15181f5d40 2023-08-25T10:00:42.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.715+0000 7f150f7fe640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14f806a0d0 0x7f14f806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:42.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.715+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f1510089870 con 0x7f15181f5d40 2023-08-25T10:00:42.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.716+0000 7f151659c640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14f806a0d0 0x7f14f806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:42.719 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.717+0000 7f151659c640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14f806a0d0 0x7f14f806c590 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1508036000 tx=0x7f1508008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:42.724 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.722+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f15100549f0 con 0x7f15181f5d40 2023-08-25T10:00:43.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.997+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f151806a0e0 con 0x7f15181f5d40 2023-08-25T10:00:43.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:42.998+0000 7f150f7fe640 1 -- 172.21.15.71:0/3698443205 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f1510018070 con 0x7f15181f5d40 2023-08-25T10:00:43.001 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:43.002 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:40.999588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:43.002 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 81 2023-08-25T10:00:43.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14f806a0d0 msgr2=0x7f14f806c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:43.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14f806a0d0 0x7f14f806c590 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1508036000 tx=0x7f1508008030 comp rx=0 tx=0).stop 2023-08-25T10:00:43.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 msgr2=0x7f1518159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:43.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 0x7f1518159f20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f151000efc0 tx=0x7f151000c490 comp rx=0 tx=0).stop 2023-08-25T10:00:43.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 shutdown_connections 2023-08-25T10:00:43.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f14f806a0d0 0x7f14f806c590 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f15181f5d40 0x7f1518159f20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1518160ce0 0x7f1518160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 --2- 172.21.15.71:0/3698443205 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f151806b350 0x7f15180c14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 >> 172.21.15.71:0/3698443205 conn(0x7f151806c280 msgr2=0x7f15180ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 shutdown_connections 2023-08-25T10:00:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:43.001+0000 7f151759e640 1 -- 172.21.15.71:0/3698443205 wait complete. 2023-08-25T10:00:43.023 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:45.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:45.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.239+0000 7f446f225640 1 Processor -- start 2023-08-25T10:00:45.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.239+0000 7f446f225640 1 -- start start 2023-08-25T10:00:45.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446f225640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4468152fc0 0x7f44680cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:45.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446f225640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4468158d60 0x7f446815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:45.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446f225640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446815c6c0 con 0x7f44680cad40 2023-08-25T10:00:45.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446f225640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44681592c0 con 0x7f4468158d60 2023-08-25T10:00:45.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446f225640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44681595a0 con 0x7f4468152fc0 2023-08-25T10:00:45.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446da22640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4468158d60 0x7f446815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:45.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446ea24640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f44680cad40 0x7f44680c97a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54236/0 (socket says 172.21.15.71:54236) 2023-08-25T10:00:45.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446ea24640 1 -- 172.21.15.71:0/3843376784 learned_addr learned my addr 172.21.15.71:0/3843376784 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:45.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.241+0000 7f446e223640 1 --2- 172.21.15.71:0/3843376784 >> v2:172.21.15.164:3300/0 conn(0x7f4468152fc0 0x7f44680cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:45.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.242+0000 7f446da22640 1 -- 172.21.15.71:0/3843376784 >> v2:172.21.15.164:3300/0 conn(0x7f4468152fc0 msgr2=0x7f44680cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:45.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.242+0000 7f446da22640 1 --2- 172.21.15.71:0/3843376784 >> v2:172.21.15.164:3300/0 conn(0x7f4468152fc0 0x7f44680cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.242+0000 7f446da22640 1 -- 172.21.15.71:0/3843376784 >> v1:172.21.15.71:6789/0 conn(0x7f44680cad40 legacy=0x7f44680c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:45.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.242+0000 7f446d221640 1 -- 172.21.15.71:0/3843376784 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425475914 0 0) 0x7f446815c6c0 con 0x7f44680cad40 2023-08-25T10:00:45.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.242+0000 7f446da22640 1 -- 172.21.15.71:0/3843376784 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4468159880 con 0x7f4468158d60 2023-08-25T10:00:45.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.243+0000 7f446da22640 1 --2- 172.21.15.71:0/3843376784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f446815cb50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f44680cf730 tx=0x7f445802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f127f6f32cb273 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:45.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.243+0000 7f446d221640 1 -- 172.21.15.71:0/3843376784 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458030610 con 0x7f4468158d60 2023-08-25T10:00:45.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.243+0000 7f446d221640 1 -- 172.21.15.71:0/3843376784 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4458030770 con 0x7f4468158d60 2023-08-25T10:00:45.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.243+0000 7f446d221640 1 -- 172.21.15.71:0/3843376784 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458030b00 con 0x7f4468158d60 2023-08-25T10:00:45.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 -- 172.21.15.71:0/3843376784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 msgr2=0x7f446815cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:45.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 --2- 172.21.15.71:0/3843376784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f446815cb50 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f44680cf730 tx=0x7f445802b960 comp rx=0 tx=0).stop 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 -- 172.21.15.71:0/3843376784 shutdown_connections 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 --2- 172.21.15.71:0/3843376784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f446815cb50 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 --2- 172.21.15.71:0/3843376784 >> v2:172.21.15.164:3300/0 conn(0x7f4468152fc0 0x7f44680cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 -- 172.21.15.71:0/3843376784 >> 172.21.15.71:0/3843376784 conn(0x7f446806a3a0 msgr2=0x7f4468070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 -- 172.21.15.71:0/3843376784 shutdown_connections 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.244+0000 7f446f225640 1 -- 172.21.15.71:0/3843376784 wait complete. 2023-08-25T10:00:45.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.246+0000 7f446f225640 1 Processor -- start 2023-08-25T10:00:45.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.246+0000 7f446f225640 1 -- start start 2023-08-25T10:00:45.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446f225640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44680cad40 0x7f446814d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:45.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446f225640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4468152fc0 0x7f4468157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446f225640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f4468157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446f225640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446814ba00 con 0x7f4468158d60 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446f225640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44681ee980 con 0x7f44680cad40 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446f225640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44681eec20 con 0x7f4468152fc0 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.247+0000 7f446ea24640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f4468157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f4468157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58272/0 (socket says 172.21.15.71:58272) 2023-08-25T10:00:45.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 -- 172.21.15.71:0/218067238 learned_addr learned my addr 172.21.15.71:0/218067238 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:45.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446e223640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44680cad40 0x7f446814d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:45.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446da22640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4468152fc0 0x7f4468157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:45.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 -- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4468152fc0 msgr2=0x7f4468157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:45.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4468152fc0 0x7f4468157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 -- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44680cad40 msgr2=0x7f446814d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:45.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44680cad40 0x7f446814d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 -- 172.21.15.71:0/218067238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458009400 con 0x7f4468158d60 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.248+0000 7f446ea24640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f4468157e50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f446000c7a0 tx=0x7f446000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.249+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446000a640 con 0x7f4468158d60 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.249+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446000a7a0 con 0x7f4468158d60 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.249+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44681eef00 con 0x7f4468158d60 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.249+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446000abd0 con 0x7f4468158d60 2023-08-25T10:00:45.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.249+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44681ef4c0 con 0x7f4468158d60 2023-08-25T10:00:45.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.250+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468157630 con 0x7f4468158d60 2023-08-25T10:00:45.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.251+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f446000ad30 con 0x7f4468158d60 2023-08-25T10:00:45.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.252+0000 7f445effd640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f444c06a050 0x7f444c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:45.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.252+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4460088350 con 0x7f4468158d60 2023-08-25T10:00:45.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.253+0000 7f446e223640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f444c06a050 0x7f444c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:45.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.253+0000 7f446e223640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f444c06a050 0x7f444c06c510 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4464006fd0 tx=0x7f4464008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:45.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.257+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4468157630 con 0x7f4468158d60 2023-08-25T10:00:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.552+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4468157630 con 0x7f4468158d60 2023-08-25T10:00:45.555 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.553+0000 7f445effd640 1 -- 172.21.15.71:0/218067238 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f4468157630 con 0x7f4468158d60 2023-08-25T10:00:45.556 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:45.556 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:45.027226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:45.556 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 82 2023-08-25T10:00:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.555+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f444c06a050 msgr2=0x7f444c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.555+0000 7f446f225640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f444c06a050 0x7f444c06c510 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4464006fd0 tx=0x7f4464008040 comp rx=0 tx=0).stop 2023-08-25T10:00:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.555+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 msgr2=0x7f4468157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:45.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.555+0000 7f446f225640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f4468157e50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f446000c7a0 tx=0x7f446000cc60 comp rx=0 tx=0).stop 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.555+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 shutdown_connections 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f444c06a050 0x7f444c06c510 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4468158d60 0x7f4468157e50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4468152fc0 0x7f4468157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 --2- 172.21.15.71:0/218067238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44680cad40 0x7f446814d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 >> 172.21.15.71:0/218067238 conn(0x7f446806a3a0 msgr2=0x7f4468151740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:45.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 shutdown_connections 2023-08-25T10:00:45.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:45.556+0000 7f446f225640 1 -- 172.21.15.71:0/218067238 wait complete. 2023-08-25T10:00:45.579 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:47.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:47.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.806+0000 7ff8e7fff640 1 Processor -- start 2023-08-25T10:00:47.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.806+0000 7ff8e7fff640 1 -- start start 2023-08-25T10:00:47.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8e7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff8e80c72f0 0x7ff8e80acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8e7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff8e80c76d0 0x7ff8e8152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80ad2f0 0x7ff8e81532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8e7fff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e80adf70 con 0x7ff8e80c76d0 2023-08-25T10:00:47.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8e7fff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e80c1060 con 0x7ff8e80c72f0 2023-08-25T10:00:47.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8e7fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80af730 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8dffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff8e80c76d0 0x7ff8e8152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.808+0000 7ff8dffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff8e80c76d0 0x7ff8e8152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58280/0 (socket says 172.21.15.71:58280) 2023-08-25T10:00:47.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8dffff640 1 -- 172.21.15.71:0/986611211 learned_addr learned my addr 172.21.15.71:0/986611211 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:47.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8e6ffd640 1 --2- 172.21.15.71:0/986611211 >> v2:172.21.15.164:3300/0 conn(0x7ff8e80c72f0 0x7ff8e80acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8e77fe640 1 --2- 172.21.15.71:0/986611211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80ad2f0 0x7ff8e81532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8dffff640 1 -- 172.21.15.71:0/986611211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80ad2f0 msgr2=0x7ff8e81532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:47.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8dffff640 1 --2- 172.21.15.71:0/986611211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80ad2f0 0x7ff8e81532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8dffff640 1 -- 172.21.15.71:0/986611211 >> v2:172.21.15.164:3300/0 conn(0x7ff8e80c72f0 msgr2=0x7ff8e80acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:47.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8dffff640 1 --2- 172.21.15.71:0/986611211 >> v2:172.21.15.164:3300/0 conn(0x7ff8e80c72f0 0x7ff8e80acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.809+0000 7ff8dffff640 1 -- 172.21.15.71:0/986611211 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e806b4a0 con 0x7ff8e80c76d0 2023-08-25T10:00:47.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.810+0000 7ff8dffff640 1 --2- 172.21.15.71:0/986611211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80c76d0 0x7ff8e8152da0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80c7c70 tx=0x7ff8d002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534a414e12e32694 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:47.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.810+0000 7ff8e67fc640 1 -- 172.21.15.71:0/986611211 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0036440 con 0x7ff8e80c76d0 2023-08-25T10:00:47.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.810+0000 7ff8e67fc640 1 -- 172.21.15.71:0/986611211 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d00365a0 con 0x7ff8e80c76d0 2023-08-25T10:00:47.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.810+0000 7ff8e67fc640 1 -- 172.21.15.71:0/986611211 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0036930 con 0x7ff8e80c76d0 2023-08-25T10:00:47.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 -- 172.21.15.71:0/986611211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80c76d0 msgr2=0x7ff8e8152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:47.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/986611211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80c76d0 0x7ff8e8152da0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff8e80c7c70 tx=0x7ff8d002e170 comp rx=0 tx=0).stop 2023-08-25T10:00:47.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 -- 172.21.15.71:0/986611211 shutdown_connections 2023-08-25T10:00:47.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/986611211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80ad2f0 0x7ff8e81532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/986611211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80c76d0 0x7ff8e8152da0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/986611211 >> v2:172.21.15.164:3300/0 conn(0x7ff8e80c72f0 0x7ff8e80acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 -- 172.21.15.71:0/986611211 >> 172.21.15.71:0/986611211 conn(0x7ff8e806c280 msgr2=0x7ff8e806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 -- 172.21.15.71:0/986611211 shutdown_connections 2023-08-25T10:00:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.811+0000 7ff8e7fff640 1 -- 172.21.15.71:0/986611211 wait complete. 2023-08-25T10:00:47.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.813+0000 7ff8e7fff640 1 Processor -- start 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.814+0000 7ff8e7fff640 1 -- start start 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.814+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 0x7ff8e8154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.814+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8e80c72f0 0x7ff8e8147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80c76d0 0x7ff8e806a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e7fff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8149970 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e7fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8155140 con 0x7ff8e80c72f0 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e7fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e81496b0 con 0x7ff8e80c76d0 2023-08-25T10:00:47.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 0x7ff8e8154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 0x7ff8e8154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:58294/0 (socket says 172.21.15.71:58294) 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 -- 172.21.15.71:0/2509203671 learned_addr learned my addr 172.21.15.71:0/2509203671 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e77fe640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80c76d0 0x7ff8e806a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8dffff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8e80c72f0 0x7ff8e8147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 -- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80c76d0 msgr2=0x7ff8e806a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80c76d0 0x7ff8e806a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 -- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8e80c72f0 msgr2=0x7ff8e8147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:47.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8e80c72f0 0x7ff8e8147b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:47.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.815+0000 7ff8e6ffd640 1 -- 172.21.15.71:0/2509203671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d000b770 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.816+0000 7ff8e6ffd640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 0x7ff8e8154c00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800d800 tx=0x7ff8d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.816+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800b6c0 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.816+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d800b820 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.816+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800bbb0 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.816+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e8153e40 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.816+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e8154400 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.818+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e804e420 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.818+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff8d800bd10 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.820+0000 7ff8e4ff9640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff8c806a050 0x7ff8c806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.820+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff8d8088fe0 con 0x7ff8e80ad2f0 2023-08-25T10:00:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.820+0000 7ff8dffff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff8c806a050 0x7ff8c806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:47.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.821+0000 7ff8dffff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff8c806a050 0x7ff8c806c510 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0038000 tx=0x7ff8d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:47.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:47.825+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff8d8054210 con 0x7ff8e80ad2f0 2023-08-25T10:00:48.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.057+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8e806ca70 con 0x7ff8e80ad2f0 2023-08-25T10:00:48.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.058+0000 7ff8e4ff9640 1 -- 172.21.15.71:0/2509203671 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6549 (secure 0 0 0) 0x7ff8d8057830 con 0x7ff8e80ad2f0 2023-08-25T10:00:48.061 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:48.062 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:45.027226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:48.062 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 82 2023-08-25T10:00:48.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff8c806a050 msgr2=0x7ff8c806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:48.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff8c806a050 0x7ff8c806c510 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0038000 tx=0x7ff8d0008040 comp rx=0 tx=0).stop 2023-08-25T10:00:48.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 msgr2=0x7ff8e8154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:48.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 0x7ff8e8154c00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800d800 tx=0x7ff8d800dcc0 comp rx=0 tx=0).stop 2023-08-25T10:00:48.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 shutdown_connections 2023-08-25T10:00:48.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff8e80c72f0 0x7ff8e8147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:48.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff8c806a050 0x7ff8c806c510 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:48.067 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff8e80c76d0 0x7ff8e806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:48.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 --2- 172.21.15.71:0/2509203671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff8e80ad2f0 0x7ff8e8154c00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:48.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 >> 172.21.15.71:0/2509203671 conn(0x7ff8e806c280 msgr2=0x7ff8e81561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:48.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 shutdown_connections 2023-08-25T10:00:48.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:48.062+0000 7ff8e7fff640 1 -- 172.21.15.71:0/2509203671 wait complete. 2023-08-25T10:00:48.085 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:50.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:50.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.290+0000 7efc16f6d640 1 Processor -- start 2023-08-25T10:00:50.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.290+0000 7efc16f6d640 1 -- start start 2023-08-25T10:00:50.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc16f6d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efc101576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.294 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc16f6d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc10157bf0 con 0x7efc10069850 2023-08-25T10:00:50.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc16f6d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1006a130 con 0x7efc101542d0 2023-08-25T10:00:50.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc16f6d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc100c4b60 con 0x7efc100c56c0 2023-08-25T10:00:50.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc1576a640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efc10069850 0x7efc100c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42234/0 (socket says 172.21.15.71:42234) 2023-08-25T10:00:50.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc1576a640 1 -- 172.21.15.71:0/195078598 learned_addr learned my addr 172.21.15.71:0/195078598 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:50.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc1676c640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efc101576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.298 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936164515 0 0) 0x7efc00001110 con 0x7efc10069850 2023-08-25T10:00:50.299 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.292+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbfc003520 con 0x7efc10069850 2023-08-25T10:00:50.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360514852 0 0) 0x7efbfc003520 con 0x7efc10069850 2023-08-25T10:00:50.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 msgr2=0x7efc101576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:50.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efc101576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 >> v1:172.21.15.164:6789/0 conn(0x7efc101542d0 legacy=0x7efc100c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc101546b0 con 0x7efc10069850 2023-08-25T10:00:50.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7efc00001800 con 0x7efc10069850 2023-08-25T10:00:50.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 >> v1:172.21.15.71:6789/0 conn(0x7efc10069850 legacy=0x7efc100c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efbfc003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbfc003f90 0x7efbfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.302 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc100c4b60 con 0x7efbfc003f90 2023-08-25T10:00:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc00bae0 con 0x7efbfc004d00 2023-08-25T10:00:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc00bca0 con 0x7efc100c56c0 2023-08-25T10:00:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.293+0000 7efc1576a640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbfc003f90 0x7efbfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc1676c640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc15f6b640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efbfc003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc1576a640 1 -- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 msgr2=0x7efbfc003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc1576a640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efbfc003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc1576a640 1 -- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc004d00 msgr2=0x7efbfc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc1576a640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc004d00 0x7efbfc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc1576a640 1 -- 172.21.15.71:0/195078598 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc003520 con 0x7efbfc003f90 2023-08-25T10:00:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.294+0000 7efc15f6b640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc100c56c0 0x7efbfc003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:00:50.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.295+0000 7efc1576a640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbfc003f90 0x7efbfc0047c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c190 tx=0x7efbfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f27d71b6497a20f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:50.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.295+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00005ad0 con 0x7efbfc003f90 2023-08-25T10:00:50.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.295+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc00005c50 con 0x7efbfc003f90 2023-08-25T10:00:50.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.295+0000 7efc14f69640 1 -- 172.21.15.71:0/195078598 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc00007070 con 0x7efbfc003f90 2023-08-25T10:00:50.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc16f6d640 1 -- 172.21.15.71:0/195078598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbfc003f90 msgr2=0x7efbfc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc16f6d640 1 --2- 172.21.15.71:0/195078598 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbfc003f90 0x7efbfc0047c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c190 tx=0x7efbfc0057f0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc1676c640 1 -- 172.21.15.71:0/195078598 reap_dead start 2023-08-25T10:00:50.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc16f6d640 1 -- 172.21.15.71:0/195078598 shutdown_connections 2023-08-25T10:00:50.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc16f6d640 1 -- 172.21.15.71:0/195078598 >> 172.21.15.71:0/195078598 conn(0x7efc1006c280 msgr2=0x7efc1006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:50.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc16f6d640 1 -- 172.21.15.71:0/195078598 shutdown_connections 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.296+0000 7efc16f6d640 1 -- 172.21.15.71:0/195078598 wait complete. 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.298+0000 7efc16f6d640 1 Processor -- start 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.298+0000 7efc16f6d640 1 -- start start 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.299+0000 7efc16f6d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc003f90 0x7efc10156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.299+0000 7efc16f6d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc10157d90 0x7efc100c3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.299+0000 7efc16f6d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 0x7efc1014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.299+0000 7efc16f6d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc101579f0 con 0x7efc1014d990 2023-08-25T10:00:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.299+0000 7efc16f6d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc10157430 con 0x7efbfc003f90 2023-08-25T10:00:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.299+0000 7efc16f6d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc10157710 con 0x7efc10157d90 2023-08-25T10:00:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 0x7efc1014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 0x7efc1014c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40714/0 (socket says 172.21.15.71:40714) 2023-08-25T10:00:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 -- 172.21.15.71:0/2875678480 learned_addr learned my addr 172.21.15.71:0/2875678480 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc15f6b640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc003f90 0x7efc10156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1576a640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc10157d90 0x7efc100c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 -- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc10157d90 msgr2=0x7efc100c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc10157d90 0x7efc100c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 -- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc003f90 msgr2=0x7efc10156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc003f90 0x7efc10156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 -- 172.21.15.71:0/2875678480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10155c20 con 0x7efc1014d990 2023-08-25T10:00:50.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.300+0000 7efc1676c640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 0x7efc1014c210 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7efc100aefa0 tx=0x7efc080115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:50.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.301+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0800e070 con 0x7efc1014d990 2023-08-25T10:00:50.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.301+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08011e10 con 0x7efc1014d990 2023-08-25T10:00:50.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.301+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08003990 con 0x7efc1014d990 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.301+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc10155f00 con 0x7efc1014d990 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.301+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc10156960 con 0x7efc1014d990 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.302+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1004e420 con 0x7efc1014d990 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.303+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efc08003af0 con 0x7efc1014d990 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.304+0000 7efc067fc640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbf406a080 0x7efbf406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.304+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7efc080890f0 con 0x7efc1014d990 2023-08-25T10:00:50.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.304+0000 7efc15f6b640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbf406a080 0x7efbf406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:50.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.305+0000 7efc15f6b640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbf406a080 0x7efbf406c540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efc0c001750 tx=0x7efc0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:50.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.308+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efc08054280 con 0x7efc1014d990 2023-08-25T10:00:50.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.563+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc100c3d50 con 0x7efc1014d990 2023-08-25T10:00:50.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.563+0000 7efc067fc640 1 -- 172.21.15.71:0/2875678480 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+6549 (secure 0 0 0) 0x7efc080578a0 con 0x7efc1014d990 2023-08-25T10:00:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:50.566 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:45.027226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:50.567 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 82 2023-08-25T10:00:50.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbf406a080 msgr2=0x7efbf406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbf406a080 0x7efbf406c540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efc0c001750 tx=0x7efc0c009040 comp rx=0 tx=0).stop 2023-08-25T10:00:50.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 msgr2=0x7efc1014c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:50.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 0x7efc1014c210 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7efc100aefa0 tx=0x7efc080115a0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 shutdown_connections 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbf406a080 0x7efbf406c540 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc1014d990 0x7efc1014c210 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc10157d90 0x7efc100c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 --2- 172.21.15.71:0/2875678480 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbfc003f90 0x7efc10156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 >> 172.21.15.71:0/2875678480 conn(0x7efc1006c280 msgr2=0x7efc100c7050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 shutdown_connections 2023-08-25T10:00:50.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:50.566+0000 7efc16f6d640 1 -- 172.21.15.71:0/2875678480 wait complete. 2023-08-25T10:00:50.586 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:00:50.587 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 4 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:00:45.027226+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:00:52.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:52.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.809+0000 7f5aaa071640 1 Processor -- start 2023-08-25T10:00:52.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.809+0000 7f5aaa071640 1 -- start start 2023-08-25T10:00:52.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.810+0000 7f5aaa071640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:52.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.810+0000 7f5aaa071640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5aa40c4fb0 0x7f5aa40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:52.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.810+0000 7f5aaa071640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa4157bf0 con 0x7f5aa41542d0 2023-08-25T10:00:52.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.810+0000 7f5aaa071640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa40c4b60 con 0x7f5aa40c4fb0 2023-08-25T10:00:52.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.810+0000 7f5aaa071640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa406a130 con 0x7f5aa4069850 2023-08-25T10:00:52.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5aa40c4fb0 0x7f5aa40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:52.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5aa40c4fb0 0x7f5aa40c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40726/0 (socket says 172.21.15.71:40726) 2023-08-25T10:00:52.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 -- 172.21.15.71:0/1214760019 learned_addr learned my addr 172.21.15.71:0/1214760019 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa886e640 1 --2- 172.21.15.71:0/1214760019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 -- 172.21.15.71:0/1214760019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 msgr2=0x7f5aa41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 --2- 172.21.15.71:0/1214760019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 -- 172.21.15.71:0/1214760019 >> v1:172.21.15.164:6789/0 conn(0x7f5aa41542d0 legacy=0x7f5aa40c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:52.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.811+0000 7f5aa9870640 1 -- 172.21.15.71:0/1214760019 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa41546b0 con 0x7f5aa40c4fb0 2023-08-25T10:00:52.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.813+0000 7f5aa9870640 1 --2- 172.21.15.71:0/1214760019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40c5410 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009e50 tx=0x7f5a9c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4980f3483c94b1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:52.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.813+0000 7f5a9bfff640 1 -- 172.21.15.71:0/1214760019 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c02cb00 con 0x7f5aa40c4fb0 2023-08-25T10:00:52.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.813+0000 7f5a9bfff640 1 -- 172.21.15.71:0/1214760019 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c02cc60 con 0x7f5aa40c4fb0 2023-08-25T10:00:52.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.813+0000 7f5a9bfff640 1 -- 172.21.15.71:0/1214760019 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c0343f0 con 0x7f5aa40c4fb0 2023-08-25T10:00:52.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 -- 172.21.15.71:0/1214760019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa40c4fb0 msgr2=0x7f5aa40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:52.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 --2- 172.21.15.71:0/1214760019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40c5410 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c009e50 tx=0x7f5a9c02bf60 comp rx=0 tx=0).stop 2023-08-25T10:00:52.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 -- 172.21.15.71:0/1214760019 shutdown_connections 2023-08-25T10:00:52.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 --2- 172.21.15.71:0/1214760019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40c5410 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:52.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 --2- 172.21.15.71:0/1214760019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:52.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 -- 172.21.15.71:0/1214760019 >> 172.21.15.71:0/1214760019 conn(0x7f5aa406c280 msgr2=0x7f5aa406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:52.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 -- 172.21.15.71:0/1214760019 shutdown_connections 2023-08-25T10:00:52.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.814+0000 7f5aaa071640 1 -- 172.21.15.71:0/1214760019 wait complete. 2023-08-25T10:00:52.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.816+0000 7f5aaa071640 1 Processor -- start 2023-08-25T10:00:52.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.816+0000 7f5aaa071640 1 -- start start 2023-08-25T10:00:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.817+0000 7f5aaa071640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.817+0000 7f5aaa071640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.817+0000 7f5aaa071640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 0x7f5aa40bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.817+0000 7f5aaa071640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa40bf140 con 0x7f5aa41542d0 2023-08-25T10:00:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.817+0000 7f5aaa071640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa41eebf0 con 0x7f5aa40c4fb0 2023-08-25T10:00:52.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.817+0000 7f5aaa071640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa41eeed0 con 0x7f5aa4069850 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 0x7f5aa40bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 0x7f5aa40bfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40734/0 (socket says 172.21.15.71:40734) 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 -- 172.21.15.71:0/26860198 learned_addr learned my addr 172.21.15.71:0/26860198 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa906f640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa886e640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 -- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 msgr2=0x7f5aa41575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:52.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 -- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aa40c4fb0 msgr2=0x7f5aa40adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40adb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.818+0000 7f5aa9870640 1 -- 172.21.15.71:0/26860198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c009b30 con 0x7f5aa41542d0 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.819+0000 7f5aa9870640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 0x7f5aa40bfe70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c036040 tx=0x7f5a9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.819+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c009510 con 0x7f5aa41542d0 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.819+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9c03e450 con 0x7f5aa41542d0 2023-08-25T10:00:52.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.819+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c03e7e0 con 0x7f5aa41542d0 2023-08-25T10:00:52.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.819+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa40bf390 con 0x7f5aa41542d0 2023-08-25T10:00:52.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.819+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa40bf950 con 0x7f5aa41542d0 2023-08-25T10:00:52.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.820+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa40bfb40 con 0x7f5aa41542d0 2023-08-25T10:00:52.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.821+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5a9c03e940 con 0x7f5aa41542d0 2023-08-25T10:00:52.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.822+0000 7f5a99ffb640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a8806a050 0x7f5a8806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:52.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.822+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f5a9c0ab8f0 con 0x7f5aa41542d0 2023-08-25T10:00:52.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.823+0000 7f5aa906f640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a8806a050 0x7f5a8806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:52.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.823+0000 7f5aa906f640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a8806a050 0x7f5a8806c510 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0000f60 tx=0x7f5aa0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:52.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:52.827+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5a9c076ba0 con 0x7f5aa41542d0 2023-08-25T10:00:53.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.090+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5aa404e420 con 0x7f5aa41542d0 2023-08-25T10:00:53.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.091+0000 7f5a99ffb640 1 -- 172.21.15.71:0/26860198 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+6547 (secure 0 0 0) 0x7f5a9c07a1c0 con 0x7f5aa41542d0 2023-08-25T10:00:53.094 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:53.095 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:52.059531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0,1,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:53.095 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 83 2023-08-25T10:00:53.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.093+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a8806a050 msgr2=0x7f5a8806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.093+0000 7f5aaa071640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a8806a050 0x7f5a8806c510 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0000f60 tx=0x7f5aa0009040 comp rx=0 tx=0).stop 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.093+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 msgr2=0x7f5aa40bfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.093+0000 7f5aaa071640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 0x7f5aa40bfe70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c036040 tx=0x7f5a9c004a40 comp rx=0 tx=0).stop 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.093+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 shutdown_connections 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.093+0000 7f5aaa071640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aa40c4fb0 0x7f5aa40adb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.094+0000 7f5aaa071640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5a8806a050 0x7f5a8806c510 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:53.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.094+0000 7f5aaa071640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5aa41542d0 0x7f5aa40bfe70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:53.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.094+0000 7f5aaa071640 1 --2- 172.21.15.71:0/26860198 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aa4069850 0x7f5aa41575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:53.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.094+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 >> 172.21.15.71:0/26860198 conn(0x7f5aa406c280 msgr2=0x7f5aa4152120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:53.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.094+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 shutdown_connections 2023-08-25T10:00:53.098 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:53.094+0000 7f5aaa071640 1 -- 172.21.15.71:0/26860198 wait complete. 2023-08-25T10:00:53.115 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:55.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:55.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.350+0000 7fc1b8d42640 1 Processor -- start 2023-08-25T10:00:55.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.350+0000 7fc1b8d42640 1 -- start start 2023-08-25T10:00:55.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b8d42640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc1b40cb190 0x7fc1b40ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b8d42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 0x7fc1b415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b8d42640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b40b1af0 con 0x7fc1b40c7f10 2023-08-25T10:00:55.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b8d42640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b40c98f0 con 0x7fc1b40cb190 2023-08-25T10:00:55.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b8d42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40b0fb0 con 0x7fc1b40cb570 2023-08-25T10:00:55.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b3fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc1b40c7f10 0x7fc1b4158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42238/0 (socket says 172.21.15.71:42238) 2023-08-25T10:00:55.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.352+0000 7fc1b3fff640 1 -- 172.21.15.71:0/1490811799 learned_addr learned my addr 172.21.15.71:0/1490811799 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:00:55.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b2ffd640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 0x7fc1b415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/1490811799 >> v2:172.21.15.164:3300/0 conn(0x7fc1b40cb190 0x7fc1b40ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532157075 0 0) 0x7fc1b40b1af0 con 0x7fc1b40c7f10 2023-08-25T10:00:55.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c003520 con 0x7fc1b40c7f10 2023-08-25T10:00:55.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3426304472 0 0) 0x7fc19c003520 con 0x7fc1b40c7f10 2023-08-25T10:00:55.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 msgr2=0x7fc1b415cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:55.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 0x7fc1b415cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 >> v2:172.21.15.164:3300/0 conn(0x7fc1b40cb190 msgr2=0x7fc1b40ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:55.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 --2- 172.21.15.71:0/1490811799 >> v2:172.21.15.164:3300/0 conn(0x7fc1b40cb190 0x7fc1b40ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40b1530 con 0x7fc1b40c7f10 2023-08-25T10:00:55.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fc1ac0019f0 con 0x7fc1b40c7f10 2023-08-25T10:00:55.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.353+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 >> v1:172.21.15.71:6789/0 conn(0x7fc1b40c7f10 legacy=0x7fc1b4158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:55.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b27fc640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc19c0039e0 0x7fc19c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b27fc640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc19c0047c0 0x7fc19c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b27fc640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 0x7fc19c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40c98f0 con 0x7fc19c0039e0 2023-08-25T10:00:55.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40b0fb0 con 0x7fc19c0047c0 2023-08-25T10:00:55.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c00b980 con 0x7fc1b40cb570 2023-08-25T10:00:55.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc19c0039e0 0x7fc19c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b2ffd640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc19c0047c0 0x7fc19c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b37fe640 1 -- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 msgr2=0x7fc19c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:55.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1b40cb570 0x7fc19c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b37fe640 1 -- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc19c0047c0 msgr2=0x7fc19c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:55.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc19c0047c0 0x7fc19c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.354+0000 7fc1b37fe640 1 -- 172.21.15.71:0/1490811799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b40b1530 con 0x7fc19c0039e0 2023-08-25T10:00:55.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.356+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc19c0039e0 0x7fc19c003e00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009ac0 tx=0x7fc19c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76becf797754e7ab server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:55.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.356+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a402c710 con 0x7fc19c0039e0 2023-08-25T10:00:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.356+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a402c870 con 0x7fc19c0039e0 2023-08-25T10:00:55.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.356+0000 7fc1b27fc640 1 -- 172.21.15.71:0/1490811799 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a402cc00 con 0x7fc19c0039e0 2023-08-25T10:00:55.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b8d42640 1 -- 172.21.15.71:0/1490811799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc19c0039e0 msgr2=0x7fc19c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:55.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/1490811799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc19c0039e0 0x7fc19c003e00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009ac0 tx=0x7fc19c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b3fff640 1 -- 172.21.15.71:0/1490811799 reap_dead start 2023-08-25T10:00:55.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b8d42640 1 -- 172.21.15.71:0/1490811799 shutdown_connections 2023-08-25T10:00:55.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b8d42640 1 -- 172.21.15.71:0/1490811799 >> 172.21.15.71:0/1490811799 conn(0x7fc1b406ae20 msgr2=0x7fc1b406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b8d42640 1 -- 172.21.15.71:0/1490811799 shutdown_connections 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.357+0000 7fc1b8d42640 1 -- 172.21.15.71:0/1490811799 wait complete. 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.360+0000 7fc1b8d42640 1 Processor -- start 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.360+0000 7fc1b8d42640 1 -- start start 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b8d42640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc19c0039e0 0x7fc1b4156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b8d42640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 0x7fc1b414dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b8d42640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b414dff0 0x7fc1b41ef870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b8d42640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b40b2ff0 con 0x7fc1b40cd340 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b8d42640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b4158b20 con 0x7fc1b414dff0 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b8d42640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b415bf90 con 0x7fc19c0039e0 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 0x7fc1b414dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 0x7fc1b414dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40750/0 (socket says 172.21.15.71:40750) 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.361+0000 7fc1b2ffd640 1 -- 172.21.15.71:0/342521272 learned_addr learned my addr 172.21.15.71:0/342521272 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:55.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc19c0039e0 0x7fc1b4156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b3fff640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b414dff0 0x7fc1b41ef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b2ffd640 1 -- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc19c0039e0 msgr2=0x7fc1b4156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b2ffd640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc19c0039e0 0x7fc1b4156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b2ffd640 1 -- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b414dff0 msgr2=0x7fc1b41ef870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b2ffd640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b414dff0 0x7fc1b41ef870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b2ffd640 1 -- 172.21.15.71:0/342521272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4158d50 con 0x7fc1b40cd340 2023-08-25T10:00:55.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b3fff640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b414dff0 0x7fc1b41ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:00:55.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b2ffd640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 0x7fc1b414dab0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8010860 tx=0x7fc1a8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:55.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8009610 con 0x7fc1b40cd340 2023-08-25T10:00:55.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a800eb60 con 0x7fc1b40cd340 2023-08-25T10:00:55.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a801c400 con 0x7fc1b40cd340 2023-08-25T10:00:55.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.362+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b4159030 con 0x7fc1b40cd340 2023-08-25T10:00:55.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.363+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b40c9a90 con 0x7fc1b40cd340 2023-08-25T10:00:55.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.364+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b404e420 con 0x7fc1b40cd340 2023-08-25T10:00:55.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.365+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc1a801c560 con 0x7fc1b40cd340 2023-08-25T10:00:55.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.366+0000 7fc19bfff640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc1a006a050 0x7fc1a006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:55.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.366+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc1a8088ff0 con 0x7fc1b40cd340 2023-08-25T10:00:55.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.366+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc1a006a050 0x7fc1a006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:55.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.367+0000 7fc1b37fe640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc1a006a050 0x7fc1a006c510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4036000 tx=0x7fc1a4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:55.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.370+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc1a80542d0 con 0x7fc1b40cd340 2023-08-25T10:00:55.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:00:55.494 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:00:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.646+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1b4159cc0 con 0x7fc1b40cd340 2023-08-25T10:00:55.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.647+0000 7fc19bfff640 1 -- 172.21.15.71:0/342521272 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6541 (secure 0 0 0) 0x7fc1a80578f0 con 0x7fc1b40cd340 2023-08-25T10:00:55.650 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:55.651 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:54.803943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:55.651 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 84 2023-08-25T10:00:55.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.649+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc1a006a050 msgr2=0x7fc1a006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.649+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc1a006a050 0x7fc1a006c510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4036000 tx=0x7fc1a4030000 comp rx=0 tx=0).stop 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.649+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 msgr2=0x7fc1b414dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.649+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 0x7fc1b414dab0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8010860 tx=0x7fc1a8010d20 comp rx=0 tx=0).stop 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 shutdown_connections 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1b414dff0 0x7fc1b41ef870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc1a006a050 0x7fc1a006c510 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc1b40cd340 0x7fc1b414dab0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 --2- 172.21.15.71:0/342521272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc19c0039e0 0x7fc1b4156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:55.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 >> 172.21.15.71:0/342521272 conn(0x7fc1b406ae20 msgr2=0x7fc1b40b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:55.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 shutdown_connections 2023-08-25T10:00:55.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:55.650+0000 7fc1b8d42640 1 -- 172.21.15.71:0/342521272 wait complete. 2023-08-25T10:00:55.672 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:00:57.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:00:57.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.912+0000 7ff59cf29640 1 Processor -- start 2023-08-25T10:00:57.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.912+0000 7ff59cf29640 1 -- start start 2023-08-25T10:00:57.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.914+0000 7ff59cf29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598152fc0 0x7ff5980cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:57.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff59cf29640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff598158d60 0x7ff59815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:57.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff59cf29640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59815c6c0 con 0x7ff5980cad40 2023-08-25T10:00:57.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff59cf29640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5981592c0 con 0x7ff598158d60 2023-08-25T10:00:57.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff59cf29640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5981595a0 con 0x7ff598152fc0 2023-08-25T10:00:57.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff596ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff598158d60 0x7ff59815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:57.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff596ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff598158d60 0x7ff59815cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40754/0 (socket says 172.21.15.71:40754) 2023-08-25T10:00:57.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff596ffd640 1 -- 172.21.15.71:0/3112262468 learned_addr learned my addr 172.21.15.71:0/3112262468 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:57.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff5977fe640 1 --2- 172.21.15.71:0/3112262468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598152fc0 0x7ff5980cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:57.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.915+0000 7ff596ffd640 1 -- 172.21.15.71:0/3112262468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598152fc0 msgr2=0x7ff5980cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:57.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.916+0000 7ff596ffd640 1 --2- 172.21.15.71:0/3112262468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598152fc0 0x7ff5980cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:57.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.916+0000 7ff596ffd640 1 -- 172.21.15.71:0/3112262468 >> v1:172.21.15.164:6789/0 conn(0x7ff5980cad40 legacy=0x7ff5980c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:00:57.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.916+0000 7ff596ffd640 1 -- 172.21.15.71:0/3112262468 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff598159880 con 0x7ff598158d60 2023-08-25T10:00:57.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.917+0000 7ff596ffd640 1 --2- 172.21.15.71:0/3112262468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff598158d60 0x7ff59815cb50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009870 tx=0x7ff58c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300d08e80f53ffa6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:57.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.917+0000 7ff5967fc640 1 -- 172.21.15.71:0/3112262468 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c030840 con 0x7ff598158d60 2023-08-25T10:00:57.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.917+0000 7ff5967fc640 1 -- 172.21.15.71:0/3112262468 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c0309a0 con 0x7ff598158d60 2023-08-25T10:00:57.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.917+0000 7ff5967fc640 1 -- 172.21.15.71:0/3112262468 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c030d30 con 0x7ff598158d60 2023-08-25T10:00:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 -- 172.21.15.71:0/3112262468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff598158d60 msgr2=0x7ff59815cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 --2- 172.21.15.71:0/3112262468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff598158d60 0x7ff59815cb50 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009870 tx=0x7ff58c02b980 comp rx=0 tx=0).stop 2023-08-25T10:00:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 -- 172.21.15.71:0/3112262468 shutdown_connections 2023-08-25T10:00:57.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 --2- 172.21.15.71:0/3112262468 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff598158d60 0x7ff59815cb50 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 --2- 172.21.15.71:0/3112262468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598152fc0 0x7ff5980cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 -- 172.21.15.71:0/3112262468 >> 172.21.15.71:0/3112262468 conn(0x7ff59806a3a0 msgr2=0x7ff598070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 -- 172.21.15.71:0/3112262468 shutdown_connections 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.918+0000 7ff59cf29640 1 -- 172.21.15.71:0/3112262468 wait complete. 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.920+0000 7ff59cf29640 1 Processor -- start 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.920+0000 7ff59cf29640 1 -- start start 2023-08-25T10:00:57.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.921+0000 7ff59cf29640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 0x7ff5980c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.921+0000 7ff59cf29640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff598152fc0 0x7ff5980c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.921+0000 7ff59cf29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598158d60 0x7ff5981465b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.921+0000 7ff59cf29640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5980cd7b0 con 0x7ff5980cad40 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.921+0000 7ff59cf29640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5981eef30 con 0x7ff598152fc0 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.921+0000 7ff59cf29640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5981ef420 con 0x7ff598158d60 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 0x7ff5980c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 0x7ff5980c3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:40762/0 (socket says 172.21.15.71:40762) 2023-08-25T10:00:57.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 -- 172.21.15.71:0/810543331 learned_addr learned my addr 172.21.15.71:0/810543331 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:00:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 -- 172.21.15.71:0/810543331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598158d60 msgr2=0x7ff5981465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598158d60 0x7ff5981465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 -- 172.21.15.71:0/810543331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff598152fc0 msgr2=0x7ff5980c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff598152fc0 0x7ff5980c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 -- 172.21.15.71:0/810543331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0094d0 con 0x7ff5980cad40 2023-08-25T10:00:57.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.922+0000 7ff5977fe640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 0x7ff5980c3d20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff58800d800 tx=0x7ff58800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.923+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58800b6c0 con 0x7ff5980cad40 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.923+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58800b820 con 0x7ff5980cad40 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.923+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5980c6e80 con 0x7ff5980cad40 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.923+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58800bbb0 con 0x7ff5980cad40 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.923+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5980c7120 con 0x7ff5980cad40 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.924+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59804e420 con 0x7ff5980cad40 2023-08-25T10:00:57.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.925+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff58800bd10 con 0x7ff5980cad40 2023-08-25T10:00:57.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.926+0000 7ff57ffff640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff58406a050 0x7ff58406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:00:57.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.926+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff588088ff0 con 0x7ff5980cad40 2023-08-25T10:00:57.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.927+0000 7ff596ffd640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff58406a050 0x7ff58406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:00:57.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.927+0000 7ff596ffd640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff58406a050 0x7ff58406c510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff58c036000 tx=0x7ff58c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:00:57.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:57.932+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff588054210 con 0x7ff5980cad40 2023-08-25T10:00:58.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.185+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5980c9580 con 0x7ff5980cad40 2023-08-25T10:00:58.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.186+0000 7ff57ffff640 1 -- 172.21.15.71:0/810543331 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6539 (secure 0 0 0) 0x7ff588057830 con 0x7ff5980cad40 2023-08-25T10:00:58.189 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:00:58.189 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:55.815533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0,1,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:00:58.190 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 85 2023-08-25T10:00:58.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff58406a050 msgr2=0x7ff58406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:58.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff58406a050 0x7ff58406c510 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff58c036000 tx=0x7ff58c03b040 comp rx=0 tx=0).stop 2023-08-25T10:00:58.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 msgr2=0x7ff5980c3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:00:58.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 0x7ff5980c3d20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff58800d800 tx=0x7ff58800dcc0 comp rx=0 tx=0).stop 2023-08-25T10:00:58.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 shutdown_connections 2023-08-25T10:00:58.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff598152fc0 0x7ff5980c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:58.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff58406a050 0x7ff58406c510 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:58.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff598158d60 0x7ff5981465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:58.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 --2- 172.21.15.71:0/810543331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5980cad40 0x7ff5980c3d20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:00:58.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.189+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 >> 172.21.15.71:0/810543331 conn(0x7ff59806a3a0 msgr2=0x7ff598151840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:00:58.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.190+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 shutdown_connections 2023-08-25T10:00:58.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:00:58.190+0000 7ff59cf29640 1 -- 172.21.15.71:0/810543331 wait complete. 2023-08-25T10:00:58.210 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:00.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:00.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.460+0000 7ff7b6857640 1 Processor -- start 2023-08-25T10:01:00.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.460+0000 7ff7b6857640 1 -- start start 2023-08-25T10:01:00.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.461+0000 7ff7b6857640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff7b01594d0 0x7ff7b00c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7b6857640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b006cc70 0x7ff7b014f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7b6857640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff7b00c3940 0x7ff7b014fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7b6857640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b006a960 con 0x7ff7b01594d0 2023-08-25T10:01:00.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7b6857640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b006ac20 con 0x7ff7b00c3940 2023-08-25T10:01:00.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7b6857640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b006af00 con 0x7ff7b006cc70 2023-08-25T10:01:00.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff7b01594d0 0x7ff7b00c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff7b01594d0 0x7ff7b00c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39476/0 (socket says 172.21.15.71:39476) 2023-08-25T10:01:00.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 -- 172.21.15.71:0/2208504846 learned_addr learned my addr 172.21.15.71:0/2208504846 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:00.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7b4df4640 1 --2- 172.21.15.71:0/2208504846 >> v2:172.21.15.164:3300/0 conn(0x7ff7b00c3940 0x7ff7b014fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b006cc70 0x7ff7b014f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 -- 172.21.15.71:0/2208504846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b006cc70 msgr2=0x7ff7b014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:00.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b006cc70 0x7ff7b014f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 -- 172.21.15.71:0/2208504846 >> v2:172.21.15.164:3300/0 conn(0x7ff7b00c3940 msgr2=0x7ff7b014fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:00.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 --2- 172.21.15.71:0/2208504846 >> v2:172.21.15.164:3300/0 conn(0x7ff7b00c3940 0x7ff7b014fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7affff640 1 -- 172.21.15.71:0/2208504846 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b00c2e60 con 0x7ff7b01594d0 2023-08-25T10:01:00.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.462+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b006cc70 0x7ff7b014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:00.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.464+0000 7ff7affff640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b01594d0 0x7ff7b00c4c90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0071cc0 tx=0x7ff7a002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82958d2e467baac9 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:00.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.464+0000 7ff7aeffd640 1 -- 172.21.15.71:0/2208504846 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0031860 con 0x7ff7b01594d0 2023-08-25T10:01:00.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.464+0000 7ff7aeffd640 1 -- 172.21.15.71:0/2208504846 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a00319c0 con 0x7ff7b01594d0 2023-08-25T10:01:00.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.464+0000 7ff7aeffd640 1 -- 172.21.15.71:0/2208504846 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0031d50 con 0x7ff7b01594d0 2023-08-25T10:01:00.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 -- 172.21.15.71:0/2208504846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b01594d0 msgr2=0x7ff7b00c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:00.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b01594d0 0x7ff7b00c4c90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0071cc0 tx=0x7ff7a002b900 comp rx=0 tx=0).stop 2023-08-25T10:01:00.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 -- 172.21.15.71:0/2208504846 shutdown_connections 2023-08-25T10:01:00.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2208504846 >> v2:172.21.15.164:3300/0 conn(0x7ff7b00c3940 0x7ff7b014fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b006cc70 0x7ff7b014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2208504846 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b01594d0 0x7ff7b00c4c90 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 -- 172.21.15.71:0/2208504846 >> 172.21.15.71:0/2208504846 conn(0x7ff7b00654f0 msgr2=0x7ff7b0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:00.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 -- 172.21.15.71:0/2208504846 shutdown_connections 2023-08-25T10:01:00.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.465+0000 7ff7b6857640 1 -- 172.21.15.71:0/2208504846 wait complete. 2023-08-25T10:01:00.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.467+0000 7ff7b6857640 1 Processor -- start 2023-08-25T10:01:00.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.467+0000 7ff7b6857640 1 -- start start 2023-08-25T10:01:00.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.468+0000 7ff7b6857640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 0x7ff7b014b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.468+0000 7ff7b6857640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b00c3940 0x7ff7b014dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.468+0000 7ff7b6857640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b01594d0 0x7ff7b0142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.468+0000 7ff7b6857640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00ca0c0 con 0x7ff7b006cc70 2023-08-25T10:01:00.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.468+0000 7ff7b6857640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0146780 con 0x7ff7b00c3940 2023-08-25T10:01:00.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.468+0000 7ff7b6857640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b014bc50 con 0x7ff7b01594d0 2023-08-25T10:01:00.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 0x7ff7b014b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 0x7ff7b014b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39492/0 (socket says 172.21.15.71:39492) 2023-08-25T10:01:00.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 -- 172.21.15.71:0/2523441509 learned_addr learned my addr 172.21.15.71:0/2523441509 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:00.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b00c3940 0x7ff7b014dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7b4df4640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b01594d0 0x7ff7b0142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 -- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b01594d0 msgr2=0x7ff7b0142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:00.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b01594d0 0x7ff7b0142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 -- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b00c3940 msgr2=0x7ff7b014dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b00c3940 0x7ff7b014dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7affff640 1 -- 172.21.15.71:0/2523441509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a00094d0 con 0x7ff7b006cc70 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b00c3940 0x7ff7b014dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.469+0000 7ff7b4df4640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b01594d0 0x7ff7b0142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.470+0000 7ff7affff640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 0x7ff7b014b710 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0000c00 tx=0x7ff7a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.470+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a003ac60 con 0x7ff7b006cc70 2023-08-25T10:01:00.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.470+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a003adc0 con 0x7ff7b006cc70 2023-08-25T10:01:00.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.470+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0044610 con 0x7ff7b006cc70 2023-08-25T10:01:00.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.470+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b00c9820 con 0x7ff7b006cc70 2023-08-25T10:01:00.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.470+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b00c9de0 con 0x7ff7b006cc70 2023-08-25T10:01:00.482 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.471+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b01464e0 con 0x7ff7b006cc70 2023-08-25T10:01:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.472+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff7a004d450 con 0x7ff7b006cc70 2023-08-25T10:01:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.473+0000 7ff7acff9640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79c06a050 0x7ff79c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.473+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff7a00ab110 con 0x7ff7b006cc70 2023-08-25T10:01:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.474+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79c06a050 0x7ff79c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.474+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79c06a050 0x7ff79c06c510 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff7b01e2c20 tx=0x7ff7a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:00.483 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.479+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff7a00762b0 con 0x7ff7b006cc70 2023-08-25T10:01:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.727+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7b0069160 con 0x7ff7b006cc70 2023-08-25T10:01:00.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.728+0000 7ff7acff9640 1 -- 172.21.15.71:0/2523441509 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6535 (secure 0 0 0) 0x7ff7a00798d0 con 0x7ff7b006cc70 2023-08-25T10:01:00.733 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:00.733 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:59.832482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:00.734 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2023-08-25T10:01:00.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79c06a050 msgr2=0x7ff79c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79c06a050 0x7ff79c06c510 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff7b01e2c20 tx=0x7ff7a4008040 comp rx=0 tx=0).stop 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 msgr2=0x7ff7b014b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 0x7ff7b014b710 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0000c00 tx=0x7ff7a00046c0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 shutdown_connections 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b00c3940 0x7ff7b014dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79c06a050 0x7ff79c06c510 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b01594d0 0x7ff7b0142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 --2- 172.21.15.71:0/2523441509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b006cc70 0x7ff7b014b710 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:00.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 >> 172.21.15.71:0/2523441509 conn(0x7ff7b00654f0 msgr2=0x7ff7b014cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:00.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 shutdown_connections 2023-08-25T10:01:00.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:00.731+0000 7ff7b6857640 1 -- 172.21.15.71:0/2523441509 wait complete. 2023-08-25T10:01:00.751 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:02.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:02.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.970+0000 7efbfe68a640 1 Processor -- start 2023-08-25T10:01:02.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.970+0000 7efbfe68a640 1 -- start start 2023-08-25T10:01:02.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfe68a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf8069850 0x7efbf81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfe68a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efbf80c4fb0 0x7efbf80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfe68a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf8157bf0 con 0x7efbf81542d0 2023-08-25T10:01:02.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfe68a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf80c4b60 con 0x7efbf80c4fb0 2023-08-25T10:01:02.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfe68a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf806a130 con 0x7efbf8069850 2023-08-25T10:01:02.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfd688640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efbf81542d0 0x7efbf80c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36546/0 (socket says 172.21.15.71:36546) 2023-08-25T10:01:02.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.972+0000 7efbfd688640 1 -- 172.21.15.71:0/310722318 learned_addr learned my addr 172.21.15.71:0/310722318 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:02.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbfce87640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf8069850 0x7efbf81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbfde89640 1 --2- 172.21.15.71:0/310722318 >> v2:172.21.15.164:3300/0 conn(0x7efbf80c4fb0 0x7efbf80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141207789 0 0) 0x7efbf4001110 con 0x7efbf81542d0 2023-08-25T10:01:02.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbe4003520 con 0x7efbf81542d0 2023-08-25T10:01:02.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113918170 0 0) 0x7efbe4003520 con 0x7efbf81542d0 2023-08-25T10:01:02.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf8069850 msgr2=0x7efbf81576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:02.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf8069850 0x7efbf81576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 >> v2:172.21.15.164:3300/0 conn(0x7efbf80c4fb0 msgr2=0x7efbf80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:02.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 --2- 172.21.15.71:0/310722318 >> v2:172.21.15.164:3300/0 conn(0x7efbf80c4fb0 0x7efbf80c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf81546b0 con 0x7efbf81542d0 2023-08-25T10:01:02.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7efbf4001800 con 0x7efbf81542d0 2023-08-25T10:01:02.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 >> v1:172.21.15.71:6789/0 conn(0x7efbf81542d0 legacy=0x7efbf80c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:02.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbe40039e0 0x7efbe4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbe40047c0 0x7efbe4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe4005180 0x7efbe400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80c4b60 con 0x7efbe40039e0 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf806a130 con 0x7efbe4005180 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbe400c020 con 0x7efbe40047c0 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbfce87640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf8069850 0x7efbf81576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.973+0000 7efbfde89640 1 --2- 172.21.15.71:0/310722318 >> v2:172.21.15.164:3300/0 conn(0x7efbf80c4fb0 0x7efbf80c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.974+0000 7efbfd688640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbe40039e0 0x7efbe4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.974+0000 7efbfd688640 1 -- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbe40047c0 msgr2=0x7efbe4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:02.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.974+0000 7efbfd688640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbe40047c0 0x7efbe4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.974+0000 7efbfd688640 1 -- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe4005180 msgr2=0x7efbe400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:02.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.974+0000 7efbfd688640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe4005180 0x7efbe400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.974+0000 7efbfd688640 1 -- 172.21.15.71:0/310722318 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe4003520 con 0x7efbe40039e0 2023-08-25T10:01:02.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.975+0000 7efbfd688640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbe40039e0 0x7efbe4003e00 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c240 tx=0x7efbe40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33745922e93f6532 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:02.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf4005ad0 con 0x7efbe40039e0 2023-08-25T10:01:02.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf4005c50 con 0x7efbe40039e0 2023-08-25T10:01:02.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbeffff640 1 -- 172.21.15.71:0/310722318 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf402c8b0 con 0x7efbe40039e0 2023-08-25T10:01:02.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfe68a640 1 -- 172.21.15.71:0/310722318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbe40039e0 msgr2=0x7efbe4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:02.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfe68a640 1 --2- 172.21.15.71:0/310722318 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbe40039e0 0x7efbe4003e00 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c240 tx=0x7efbe40057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfde89640 1 -- 172.21.15.71:0/310722318 reap_dead start 2023-08-25T10:01:02.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfe68a640 1 -- 172.21.15.71:0/310722318 shutdown_connections 2023-08-25T10:01:02.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfe68a640 1 -- 172.21.15.71:0/310722318 >> 172.21.15.71:0/310722318 conn(0x7efbf806c280 msgr2=0x7efbf806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:02.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfe68a640 1 -- 172.21.15.71:0/310722318 shutdown_connections 2023-08-25T10:01:02.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.976+0000 7efbfe68a640 1 -- 172.21.15.71:0/310722318 wait complete. 2023-08-25T10:01:02.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.979+0000 7efbfe68a640 1 Processor -- start 2023-08-25T10:01:02.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.980+0000 7efbfe68a640 1 -- start start 2023-08-25T10:01:02.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfe68a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe40039e0 0x7efbf80ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfe68a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 0x7efbf814be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfe68a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf814c3d0 0x7efbf80c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfe68a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80c06a0 con 0x7efbf81576d0 2023-08-25T10:01:02.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfe68a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80bf270 con 0x7efbe40039e0 2023-08-25T10:01:02.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfe68a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8150960 con 0x7efbf814c3d0 2023-08-25T10:01:02.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfce87640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 0x7efbf814be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfce87640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 0x7efbf814be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39500/0 (socket says 172.21.15.71:39500) 2023-08-25T10:01:02.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.981+0000 7efbfce87640 1 -- 172.21.15.71:0/3079789699 learned_addr learned my addr 172.21.15.71:0/3079789699 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:02.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfde89640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf814c3d0 0x7efbf80c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfd688640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe40039e0 0x7efbf80ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.992 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfce87640 1 -- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf814c3d0 msgr2=0x7efbf80c31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:02.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfce87640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf814c3d0 0x7efbf80c31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfce87640 1 -- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe40039e0 msgr2=0x7efbf80ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:02.993 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfce87640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe40039e0 0x7efbf80ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:02.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfce87640 1 -- 172.21.15.71:0/3079789699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8157180 con 0x7efbf81576d0 2023-08-25T10:01:02.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbfce87640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 0x7efbf814be90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7efbe800e540 tx=0x7efbe800ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:02.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.982+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8004230 con 0x7efbf81576d0 2023-08-25T10:01:02.994 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.983+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe80043b0 con 0x7efbf81576d0 2023-08-25T10:01:02.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.983+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8004520 con 0x7efbf81576d0 2023-08-25T10:01:02.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.983+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf8157440 con 0x7efbf81576d0 2023-08-25T10:01:02.995 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.983+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf81eee80 con 0x7efbf81576d0 2023-08-25T10:01:02.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.984+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf80bf420 con 0x7efbf81576d0 2023-08-25T10:01:02.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.985+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efbe8003850 con 0x7efbf81576d0 2023-08-25T10:01:02.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.987+0000 7efbedffb640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbdc06a050 0x7efbdc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:02.997 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.987+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7efbe8088460 con 0x7efbf81576d0 2023-08-25T10:01:02.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.987+0000 7efbfd688640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbdc06a050 0x7efbdc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:02.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.988+0000 7efbfd688640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbdc06a050 0x7efbdc06c510 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efbf4034000 tx=0x7efbf4007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:02.998 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:02.992+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efbf80bf420 con 0x7efbf81576d0 2023-08-25T10:01:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.248+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efbf80bf420 con 0x7efbf81576d0 2023-08-25T10:01:03.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.249+0000 7efbedffb640 1 -- 172.21.15.71:0/3079789699 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6535 (secure 0 0 0) 0x7efbf80bf420 con 0x7efbf81576d0 2023-08-25T10:01:03.251 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:03.251 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:59.832482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:03.251 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2023-08-25T10:01:03.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbdc06a050 msgr2=0x7efbdc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbdc06a050 0x7efbdc06c510 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efbf4034000 tx=0x7efbf4007000 comp rx=0 tx=0).stop 2023-08-25T10:01:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 msgr2=0x7efbf814be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 0x7efbf814be90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7efbe800e540 tx=0x7efbe800ea00 comp rx=0 tx=0).stop 2023-08-25T10:01:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 shutdown_connections 2023-08-25T10:01:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efbdc06a050 0x7efbdc06c510 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:03.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efbf814c3d0 0x7efbf80c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efbf81576d0 0x7efbf814be90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 --2- 172.21.15.71:0/3079789699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efbe40039e0 0x7efbf80ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 >> 172.21.15.71:0/3079789699 conn(0x7efbf806c280 msgr2=0x7efbf81530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 shutdown_connections 2023-08-25T10:01:03.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:03.252+0000 7efbfe68a640 1 -- 172.21.15.71:0/3079789699 wait complete. 2023-08-25T10:01:03.272 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:01:03.273 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:00:59.832482+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:01:05.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:05.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.526+0000 7f9a01d67640 1 Processor -- start 2023-08-25T10:01:05.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.526+0000 7f9a01d67640 1 -- start start 2023-08-25T10:01:05.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01d67640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 0x7f99fc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01d67640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f99fc15ada0 0x7f99fc150830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01d67640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc15a3d0 con 0x7f99fc0c61c0 2023-08-25T10:01:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01d67640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99fc157060 con 0x7f99fc15ada0 2023-08-25T10:01:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01d67640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc157340 con 0x7f99fc0c5de0 2023-08-25T10:01:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01566640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f99fc0c61c0 0x7f99fc06f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36558/0 (socket says 172.21.15.71:36558) 2023-08-25T10:01:05.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.530+0000 7f9a01566640 1 -- 172.21.15.71:0/3107321638 learned_addr learned my addr 172.21.15.71:0/3107321638 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fbfff640 1 --2- 172.21.15.71:0/3107321638 >> v2:172.21.15.164:3300/0 conn(0x7f99fc15ada0 0x7f99fc150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f9a00d65640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 0x7f99fc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895057606 0 0) 0x7f99fc15a3d0 con 0x7f99fc0c61c0 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ec003520 con 0x7f99fc0c61c0 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3342244638 0 0) 0x7f99ec003520 con 0x7f99fc0c61c0 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 msgr2=0x7f99fc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 0x7f99fc15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 >> v2:172.21.15.164:3300/0 conn(0x7f99fc15ada0 msgr2=0x7f99fc150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 --2- 172.21.15.71:0/3107321638 >> v2:172.21.15.164:3300/0 conn(0x7f99fc15ada0 0x7f99fc150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc157620 con 0x7f99fc0c61c0 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f99f40019f0 con 0x7f99fc0c61c0 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 >> v1:172.21.15.71:6789/0 conn(0x7f99fc0c61c0 legacy=0x7f99fc06f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99ec003970 0x7f99ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 0x7f99ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec004d00 0x7f99ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc157060 con 0x7f99ec003970 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc157340 con 0x7f99ec004d00 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.531+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec00b980 con 0x7f99fc0c5de0 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a00d65640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99ec003970 0x7f99ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a01566640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec004d00 0x7f99ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f99fbfff640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 0x7f99ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a00d65640 1 -- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 msgr2=0x7f99ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:05.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a00d65640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc0c5de0 0x7f99ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a00d65640 1 -- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec004d00 msgr2=0x7f99ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:05.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a00d65640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec004d00 0x7f99ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.532+0000 7f9a00d65640 1 -- 172.21.15.71:0/3107321638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec003520 con 0x7f99ec003970 2023-08-25T10:01:05.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.534+0000 7f9a00d65640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99ec003970 0x7f99ec003d90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009ac0 tx=0x7f99ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364e8c2ce4704703 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:05.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.534+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f002c6b0 con 0x7f99ec003970 2023-08-25T10:01:05.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.534+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f002c810 con 0x7f99ec003970 2023-08-25T10:01:05.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.534+0000 7f99fb7fe640 1 -- 172.21.15.71:0/3107321638 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f002cba0 con 0x7f99ec003970 2023-08-25T10:01:05.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01d67640 1 -- 172.21.15.71:0/3107321638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99ec003970 msgr2=0x7f99ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01d67640 1 --2- 172.21.15.71:0/3107321638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99ec003970 0x7f99ec003d90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009ac0 tx=0x7f99ec0057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01566640 1 -- 172.21.15.71:0/3107321638 reap_dead start 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01d67640 1 -- 172.21.15.71:0/3107321638 shutdown_connections 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01d67640 1 -- 172.21.15.71:0/3107321638 >> 172.21.15.71:0/3107321638 conn(0x7f99fc0c7c70 msgr2=0x7f99fc0c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01d67640 1 -- 172.21.15.71:0/3107321638 shutdown_connections 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.535+0000 7f9a01d67640 1 -- 172.21.15.71:0/3107321638 wait complete. 2023-08-25T10:01:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.537+0000 7f9a01d67640 1 Processor -- start 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.537+0000 7f9a01d67640 1 -- start start 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.538+0000 7f9a01d67640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec003970 0x7f99fc14fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.538+0000 7f9a01d67640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 0x7f99fc155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.538+0000 7f9a01d67640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc155ba0 0x7f99fc142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.538+0000 7f9a01d67640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc1ecbc0 con 0x7f99fc154b00 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.538+0000 7f9a01d67640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc0caf90 con 0x7f99ec003970 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.538+0000 7f9a01d67640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc14d050 con 0x7f99fc155ba0 2023-08-25T10:01:05.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 0x7f99fc155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 0x7f99fc155660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39516/0 (socket says 172.21.15.71:39516) 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 -- 172.21.15.71:0/2481097712 learned_addr learned my addr 172.21.15.71:0/2481097712 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f9a00d65640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec003970 0x7f99fc14fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f9a01566640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc155ba0 0x7f99fc142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 -- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc155ba0 msgr2=0x7f99fc142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc155ba0 0x7f99fc142900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 -- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec003970 msgr2=0x7f99fc14fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec003970 0x7f99fc14fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 -- 172.21.15.71:0/2481097712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc1ecfe0 con 0x7f99fc154b00 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.539+0000 7f99fbfff640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 0x7f99fc155660 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f99e8011610 tx=0x7f99e8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.540+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8004220 con 0x7f99fc154b00 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.540+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8004380 con 0x7f99fc154b00 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.540+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc0cacb0 con 0x7f99fc154b00 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.540+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e800f930 con 0x7f99fc154b00 2023-08-25T10:01:05.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.540+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc1ec500 con 0x7f99fc154b00 2023-08-25T10:01:05.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.541+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc1ecd00 con 0x7f99fc154b00 2023-08-25T10:01:05.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.542+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f99e800fa90 con 0x7f99fc154b00 2023-08-25T10:01:05.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.544+0000 7f99f97fa640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f99e006a050 0x7f99e006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:05.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.544+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f99e8088cc0 con 0x7f99fc154b00 2023-08-25T10:01:05.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.544+0000 7f9a00d65640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f99e006a050 0x7f99e006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:05.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.545+0000 7f9a00d65640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f99e006a050 0x7f99e006c510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f99f0036000 tx=0x7f99f0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:05.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.548+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f99fc1ecd00 con 0x7f99fc154b00 2023-08-25T10:01:05.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.831+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99fc1ecd00 con 0x7f99fc154b00 2023-08-25T10:01:05.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.832+0000 7f99f97fa640 1 -- 172.21.15.71:0/2481097712 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+6535 (secure 0 0 0) 0x7f99fc1ecd00 con 0x7f99fc154b00 2023-08-25T10:01:05.834 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:05.834 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:00:59.832482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[2,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:05.835 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 86 2023-08-25T10:01:05.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f99e006a050 msgr2=0x7f99e006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:05.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f99e006a050 0x7f99e006c510 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f99f0036000 tx=0x7f99f0030000 comp rx=0 tx=0).stop 2023-08-25T10:01:05.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 msgr2=0x7f99fc155660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:05.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 0x7f99fc155660 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f99e8011610 tx=0x7f99e8011ad0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 shutdown_connections 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f99e006a050 0x7f99e006c510 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f99fc155ba0 0x7f99fc142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99fc154b00 0x7f99fc155660 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 --2- 172.21.15.71:0/2481097712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99ec003970 0x7f99fc14fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 >> 172.21.15.71:0/2481097712 conn(0x7f99fc0c7c70 msgr2=0x7f99fc0654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:05.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 shutdown_connections 2023-08-25T10:01:05.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:05.835+0000 7f9a01d67640 1 -- 172.21.15.71:0/2481097712 wait complete. 2023-08-25T10:01:05.856 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:07.858 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:08.100 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.098+0000 7f1f88917640 1 Processor -- start 2023-08-25T10:01:08.100 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.098+0000 7f1f88917640 1 -- start start 2023-08-25T10:01:08.102 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.099+0000 7f1f88917640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1f840cb190 0x7f1f840ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:08.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.099+0000 7f1f88917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f8415cd00 0x7f1f84158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:08.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.099+0000 7f1f88917640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840b1af0 con 0x7f1f840cb570 2023-08-25T10:01:08.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.099+0000 7f1f88917640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f840c98f0 con 0x7f1f840cb190 2023-08-25T10:01:08.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.099+0000 7f1f88917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8415d170 con 0x7f1f8415cd00 2023-08-25T10:01:08.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f837fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1f840cb190 0x7f1f840ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:08.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f82ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1f840cb570 0x7f1f840c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:36566/0 (socket says 172.21.15.71:36566) 2023-08-25T10:01:08.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f82ffd640 1 -- 172.21.15.71:0/2468875494 learned_addr learned my addr 172.21.15.71:0/2468875494 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:08.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f83fff640 1 --2- 172.21.15.71:0/2468875494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f8415cd00 0x7f1f84158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:08.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f837fe640 1 -- 172.21.15.71:0/2468875494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f8415cd00 msgr2=0x7f1f84158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:08.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f837fe640 1 --2- 172.21.15.71:0/2468875494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f8415cd00 0x7f1f84158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f837fe640 1 -- 172.21.15.71:0/2468875494 >> v1:172.21.15.71:6789/0 conn(0x7f1f840cb570 legacy=0x7f1f840c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:08.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f837fe640 1 -- 172.21.15.71:0/2468875494 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840b0fb0 con 0x7f1f840cb190 2023-08-25T10:01:08.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.100+0000 7f1f827fc640 1 -- 172.21.15.71:0/2468875494 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739065863 0 0) 0x7f1f840b1af0 con 0x7f1f840cb570 2023-08-25T10:01:08.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.101+0000 7f1f837fe640 1 --2- 172.21.15.71:0/2468875494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb190 0x7f1f840ca910 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1f740097a0 tx=0x7f1f7402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb5dc1f92365f54 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:08.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f827fc640 1 -- 172.21.15.71:0/2468875494 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74030800 con 0x7f1f840cb190 2023-08-25T10:01:08.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f827fc640 1 -- 172.21.15.71:0/2468875494 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f74030960 con 0x7f1f840cb190 2023-08-25T10:01:08.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f827fc640 1 -- 172.21.15.71:0/2468875494 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74030cf0 con 0x7f1f840cb190 2023-08-25T10:01:08.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f88917640 1 -- 172.21.15.71:0/2468875494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb190 msgr2=0x7f1f840ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:08.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f88917640 1 --2- 172.21.15.71:0/2468875494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb190 0x7f1f840ca910 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1f740097a0 tx=0x7f1f7402b940 comp rx=0 tx=0).stop 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f88917640 1 -- 172.21.15.71:0/2468875494 shutdown_connections 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f88917640 1 --2- 172.21.15.71:0/2468875494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f8415cd00 0x7f1f84158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f88917640 1 --2- 172.21.15.71:0/2468875494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb190 0x7f1f840ca910 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.102+0000 7f1f88917640 1 -- 172.21.15.71:0/2468875494 >> 172.21.15.71:0/2468875494 conn(0x7f1f8406ae20 msgr2=0x7f1f8406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.103+0000 7f1f88917640 1 -- 172.21.15.71:0/2468875494 shutdown_connections 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.103+0000 7f1f88917640 1 -- 172.21.15.71:0/2468875494 wait complete. 2023-08-25T10:01:08.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.105+0000 7f1f88917640 1 Processor -- start 2023-08-25T10:01:08.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.105+0000 7f1f88917640 1 -- start start 2023-08-25T10:01:08.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f88917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840cb190 0x7f1f840b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:08.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f88917640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 0x7f1f8414e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f88917640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415cd00 0x7f1f8414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f88917640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84153c00 con 0x7f1f840cb570 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f88917640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840cf320 con 0x7f1f8415cd00 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f88917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f841598f0 con 0x7f1f840cb190 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f82ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 0x7f1f8414e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f82ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 0x7f1f8414e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39534/0 (socket says 172.21.15.71:39534) 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f82ffd640 1 -- 172.21.15.71:0/1700019535 learned_addr learned my addr 172.21.15.71:0/1700019535 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:08.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.106+0000 7f1f837fe640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840cb190 0x7f1f840b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f83fff640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415cd00 0x7f1f8414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f82ffd640 1 -- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840cb190 msgr2=0x7f1f840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f82ffd640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840cb190 0x7f1f840b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f82ffd640 1 -- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415cd00 msgr2=0x7f1f8414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f82ffd640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415cd00 0x7f1f8414ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f82ffd640 1 -- 172.21.15.71:0/1700019535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74009400 con 0x7f1f840cb570 2023-08-25T10:01:08.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f82ffd640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 0x7f1f8414e820 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c4b0 tx=0x7f1f7800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:08.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780074e0 con 0x7f1f840cb570 2023-08-25T10:01:08.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f78007660 con 0x7f1f840cb570 2023-08-25T10:01:08.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f840ced60 con 0x7f1f840cb570 2023-08-25T10:01:08.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.107+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78004a20 con 0x7f1f840cb570 2023-08-25T10:01:08.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.108+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f84153890 con 0x7f1f840cb570 2023-08-25T10:01:08.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.109+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f84156850 con 0x7f1f840cb570 2023-08-25T10:01:08.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.110+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1f78004b80 con 0x7f1f840cb570 2023-08-25T10:01:08.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.111+0000 7f1f6bfff640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f7006a080 0x7f1f7006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:08.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.111+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f1f78087e60 con 0x7f1f840cb570 2023-08-25T10:01:08.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.111+0000 7f1f837fe640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f7006a080 0x7f1f7006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:08.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.112+0000 7f1f837fe640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f7006a080 0x7f1f7006c540 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f74036000 tx=0x7f1f7403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:08.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.116+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1f780531e0 con 0x7f1f840cb570 2023-08-25T10:01:08.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.358+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f841536f0 con 0x7f1f840cb570 2023-08-25T10:01:08.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.359+0000 7f1f6bfff640 1 -- 172.21.15.71:0/1700019535 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f1f841536f0 con 0x7f1f840cb570 2023-08-25T10:01:08.361 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:08.361 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:06.866417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:08.361 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 88 2023-08-25T10:01:08.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f7006a080 msgr2=0x7f1f7006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:08.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f7006a080 0x7f1f7006c540 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f74036000 tx=0x7f1f7403b040 comp rx=0 tx=0).stop 2023-08-25T10:01:08.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 msgr2=0x7f1f8414e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:08.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 0x7f1f8414e820 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800c4b0 tx=0x7f1f7800c970 comp rx=0 tx=0).stop 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 shutdown_connections 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f8415cd00 0x7f1f8414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1f7006a080 0x7f1f7006c540 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1f840cb570 0x7f1f8414e820 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 --2- 172.21.15.71:0/1700019535 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f840cb190 0x7f1f840b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 >> 172.21.15.71:0/1700019535 conn(0x7f1f8406ae20 msgr2=0x7f1f840c6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:08.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 shutdown_connections 2023-08-25T10:01:08.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:08.362+0000 7f1f88917640 1 -- 172.21.15.71:0/1700019535 wait complete. 2023-08-25T10:01:08.383 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:10.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:10.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.620+0000 7fb0759a4640 1 Processor -- start 2023-08-25T10:01:10.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.621+0000 7fb0759a4640 1 -- start start 2023-08-25T10:01:10.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.622+0000 7fb0759a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb0701542d0 0x7fb0701576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:10.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.622+0000 7fb0759a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb070069850 0x7fb0700c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:10.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.622+0000 7fb0759a4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb070157bf0 con 0x7fb0700c75c0 2023-08-25T10:01:10.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.622+0000 7fb0759a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0700c4b60 con 0x7fb0701542d0 2023-08-25T10:01:10.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.622+0000 7fb0759a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07006a130 con 0x7fb070069850 2023-08-25T10:01:10.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.623+0000 7fb0751a3640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb0700c75c0 0x7fb0700c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38072/0 (socket says 172.21.15.71:38072) 2023-08-25T10:01:10.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.623+0000 7fb0751a3640 1 -- 172.21.15.71:0/2834069660 learned_addr learned my addr 172.21.15.71:0/2834069660 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:10.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.623+0000 7fb0749a2640 1 --2- 172.21.15.71:0/2834069660 >> v2:172.21.15.164:3300/0 conn(0x7fb0701542d0 0x7fb0701576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:10.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.623+0000 7fb06ffff640 1 --2- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb070069850 0x7fb0700c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:10.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.623+0000 7fb06f7fe640 1 -- 172.21.15.71:0/2834069660 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831286446 0 0) 0x7fb068001490 con 0x7fb0700c75c0 2023-08-25T10:01:10.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.623+0000 7fb06f7fe640 1 -- 172.21.15.71:0/2834069660 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb060003520 con 0x7fb0700c75c0 2023-08-25T10:01:10.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.624+0000 7fb0749a2640 1 -- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb070069850 msgr2=0x7fb0700c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:10.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.624+0000 7fb0749a2640 1 --2- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb070069850 0x7fb0700c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.624+0000 7fb0749a2640 1 -- 172.21.15.71:0/2834069660 >> v1:172.21.15.71:6789/0 conn(0x7fb0700c75c0 legacy=0x7fb0700c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:10.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.624+0000 7fb0749a2640 1 -- 172.21.15.71:0/2834069660 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0701546b0 con 0x7fb0701542d0 2023-08-25T10:01:10.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.625+0000 7fb0749a2640 1 --2- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0701542d0 0x7fb0701576b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0640097a0 tx=0x7fb06402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd9c0a41e85e30e2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:10.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.626+0000 7fb06f7fe640 1 -- 172.21.15.71:0/2834069660 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06402d040 con 0x7fb0701542d0 2023-08-25T10:01:10.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.626+0000 7fb06f7fe640 1 -- 172.21.15.71:0/2834069660 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064031810 con 0x7fb0701542d0 2023-08-25T10:01:10.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.626+0000 7fb06f7fe640 1 -- 172.21.15.71:0/2834069660 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064031ba0 con 0x7fb0701542d0 2023-08-25T10:01:10.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.626+0000 7fb0759a4640 1 -- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0701542d0 msgr2=0x7fb0701576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:10.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0701542d0 0x7fb0701576b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb0640097a0 tx=0x7fb06402b8b0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 -- 172.21.15.71:0/2834069660 shutdown_connections 2023-08-25T10:01:10.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb070069850 0x7fb0700c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2834069660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0701542d0 0x7fb0701576b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 -- 172.21.15.71:0/2834069660 >> 172.21.15.71:0/2834069660 conn(0x7fb07006c280 msgr2=0x7fb07006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:10.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 -- 172.21.15.71:0/2834069660 shutdown_connections 2023-08-25T10:01:10.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.627+0000 7fb0759a4640 1 -- 172.21.15.71:0/2834069660 wait complete. 2023-08-25T10:01:10.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.629+0000 7fb0759a4640 1 Processor -- start 2023-08-25T10:01:10.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.629+0000 7fb0759a4640 1 -- start start 2023-08-25T10:01:10.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0759a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 0x7fb0700c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:10.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0759a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0700c75c0 0x7fb0700711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:10.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0759a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0701542d0 0x7fb0701ec000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:10.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0759a4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb070070940 con 0x7fb070069850 2023-08-25T10:01:10.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0759a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb070154a00 con 0x7fb0700c75c0 2023-08-25T10:01:10.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0759a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701eb8d0 con 0x7fb0701542d0 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0749a2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 0x7fb0700c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0749a2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 0x7fb0700c3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49332/0 (socket says 172.21.15.71:49332) 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0749a2640 1 -- 172.21.15.71:0/2220146633 learned_addr learned my addr 172.21.15.71:0/2220146633 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb06ffff640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0700c75c0 0x7fb0700711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.630+0000 7fb0751a3640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0701542d0 0x7fb0701ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0749a2640 1 -- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0701542d0 msgr2=0x7fb0701ec000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:10.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0749a2640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0701542d0 0x7fb0701ec000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0749a2640 1 -- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0700c75c0 msgr2=0x7fb0700711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:10.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0749a2640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0700c75c0 0x7fb0700711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0749a2640 1 -- 172.21.15.71:0/2220146633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb064009400 con 0x7fb070069850 2023-08-25T10:01:10.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0749a2640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 0x7fb0700c3670 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb064000c00 tx=0x7fb0640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:10.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06403ad60 con 0x7fb070069850 2023-08-25T10:01:10.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064044400 con 0x7fb070069850 2023-08-25T10:01:10.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064044590 con 0x7fb070069850 2023-08-25T10:01:10.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.631+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0701ebb70 con 0x7fb070069850 2023-08-25T10:01:10.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.632+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb070156dd0 con 0x7fb070069850 2023-08-25T10:01:10.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.633+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07004e420 con 0x7fb070069850 2023-08-25T10:01:10.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.634+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb06404d440 con 0x7fb070069850 2023-08-25T10:01:10.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.635+0000 7fb06d7fa640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb05406a050 0x7fb05406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:10.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.635+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb0640aac60 con 0x7fb070069850 2023-08-25T10:01:10.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.635+0000 7fb06ffff640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb05406a050 0x7fb05406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:10.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.636+0000 7fb06ffff640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb05406a050 0x7fb05406c510 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb0700ae780 tx=0x7fb05c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:10.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.640+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb064075e00 con 0x7fb070069850 2023-08-25T10:01:10.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.894+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb070069f10 con 0x7fb070069850 2023-08-25T10:01:10.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.899+0000 7fb06d7fa640 1 -- 172.21.15.71:0/2220146633 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fb064079420 con 0x7fb070069850 2023-08-25T10:01:10.901 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:10.902 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:09.883759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:10.902 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 89 2023-08-25T10:01:10.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb05406a050 msgr2=0x7fb05406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:10.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb05406a050 0x7fb05406c510 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb0700ae780 tx=0x7fb05c008040 comp rx=0 tx=0).stop 2023-08-25T10:01:10.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 msgr2=0x7fb0700c3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:10.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 0x7fb0700c3670 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fb064000c00 tx=0x7fb0640046c0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 shutdown_connections 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb0700c75c0 0x7fb0700711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb05406a050 0x7fb05406c510 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb0701542d0 0x7fb0701ec000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 --2- 172.21.15.71:0/2220146633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb070069850 0x7fb0700c3670 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.902+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 >> 172.21.15.71:0/2220146633 conn(0x7fb07006c280 msgr2=0x7fb07006c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.903+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 shutdown_connections 2023-08-25T10:01:10.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:10.903+0000 7fb0759a4640 1 -- 172.21.15.71:0/2220146633 wait complete. 2023-08-25T10:01:10.923 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:12.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:13.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.146+0000 7f0ec94c0640 1 Processor -- start 2023-08-25T10:01:13.149 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.147+0000 7f0ec94c0640 1 -- start start 2023-08-25T10:01:13.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.149+0000 7f0ec94c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec4152db0 0x7f0ec40c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.149+0000 7f0ec94c0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ec40c60e0 0x7f0ec40c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.149+0000 7f0ec94c0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ec4071be0 0x7f0ec4158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.149+0000 7f0ec94c0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec4158b90 con 0x7f0ec40c60e0 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.149+0000 7f0ec94c0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec40c5260 con 0x7f0ec4071be0 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.149+0000 7f0ec94c0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40c6580 con 0x7f0ec4152db0 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ec40c60e0 0x7f0ec40c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0ec40c60e0 0x7f0ec40c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49348/0 (socket says 172.21.15.71:49348) 2023-08-25T10:01:13.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 -- 172.21.15.71:0/1480407751 learned_addr learned my addr 172.21.15.71:0/1480407751 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:13.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec3fff640 1 --2- 172.21.15.71:0/1480407751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec4152db0 0x7f0ec40c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec8cbf640 1 --2- 172.21.15.71:0/1480407751 >> v2:172.21.15.164:3300/0 conn(0x7f0ec4071be0 0x7f0ec4158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 -- 172.21.15.71:0/1480407751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec4152db0 msgr2=0x7f0ec40c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 --2- 172.21.15.71:0/1480407751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec4152db0 0x7f0ec40c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 -- 172.21.15.71:0/1480407751 >> v2:172.21.15.164:3300/0 conn(0x7f0ec4071be0 msgr2=0x7f0ec4158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 --2- 172.21.15.71:0/1480407751 >> v2:172.21.15.164:3300/0 conn(0x7f0ec4071be0 0x7f0ec4158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec37fe640 1 -- 172.21.15.71:0/1480407751 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec415cc30 con 0x7f0ec40c60e0 2023-08-25T10:01:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.150+0000 7f0ec8cbf640 1 --2- 172.21.15.71:0/1480407751 >> v2:172.21.15.164:3300/0 conn(0x7f0ec4071be0 0x7f0ec4158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:13.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.152+0000 7f0ec37fe640 1 --2- 172.21.15.71:0/1480407751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40c60e0 0x7f0ec40c6d00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0ec406eb50 tx=0x7f0eb802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ecbf83806fb6d8c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:13.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.152+0000 7f0ec2ffd640 1 -- 172.21.15.71:0/1480407751 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8031880 con 0x7f0ec40c60e0 2023-08-25T10:01:13.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.152+0000 7f0ec2ffd640 1 -- 172.21.15.71:0/1480407751 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb80319e0 con 0x7f0ec40c60e0 2023-08-25T10:01:13.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.152+0000 7f0ec2ffd640 1 -- 172.21.15.71:0/1480407751 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8031d70 con 0x7f0ec40c60e0 2023-08-25T10:01:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 -- 172.21.15.71:0/1480407751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40c60e0 msgr2=0x7f0ec40c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/1480407751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40c60e0 0x7f0ec40c6d00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f0ec406eb50 tx=0x7f0eb802b920 comp rx=0 tx=0).stop 2023-08-25T10:01:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 -- 172.21.15.71:0/1480407751 shutdown_connections 2023-08-25T10:01:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/1480407751 >> v2:172.21.15.164:3300/0 conn(0x7f0ec4071be0 0x7f0ec4158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/1480407751 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec40c60e0 0x7f0ec40c6d00 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/1480407751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec4152db0 0x7f0ec40c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 -- 172.21.15.71:0/1480407751 >> 172.21.15.71:0/1480407751 conn(0x7f0ec406d0d0 msgr2=0x7f0ec40ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:13.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 -- 172.21.15.71:0/1480407751 shutdown_connections 2023-08-25T10:01:13.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.153+0000 7f0ec94c0640 1 -- 172.21.15.71:0/1480407751 wait complete. 2023-08-25T10:01:13.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.155+0000 7f0ec94c0640 1 Processor -- start 2023-08-25T10:01:13.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.155+0000 7f0ec94c0640 1 -- start start 2023-08-25T10:01:13.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.156+0000 7f0ec94c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ec4071be0 0x7f0ec41f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.156+0000 7f0ec94c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec40c60e0 0x7f0ec406fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.156+0000 7f0ec94c0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 0x7f0ec4153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.156+0000 7f0ec94c0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec40c9780 con 0x7f0ec4152db0 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.156+0000 7f0ec94c0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec41f17e0 con 0x7f0ec4071be0 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.156+0000 7f0ec94c0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec41553c0 con 0x7f0ec40c60e0 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 0x7f0ec4153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 0x7f0ec4153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49350/0 (socket says 172.21.15.71:49350) 2023-08-25T10:01:13.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 -- 172.21.15.71:0/2401587458 learned_addr learned my addr 172.21.15.71:0/2401587458 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:13.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec3fff640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ec4071be0 0x7f0ec41f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec37fe640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec40c60e0 0x7f0ec406fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 -- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec40c60e0 msgr2=0x7f0ec406fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec40c60e0 0x7f0ec406fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 -- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ec4071be0 msgr2=0x7f0ec41f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ec4071be0 0x7f0ec41f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec8cbf640 1 -- 172.21.15.71:0/2401587458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80094d0 con 0x7f0ec4152db0 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec37fe640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec40c60e0 0x7f0ec406fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:13.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.157+0000 7f0ec3fff640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ec4071be0 0x7f0ec41f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.158+0000 7f0ec8cbf640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 0x7f0ec4153e10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00cd50 tx=0x7f0ebc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.158+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc00f680 con 0x7f0ec4152db0 2023-08-25T10:01:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.158+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc00f800 con 0x7f0ec4152db0 2023-08-25T10:01:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.158+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec40c8f20 con 0x7f0ec4152db0 2023-08-25T10:01:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.158+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc00fbb0 con 0x7f0ec4152db0 2023-08-25T10:01:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.158+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec40c91c0 con 0x7f0ec4152db0 2023-08-25T10:01:13.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.159+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec41f1920 con 0x7f0ec4152db0 2023-08-25T10:01:13.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.161+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0ebc00b040 con 0x7f0ec4152db0 2023-08-25T10:01:13.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.162+0000 7f0ec0ff9640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0eb006a080 0x7f0eb006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:13.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.162+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f0ebc089410 con 0x7f0ec4152db0 2023-08-25T10:01:13.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.162+0000 7f0ec3fff640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0eb006a080 0x7f0eb006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:13.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.163+0000 7f0ec3fff640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0eb006a080 0x7f0eb006c540 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4006fd0 tx=0x7f0eb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:13.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.168+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0ec41f1920 con 0x7f0ec4152db0 2023-08-25T10:01:13.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.420+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ec41f1920 con 0x7f0ec4152db0 2023-08-25T10:01:13.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.421+0000 7f0ec0ff9640 1 -- 172.21.15.71:0/2401587458 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f0ec41f1920 con 0x7f0ec4152db0 2023-08-25T10:01:13.424 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:13.424 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:09.883759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:13.424 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 89 2023-08-25T10:01:13.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0eb006a080 msgr2=0x7f0eb006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:13.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0eb006a080 0x7f0eb006c540 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4006fd0 tx=0x7f0eb4008040 comp rx=0 tx=0).stop 2023-08-25T10:01:13.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 msgr2=0x7f0ec4153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:13.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 0x7f0ec4153e10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00cd50 tx=0x7f0ebc00a650 comp rx=0 tx=0).stop 2023-08-25T10:01:13.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 shutdown_connections 2023-08-25T10:01:13.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0eb006a080 0x7f0eb006c540 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ec4152db0 0x7f0ec4153e10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ec40c60e0 0x7f0ec406fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 --2- 172.21.15.71:0/2401587458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ec4071be0 0x7f0ec41f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:13.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 >> 172.21.15.71:0/2401587458 conn(0x7f0ec406d0d0 msgr2=0x7f0ec4158210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:13.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 shutdown_connections 2023-08-25T10:01:13.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:13.424+0000 7f0ec94c0640 1 -- 172.21.15.71:0/2401587458 wait complete. 2023-08-25T10:01:13.445 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:15.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:15.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.664+0000 7fc78359e640 1 Processor -- start 2023-08-25T10:01:15.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.664+0000 7fc78359e640 1 -- start start 2023-08-25T10:01:15.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc78359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc7840cb190 0x7fc7840ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:15.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc78359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc78415cd00 0x7fc784158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:15.667 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc78359e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7840b1af0 con 0x7fc7840cb570 2023-08-25T10:01:15.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc78359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7840c98f0 con 0x7fc78415cd00 2023-08-25T10:01:15.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc78359e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7840b0fb0 con 0x7fc7840cb190 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc782d9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc78415cd00 0x7fc784158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc781d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc7840cb570 0x7fc7840c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38080/0 (socket says 172.21.15.71:38080) 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc781d9b640 1 -- 172.21.15.71:0/210338746 learned_addr learned my addr 172.21.15.71:0/210338746 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.666+0000 7fc78259c640 1 --2- 172.21.15.71:0/210338746 >> v2:172.21.15.164:3300/0 conn(0x7fc7840cb190 0x7fc7840ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.667+0000 7fc782d9d640 1 -- 172.21.15.71:0/210338746 >> v2:172.21.15.164:3300/0 conn(0x7fc7840cb190 msgr2=0x7fc7840ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.667+0000 7fc782d9d640 1 --2- 172.21.15.71:0/210338746 >> v2:172.21.15.164:3300/0 conn(0x7fc7840cb190 0x7fc7840ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.667+0000 7fc782d9d640 1 -- 172.21.15.71:0/210338746 >> v1:172.21.15.71:6789/0 conn(0x7fc7840cb570 legacy=0x7fc7840c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:15.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.667+0000 7fc782d9d640 1 -- 172.21.15.71:0/210338746 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78415d130 con 0x7fc78415cd00 2023-08-25T10:01:15.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.668+0000 7fc782d9d640 1 --2- 172.21.15.71:0/210338746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158d80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc7840cc000 tx=0x7fc77c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e8ee3107394be2 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:15.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.668+0000 7fc78159a640 1 -- 172.21.15.71:0/210338746 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c02cd10 con 0x7fc78415cd00 2023-08-25T10:01:15.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.668+0000 7fc78159a640 1 -- 172.21.15.71:0/210338746 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c02ce70 con 0x7fc78415cd00 2023-08-25T10:01:15.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.668+0000 7fc78159a640 1 -- 172.21.15.71:0/210338746 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c034640 con 0x7fc78415cd00 2023-08-25T10:01:15.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 -- 172.21.15.71:0/210338746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 msgr2=0x7fc784158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:15.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 --2- 172.21.15.71:0/210338746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158d80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc7840cc000 tx=0x7fc77c02bf60 comp rx=0 tx=0).stop 2023-08-25T10:01:15.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 -- 172.21.15.71:0/210338746 shutdown_connections 2023-08-25T10:01:15.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 --2- 172.21.15.71:0/210338746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158d80 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 --2- 172.21.15.71:0/210338746 >> v2:172.21.15.164:3300/0 conn(0x7fc7840cb190 0x7fc7840ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 -- 172.21.15.71:0/210338746 >> 172.21.15.71:0/210338746 conn(0x7fc78406ae20 msgr2=0x7fc78406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:15.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 -- 172.21.15.71:0/210338746 shutdown_connections 2023-08-25T10:01:15.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.669+0000 7fc78359e640 1 -- 172.21.15.71:0/210338746 wait complete. 2023-08-25T10:01:15.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.671+0000 7fc78359e640 1 Processor -- start 2023-08-25T10:01:15.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.671+0000 7fc78359e640 1 -- start start 2023-08-25T10:01:15.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc78359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7840cb190 0x7fc784157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:15.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc78359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7840cb570 0x7fc7841560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:15.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc78359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:15.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc78359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc784157250 con 0x7fc78415cd00 2023-08-25T10:01:15.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc78359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc784156b90 con 0x7fc7840cb570 2023-08-25T10:01:15.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc78359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc784158680 con 0x7fc7840cb190 2023-08-25T10:01:15.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc782d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:15.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc782d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49368/0 (socket says 172.21.15.71:49368) 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc782d9d640 1 -- 172.21.15.71:0/827588734 learned_addr learned my addr 172.21.15.71:0/827588734 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.672+0000 7fc781d9b640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7840cb570 0x7fc7841560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc78259c640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7840cb190 0x7fc784157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc782d9d640 1 -- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7840cb190 msgr2=0x7fc784157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc782d9d640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7840cb190 0x7fc784157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc782d9d640 1 -- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7840cb570 msgr2=0x7fc7841560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:15.676 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc782d9d640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7840cb570 0x7fc7841560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc782d9d640 1 -- 172.21.15.71:0/827588734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c009a70 con 0x7fc78415cd00 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc782d9d640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158b30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc77c02c570 tx=0x7fc77c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c03d600 con 0x7fc78415cd00 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c03d760 con 0x7fc78415cd00 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7841546d0 con 0x7fc78415cd00 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.673+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c03daf0 con 0x7fc78415cd00 2023-08-25T10:01:15.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.674+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc784154c90 con 0x7fc78415cd00 2023-08-25T10:01:15.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.675+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc784158850 con 0x7fc78415cd00 2023-08-25T10:01:15.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.676+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc77c03b040 con 0x7fc78415cd00 2023-08-25T10:01:15.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.677+0000 7fc772ffd640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc76406a050 0x7fc76406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:15.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.677+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc77c0ab160 con 0x7fc78415cd00 2023-08-25T10:01:15.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.677+0000 7fc78259c640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc76406a050 0x7fc76406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:15.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.678+0000 7fc78259c640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc76406a050 0x7fc76406c510 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc774006fd0 tx=0x7fc774008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:15.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.682+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc784158850 con 0x7fc78415cd00 2023-08-25T10:01:15.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.955+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc784158850 con 0x7fc78415cd00 2023-08-25T10:01:15.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.956+0000 7fc772ffd640 1 -- 172.21.15.71:0/827588734 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fc784158850 con 0x7fc78415cd00 2023-08-25T10:01:15.958 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:15.958 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:09.883759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:15.958 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 89 2023-08-25T10:01:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.958+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc76406a050 msgr2=0x7fc76406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.958+0000 7fc78359e640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc76406a050 0x7fc76406c510 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc774006fd0 tx=0x7fc774008040 comp rx=0 tx=0).stop 2023-08-25T10:01:15.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.958+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 msgr2=0x7fc784158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158b30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc77c02c570 tx=0x7fc77c004a40 comp rx=0 tx=0).stop 2023-08-25T10:01:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 shutdown_connections 2023-08-25T10:01:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7840cb570 0x7fc7841560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc76406a050 0x7fc76406c510 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc78415cd00 0x7fc784158b30 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 --2- 172.21.15.71:0/827588734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7840cb190 0x7fc784157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:15.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 >> 172.21.15.71:0/827588734 conn(0x7fc78406ae20 msgr2=0x7fc784072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:15.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 shutdown_connections 2023-08-25T10:01:15.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:15.959+0000 7fc78359e640 1 -- 172.21.15.71:0/827588734 wait complete. 2023-08-25T10:01:15.982 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:01:15.984 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:01:09.883759+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:01:17.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:18.223 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.222+0000 7feb1f59e640 1 Processor -- start 2023-08-25T10:01:18.224 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.222+0000 7feb1f59e640 1 -- start start 2023-08-25T10:01:18.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.224+0000 7feb1f59e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feb2006d3b0 0x7feb200c7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:18.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.224+0000 7feb1f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb200c60b0 0x7feb200c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1f59e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200c7b70 con 0x7feb2006d790 2023-08-25T10:01:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1f59e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb20159c90 con 0x7feb2006d3b0 2023-08-25T10:01:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb200aee20 con 0x7feb200c60b0 2023-08-25T10:01:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feb2006d3b0 0x7feb200c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7feb2006d3b0 0x7feb200c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49384/0 (socket says 172.21.15.71:49384) 2023-08-25T10:01:18.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 -- 172.21.15.71:0/3560319831 learned_addr learned my addr 172.21.15.71:0/3560319831 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:18.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1ed9d640 1 --2- 172.21.15.71:0/3560319831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb200c60b0 0x7feb200c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:18.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 -- 172.21.15.71:0/3560319831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb200c60b0 msgr2=0x7feb200c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 --2- 172.21.15.71:0/3560319831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb200c60b0 0x7feb200c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 -- 172.21.15.71:0/3560319831 >> v1:172.21.15.164:6789/0 conn(0x7feb2006d790 legacy=0x7feb200b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.225+0000 7feb1e59c640 1 -- 172.21.15.71:0/3560319831 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200c6a80 con 0x7feb2006d3b0 2023-08-25T10:01:18.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.227+0000 7feb1e59c640 1 --2- 172.21.15.71:0/3560319831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb200c7150 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7feb10009870 tx=0x7feb1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b2f9d9a598268c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:18.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.227+0000 7feb1d59a640 1 -- 172.21.15.71:0/3560319831 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10030840 con 0x7feb2006d3b0 2023-08-25T10:01:18.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.227+0000 7feb1d59a640 1 -- 172.21.15.71:0/3560319831 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb100309a0 con 0x7feb2006d3b0 2023-08-25T10:01:18.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.227+0000 7feb1d59a640 1 -- 172.21.15.71:0/3560319831 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10030d30 con 0x7feb2006d3b0 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 -- 172.21.15.71:0/3560319831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 msgr2=0x7feb200c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 --2- 172.21.15.71:0/3560319831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb200c7150 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7feb10009870 tx=0x7feb1002b980 comp rx=0 tx=0).stop 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 -- 172.21.15.71:0/3560319831 shutdown_connections 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 --2- 172.21.15.71:0/3560319831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb200c60b0 0x7feb200c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 --2- 172.21.15.71:0/3560319831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb200c7150 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 -- 172.21.15.71:0/3560319831 >> 172.21.15.71:0/3560319831 conn(0x7feb200cb4b0 msgr2=0x7feb200cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 -- 172.21.15.71:0/3560319831 shutdown_connections 2023-08-25T10:01:18.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.228+0000 7feb1f59e640 1 -- 172.21.15.71:0/3560319831 wait complete. 2023-08-25T10:01:18.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.230+0000 7feb1f59e640 1 Processor -- start 2023-08-25T10:01:18.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.230+0000 7feb1f59e640 1 -- start start 2023-08-25T10:01:18.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.231+0000 7feb1f59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb201ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:18.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.231+0000 7feb1f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb2006d790 0x7feb200c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:18.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.231+0000 7feb1f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb200c60b0 0x7feb2006bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:18.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.231+0000 7feb1f59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200c0a10 con 0x7feb2006d3b0 2023-08-25T10:01:18.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.231+0000 7feb1f59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2014e5a0 con 0x7feb200c60b0 2023-08-25T10:01:18.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.231+0000 7feb1f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2014e7f0 con 0x7feb2006d790 2023-08-25T10:01:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb201ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb201ee2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49394/0 (socket says 172.21.15.71:49394) 2023-08-25T10:01:18.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 -- 172.21.15.71:0/1007229474 learned_addr learned my addr 172.21.15.71:0/1007229474 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:18.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1ed9d640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb200c60b0 0x7feb2006bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:18.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1dd9b640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb2006d790 0x7feb200c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:18.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 -- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb2006d790 msgr2=0x7feb200c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:18.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb2006d790 0x7feb200c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 -- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb200c60b0 msgr2=0x7feb2006bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:18.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb200c60b0 0x7feb2006bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.232+0000 7feb1e59c640 1 -- 172.21.15.71:0/1007229474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100094d0 con 0x7feb2006d3b0 2023-08-25T10:01:18.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.233+0000 7feb1e59c640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb201ee2a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7feb10009870 tx=0x7feb10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:18.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.233+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10039a90 con 0x7feb2006d3b0 2023-08-25T10:01:18.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.233+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10039bf0 con 0x7feb2006d3b0 2023-08-25T10:01:18.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.233+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb100433f0 con 0x7feb2006d3b0 2023-08-25T10:01:18.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.233+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2014ead0 con 0x7feb2006d3b0 2023-08-25T10:01:18.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.233+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2014f090 con 0x7feb2006d3b0 2023-08-25T10:01:18.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.234+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2004e420 con 0x7feb2006d3b0 2023-08-25T10:01:18.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.235+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7feb10042390 con 0x7feb2006d3b0 2023-08-25T10:01:18.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.237+0000 7feb0effd640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb0006a050 0x7feb0006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:18.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.237+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7feb100aacc0 con 0x7feb2006d3b0 2023-08-25T10:01:18.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.237+0000 7feb1dd9b640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb0006a050 0x7feb0006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:18.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.238+0000 7feb1dd9b640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb0006a050 0x7feb0006c510 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7feb2006ce50 tx=0x7feb14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:18.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.241+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7feb10075ef0 con 0x7feb2006d3b0 2023-08-25T10:01:18.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.470+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb200734c0 con 0x7feb2006d3b0 2023-08-25T10:01:18.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.471+0000 7feb0effd640 1 -- 172.21.15.71:0/1007229474 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6530 (secure 0 0 0) 0x7feb10048090 con 0x7feb2006d3b0 2023-08-25T10:01:18.473 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:18.473 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:09.883759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:18.473 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 89 2023-08-25T10:01:18.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb0006a050 msgr2=0x7feb0006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:18.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb0006a050 0x7feb0006c510 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7feb2006ce50 tx=0x7feb14009040 comp rx=0 tx=0).stop 2023-08-25T10:01:18.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 msgr2=0x7feb201ee2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:18.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb201ee2a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7feb10009870 tx=0x7feb10004730 comp rx=0 tx=0).stop 2023-08-25T10:01:18.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 shutdown_connections 2023-08-25T10:01:18.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb200c60b0 0x7feb2006bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb0006a050 0x7feb0006c510 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb2006d790 0x7feb200c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 --2- 172.21.15.71:0/1007229474 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb2006d3b0 0x7feb201ee2a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:18.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.473+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 >> 172.21.15.71:0/1007229474 conn(0x7feb200cb4b0 msgr2=0x7feb200c9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:18.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.474+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 shutdown_connections 2023-08-25T10:01:18.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:18.474+0000 7feb1f59e640 1 -- 172.21.15.71:0/1007229474 wait complete. 2023-08-25T10:01:18.493 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:20.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:20.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.726+0000 7f6c1e88a640 1 Processor -- start 2023-08-25T10:01:20.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.727+0000 7f6c1e88a640 1 -- start start 2023-08-25T10:01:20.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.728+0000 7f6c1e88a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c180ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1e88a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6c1815cd00 0x7f6c18158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1e88a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c180b1af0 con 0x7f6c180cb570 2023-08-25T10:01:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1e88a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c180c98f0 con 0x7f6c1815cd00 2023-08-25T10:01:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1e88a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1815d190 con 0x7f6c180cb190 2023-08-25T10:01:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1d087640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6c180cb570 0x7f6c180c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37050/0 (socket says 172.21.15.71:37050) 2023-08-25T10:01:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1d087640 1 -- 172.21.15.71:0/312440992 learned_addr learned my addr 172.21.15.71:0/312440992 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1d888640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c180ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.729+0000 7f6c1e089640 1 --2- 172.21.15.71:0/312440992 >> v2:172.21.15.164:3300/0 conn(0x7f6c1815cd00 0x7f6c18158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13339005 0 0) 0x7f6c180b1af0 con 0x7f6c180cb570 2023-08-25T10:01:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c00003520 con 0x7f6c180cb570 2023-08-25T10:01:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300301205 0 0) 0x7f6c00003520 con 0x7f6c180cb570 2023-08-25T10:01:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 msgr2=0x7f6c180ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c180ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 >> v2:172.21.15.164:3300/0 conn(0x7f6c1815cd00 msgr2=0x7f6c18158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 --2- 172.21.15.71:0/312440992 >> v2:172.21.15.164:3300/0 conn(0x7f6c1815cd00 0x7f6c18158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180b0fb0 con 0x7f6c180cb570 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f6c140019f0 con 0x7f6c180cb570 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 >> v1:172.21.15.71:6789/0 conn(0x7f6c180cb570 legacy=0x7f6c180c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c000039e0 0x7f6c00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c000047c0 0x7f6c00004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c180c98f0 con 0x7f6c000047c0 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1815d190 con 0x7f6c000039e0 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0000b9d0 con 0x7f6c180cb190 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.730+0000 7f6c1d087640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c000047c0 0x7f6c00004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d888640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c000039e0 0x7f6c00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1e089640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d087640 1 -- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 msgr2=0x7f6c00005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:20.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d087640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c00005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d087640 1 -- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c000039e0 msgr2=0x7f6c00003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d087640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c000039e0 0x7f6c00003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d087640 1 -- 172.21.15.71:0/312440992 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c00003520 con 0x7f6c000047c0 2023-08-25T10:01:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1d888640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c000039e0 0x7f6c00003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:20.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.731+0000 7f6c1e089640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c180cb190 0x7f6c00005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:20.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.732+0000 7f6c1d087640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c000047c0 0x7f6c00004c40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400a860 tx=0x7f6c000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ea7e74a27ee0a6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.732+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14034630 con 0x7f6c000047c0 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.732+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1402f070 con 0x7f6c000047c0 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.732+0000 7f6c1c886640 1 -- 172.21.15.71:0/312440992 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140349f0 con 0x7f6c000047c0 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e88a640 1 -- 172.21.15.71:0/312440992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c000047c0 msgr2=0x7f6c00004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/312440992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c000047c0 0x7f6c00004c40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400a860 tx=0x7f6c000057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e089640 1 -- 172.21.15.71:0/312440992 reap_dead start 2023-08-25T10:01:20.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e88a640 1 -- 172.21.15.71:0/312440992 shutdown_connections 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e88a640 1 -- 172.21.15.71:0/312440992 >> 172.21.15.71:0/312440992 conn(0x7f6c1806ae20 msgr2=0x7f6c1806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e88a640 1 -- 172.21.15.71:0/312440992 shutdown_connections 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.733+0000 7f6c1e88a640 1 -- 172.21.15.71:0/312440992 wait complete. 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.735+0000 7f6c1e88a640 1 Processor -- start 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.735+0000 7f6c1e88a640 1 -- start start 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e88a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c000047c0 0x7f6c18156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e88a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c18156bf0 0x7f6c180c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e88a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 0x7f6c1814e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e88a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c181549b0 con 0x7f6c181f0d50 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e88a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18155700 con 0x7f6c18156bf0 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e88a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c180b1bd0 con 0x7f6c000047c0 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.736+0000 7f6c1e089640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 0x7f6c1814e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 0x7f6c1814e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55574/0 (socket says 172.21.15.71:55574) 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 -- 172.21.15.71:0/1345716127 learned_addr learned my addr 172.21.15.71:0/1345716127 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1d888640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c000047c0 0x7f6c18156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1d087640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c18156bf0 0x7f6c180c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 -- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c000047c0 msgr2=0x7f6c18156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c000047c0 0x7f6c18156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 -- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c18156bf0 msgr2=0x7f6c180c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c18156bf0 0x7f6c180c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 -- 172.21.15.71:0/1345716127 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180cdd10 con 0x7f6c181f0d50 2023-08-25T10:01:20.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.737+0000 7f6c1e089640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 0x7f6c1814e820 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6c1815d0e0 tx=0x7f6c08010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:20.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.738+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0800e8a0 con 0x7f6c181f0d50 2023-08-25T10:01:20.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.738+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0800ea00 con 0x7f6c181f0d50 2023-08-25T10:01:20.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.738+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0800ed90 con 0x7f6c181f0d50 2023-08-25T10:01:20.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.738+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c18154c20 con 0x7f6c181f0d50 2023-08-25T10:01:20.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.738+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c181551e0 con 0x7f6c181f0d50 2023-08-25T10:01:20.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.739+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1804e420 con 0x7f6c181f0d50 2023-08-25T10:01:20.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.740+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6c08020410 con 0x7f6c181f0d50 2023-08-25T10:01:20.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.742+0000 7f6c0e7fc640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c0406a050 0x7f6c0406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:20.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.742+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6c08088450 con 0x7f6c181f0d50 2023-08-25T10:01:20.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.742+0000 7f6c1d888640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c0406a050 0x7f6c0406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:20.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.743+0000 7f6c1d888640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c0406a050 0x7f6c0406c510 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6c100099c0 tx=0x7f6c10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:20.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:20.747+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6c08053600 con 0x7f6c181f0d50 2023-08-25T10:01:21.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.009+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c180c9df0 con 0x7f6c181f0d50 2023-08-25T10:01:21.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.010+0000 7f6c0e7fc640 1 -- 172.21.15.71:0/1345716127 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f6c08056c20 con 0x7f6c181f0d50 2023-08-25T10:01:21.012 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:21.012 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:09.883759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:21.013 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 89 2023-08-25T10:01:21.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c0406a050 msgr2=0x7f6c0406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:21.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c0406a050 0x7f6c0406c510 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6c100099c0 tx=0x7f6c10008040 comp rx=0 tx=0).stop 2023-08-25T10:01:21.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 msgr2=0x7f6c1814e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:21.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 0x7f6c1814e820 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6c1815d0e0 tx=0x7f6c08010c20 comp rx=0 tx=0).stop 2023-08-25T10:01:21.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 shutdown_connections 2023-08-25T10:01:21.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c18156bf0 0x7f6c180c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:21.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6c0406a050 0x7f6c0406c510 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c181f0d50 0x7f6c1814e820 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 --2- 172.21.15.71:0/1345716127 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c000047c0 0x7f6c18156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 >> 172.21.15.71:0/1345716127 conn(0x7f6c1806ae20 msgr2=0x7f6c18152900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 shutdown_connections 2023-08-25T10:01:21.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:21.013+0000 7f6c1e88a640 1 -- 172.21.15.71:0/1345716127 wait complete. 2023-08-25T10:01:21.033 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:23.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:23.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.245+0000 7fb21ec8e640 1 Processor -- start 2023-08-25T10:01:23.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.245+0000 7fb21ec8e640 1 -- start start 2023-08-25T10:01:23.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ec8e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb218152d80 0x7fb2180c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:23.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ec8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb2180c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:23.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ec8e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2181591c0 con 0x7fb21815c740 2023-08-25T10:01:23.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ec8e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2181594a0 con 0x7fb218152d80 2023-08-25T10:01:23.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ec8e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb218159780 con 0x7fb2180cacb0 2023-08-25T10:01:23.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ca2a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb218152d80 0x7fb2180c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:23.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ca2a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fb218152d80 0x7fb2180c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55580/0 (socket says 172.21.15.71:55580) 2023-08-25T10:01:23.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.247+0000 7fb21ca2a640 1 -- 172.21.15.71:0/3713462628 learned_addr learned my addr 172.21.15.71:0/3713462628 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:23.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.248+0000 7fb217fff640 1 --2- 172.21.15.71:0/3713462628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb2180c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:23.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.248+0000 7fb21ca2a640 1 -- 172.21.15.71:0/3713462628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 msgr2=0x7fb2180c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:23.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.248+0000 7fb21ca2a640 1 --2- 172.21.15.71:0/3713462628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb2180c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.248+0000 7fb21ca2a640 1 -- 172.21.15.71:0/3713462628 >> v1:172.21.15.164:6789/0 conn(0x7fb21815c740 legacy=0x7fb21815cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:23.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.248+0000 7fb21ca2a640 1 -- 172.21.15.71:0/3713462628 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb218159a60 con 0x7fb218152d80 2023-08-25T10:01:23.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.249+0000 7fb21ca2a640 1 --2- 172.21.15.71:0/3713462628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb218152d80 0x7fb2180c9360 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009870 tx=0x7fb20c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3724b3543860cfca server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:23.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.249+0000 7fb2177fe640 1 -- 172.21.15.71:0/3713462628 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c030840 con 0x7fb218152d80 2023-08-25T10:01:23.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb2177fe640 1 -- 172.21.15.71:0/3713462628 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c0309a0 con 0x7fb218152d80 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb2177fe640 1 -- 172.21.15.71:0/3713462628 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c030d30 con 0x7fb218152d80 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3713462628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb218152d80 msgr2=0x7fb2180c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3713462628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb218152d80 0x7fb2180c9360 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb20c009870 tx=0x7fb20c02b980 comp rx=0 tx=0).stop 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3713462628 shutdown_connections 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3713462628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb2180c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3713462628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb218152d80 0x7fb2180c9360 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3713462628 >> 172.21.15.71:0/3713462628 conn(0x7fb21806b8f0 msgr2=0x7fb2180654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:23.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3713462628 shutdown_connections 2023-08-25T10:01:23.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.250+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3713462628 wait complete. 2023-08-25T10:01:23.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.252+0000 7fb21ec8e640 1 Processor -- start 2023-08-25T10:01:23.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.252+0000 7fb21ec8e640 1 -- start start 2023-08-25T10:01:23.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.253+0000 7fb21ec8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb21815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:23.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.253+0000 7fb21ec8e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb218152d80 0x7fb218158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:23.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.253+0000 7fb21ec8e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 0x7fb218146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:23.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.253+0000 7fb21ec8e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2181ee760 con 0x7fb21815c740 2023-08-25T10:01:23.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.253+0000 7fb21ec8e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2181f2780 con 0x7fb218152d80 2023-08-25T10:01:23.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.253+0000 7fb21ec8e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2181f2a60 con 0x7fb2180cacb0 2023-08-25T10:01:23.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 0x7fb218146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:23.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 0x7fb218146d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55596/0 (socket says 172.21.15.71:55596) 2023-08-25T10:01:23.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 -- 172.21.15.71:0/3101636920 learned_addr learned my addr 172.21.15.71:0/3101636920 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:23.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21ca2a640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb21815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:23.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb217fff640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb218152d80 0x7fb218158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:23.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 -- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 msgr2=0x7fb21815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:23.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb21815ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 -- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb218152d80 msgr2=0x7fb218158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:23.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb218152d80 0x7fb218158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 -- 172.21.15.71:0/3101636920 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0094d0 con 0x7fb21815c740 2023-08-25T10:01:23.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb217fff640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb218152d80 0x7fb218158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:23.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.254+0000 7fb21d22b640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 0x7fb218146d50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb21000b970 tx=0x7fb21000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:23.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.255+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21000cb40 con 0x7fb21815c740 2023-08-25T10:01:23.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.255+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21000cca0 con 0x7fb21815c740 2023-08-25T10:01:23.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.255+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210014680 con 0x7fb21815c740 2023-08-25T10:01:23.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.255+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2181eea20 con 0x7fb21815c740 2023-08-25T10:01:23.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.255+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2181eefe0 con 0x7fb21815c740 2023-08-25T10:01:23.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.256+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2180b1a60 con 0x7fb21815c740 2023-08-25T10:01:23.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.257+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb21001d3b0 con 0x7fb21815c740 2023-08-25T10:01:23.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.259+0000 7fb2157fa640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1fc06a050 0x7fb1fc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:23.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.259+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb2100889b0 con 0x7fb21815c740 2023-08-25T10:01:23.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.259+0000 7fb21ca2a640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1fc06a050 0x7fb1fc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:23.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.260+0000 7fb21ca2a640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1fc06a050 0x7fb1fc06c510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb20c036000 tx=0x7fb20c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:23.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.264+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb210053ab0 con 0x7fb21815c740 2023-08-25T10:01:23.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.534+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2181ef200 con 0x7fb21815c740 2023-08-25T10:01:23.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.535+0000 7fb2157fa640 1 -- 172.21.15.71:0/3101636920 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+6530 (secure 0 0 0) 0x7fb2100570d0 con 0x7fb21815c740 2023-08-25T10:01:23.537 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:23.537 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:09.883759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:23.538 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 89 2023-08-25T10:01:23.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.537+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1fc06a050 msgr2=0x7fb1fc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:23.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.537+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1fc06a050 0x7fb1fc06c510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb20c036000 tx=0x7fb20c008030 comp rx=0 tx=0).stop 2023-08-25T10:01:23.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.537+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 msgr2=0x7fb218146d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:23.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.537+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 0x7fb218146d50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb21000b970 tx=0x7fb21000be30 comp rx=0 tx=0).stop 2023-08-25T10:01:23.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 shutdown_connections 2023-08-25T10:01:23.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb218152d80 0x7fb218158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb1fc06a050 0x7fb1fc06c510 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb21815c740 0x7fb218146d50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 --2- 172.21.15.71:0/3101636920 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2180cacb0 0x7fb21815ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:23.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 >> 172.21.15.71:0/3101636920 conn(0x7fb21806b8f0 msgr2=0x7fb2180c3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:23.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 shutdown_connections 2023-08-25T10:01:23.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:23.538+0000 7fb21ec8e640 1 -- 172.21.15.71:0/3101636920 wait complete. 2023-08-25T10:01:23.556 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:25.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:25.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:01:25.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:01:25.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.786+0000 7f2a96bc4640 1 Processor -- start 2023-08-25T10:01:25.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.787+0000 7f2a96bc4640 1 -- start start 2023-08-25T10:01:25.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.788+0000 7f2a96bc4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2a900c0260 0x7f2a90164540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.788+0000 7f2a96bc4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a90159ae0 con 0x7f2a90152fa0 2023-08-25T10:01:25.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a96bc4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9006fa50 con 0x7f2a90152bc0 2023-08-25T10:01:25.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a96bc4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a90158f60 con 0x7f2a900c0260 2023-08-25T10:01:25.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a953c1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f2a90152fa0 0x7f2a9006fee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37058/0 (socket says 172.21.15.71:37058) 2023-08-25T10:01:25.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a953c1640 1 -- 172.21.15.71:0/1234572243 learned_addr learned my addr 172.21.15.71:0/1234572243 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:25.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a963c3640 1 --2- 172.21.15.71:0/1234572243 >> v2:172.21.15.164:3300/0 conn(0x7f2a900c0260 0x7f2a90164540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851549427 0 0) 0x7f2a90159ae0 con 0x7f2a90152fa0 2023-08-25T10:01:25.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.789+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c003520 con 0x7f2a90152fa0 2023-08-25T10:01:25.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219983786 0 0) 0x7f2a7c003520 con 0x7f2a90152fa0 2023-08-25T10:01:25.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 >> v2:172.21.15.164:3300/0 conn(0x7f2a900c0260 msgr2=0x7f2a90164540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:25.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 --2- 172.21.15.71:0/1234572243 >> v2:172.21.15.164:3300/0 conn(0x7f2a900c0260 0x7f2a90164540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 >> v1:172.21.15.164:6789/0 conn(0x7f2a90152bc0 legacy=0x7f2a900c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:25.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a90159240 con 0x7f2a90152fa0 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f2a80001670 con 0x7f2a90152fa0 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 >> v1:172.21.15.71:6789/0 conn(0x7f2a90152fa0 legacy=0x7f2a9006fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a7c0039e0 0x7f2a7c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a7c0047c0 0x7f2a7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c005180 0x7f2a7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a90158f60 con 0x7f2a7c0039e0 2023-08-25T10:01:25.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c00c0b0 con 0x7f2a7c005180 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c00c270 con 0x7f2a7c0047c0 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.790+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a7c0039e0 0x7f2a7c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a953c1640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a7c0047c0 0x7f2a7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a95bc2640 1 -- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a7c0047c0 msgr2=0x7f2a7c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a7c0047c0 0x7f2a7c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a963c3640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c005180 0x7f2a7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a95bc2640 1 -- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c005180 msgr2=0x7f2a7c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:25.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c005180 0x7f2a7c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a95bc2640 1 -- 172.21.15.71:0/1234572243 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c003520 con 0x7f2a7c0039e0 2023-08-25T10:01:25.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.791+0000 7f2a963c3640 1 -- 172.21.15.71:0/1234572243 reap_dead start 2023-08-25T10:01:25.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.792+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a7c0039e0 0x7f2a7c003e00 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c001750 tx=0x7f2a7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771602802fa63d0b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:25.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.792+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c0052d0 con 0x7f2a7c0039e0 2023-08-25T10:01:25.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.792+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8c02e050 con 0x7f2a7c0039e0 2023-08-25T10:01:25.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.792+0000 7f2a94bc0640 1 -- 172.21.15.71:0/1234572243 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c003830 con 0x7f2a7c0039e0 2023-08-25T10:01:25.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1234572243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a7c0039e0 msgr2=0x7f2a7c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:25.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a7c0039e0 0x7f2a7c003e00 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c001750 tx=0x7f2a7c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1234572243 shutdown_connections 2023-08-25T10:01:25.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1234572243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a7c0039e0 0x7f2a7c003e00 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1234572243 >> 172.21.15.71:0/1234572243 conn(0x7f2a900cb260 msgr2=0x7f2a900cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:25.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1234572243 shutdown_connections 2023-08-25T10:01:25.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.793+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1234572243 wait complete. 2023-08-25T10:01:25.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.795+0000 7f2a96bc4640 1 Processor -- start 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.795+0000 7f2a96bc4640 1 -- start start 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a96bc4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c0039e0 0x7f2a901f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a96bc4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 0x7f2a901512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a96bc4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a9015c870 0x7f2a9014ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a96bc4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a900c3420 con 0x7f2a9015a750 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a96bc4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a900c2b80 con 0x7f2a7c0039e0 2023-08-25T10:01:25.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a96bc4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a900c2e60 con 0x7f2a9015c870 2023-08-25T10:01:25.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a953c1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 0x7f2a901512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a953c1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 0x7f2a901512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55608/0 (socket says 172.21.15.71:55608) 2023-08-25T10:01:25.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a953c1640 1 -- 172.21.15.71:0/1782656131 learned_addr learned my addr 172.21.15.71:0/1782656131 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:25.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a963c3640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a9015c870 0x7f2a9014ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.796+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c0039e0 0x7f2a901f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a953c1640 1 -- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a9015c870 msgr2=0x7f2a9014ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a953c1640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a9015c870 0x7f2a9014ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a953c1640 1 -- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c0039e0 msgr2=0x7f2a901f1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a953c1640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c0039e0 0x7f2a901f1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a953c1640 1 -- 172.21.15.71:0/1782656131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a900c3140 con 0x7f2a9015a750 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a953c1640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 0x7f2a901512b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000d720 tx=0x7f2a8000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a800074e0 con 0x7f2a9015a750 2023-08-25T10:01:25.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a80007640 con 0x7f2a9015a750 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9015c070 con 0x7f2a9015a750 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.797+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80004a00 con 0x7f2a9015a750 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.798+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a901f1f90 con 0x7f2a9015a750 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.799+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a9015bed0 con 0x7f2a9015a750 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.800+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2a80004b60 con 0x7f2a9015a750 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.801+0000 7f2a867fc640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7406a080 0x7f2a7406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:25.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.801+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f2a800883b0 con 0x7f2a9015a750 2023-08-25T10:01:25.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.802+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7406a080 0x7f2a7406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:25.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.802+0000 7f2a95bc2640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7406a080 0x7f2a7406c540 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c033000 tx=0x7f2a8c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:25.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:25.806+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2a9015bed0 con 0x7f2a9015a750 2023-08-25T10:01:26.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.063+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a9015bed0 con 0x7f2a9015a750 2023-08-25T10:01:26.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.064+0000 7f2a867fc640 1 -- 172.21.15.71:0/1782656131 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6528 (secure 0 0 0) 0x7f2a9015bed0 con 0x7f2a9015a750 2023-08-25T10:01:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:26.066 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:25.985303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:26.066 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 91 2023-08-25T10:01:26.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.066+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7406a080 msgr2=0x7f2a7406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:26.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.066+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7406a080 0x7f2a7406c540 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c033000 tx=0x7f2a8c009030 comp rx=0 tx=0).stop 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.066+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 msgr2=0x7f2a901512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.066+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 0x7f2a901512b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000d720 tx=0x7f2a8000dbe0 comp rx=0 tx=0).stop 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 shutdown_connections 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7406a080 0x7f2a7406c540 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a9015c870 0x7f2a9014ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a9015a750 0x7f2a901512b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:26.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 --2- 172.21.15.71:0/1782656131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a7c0039e0 0x7f2a901f1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 >> 172.21.15.71:0/1782656131 conn(0x7f2a900cb260 msgr2=0x7f2a900c0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 shutdown_connections 2023-08-25T10:01:26.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:26.067+0000 7f2a96bc4640 1 -- 172.21.15.71:0/1782656131 wait complete. 2023-08-25T10:01:26.087 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:28.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:28.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.316+0000 7f0464845640 1 Processor -- start 2023-08-25T10:01:28.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.316+0000 7f0464845640 1 -- start start 2023-08-25T10:01:28.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f0464845640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 0x7f04600c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f0464845640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04601591c0 con 0x7f04600cacb0 2023-08-25T10:01:28.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f0464845640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04601594a0 con 0x7f046015ce50 2023-08-25T10:01:28.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f0464845640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0460159780 con 0x7f0460152d80 2023-08-25T10:01:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f045dd9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f04600cacb0 0x7f046015c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37072/0 (socket says 172.21.15.71:37072) 2023-08-25T10:01:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f045dd9b640 1 -- 172.21.15.71:0/2978544540 learned_addr learned my addr 172.21.15.71:0/2978544540 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.318+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701883114 0 0) 0x7f04601591c0 con 0x7f04600cacb0 2023-08-25T10:01:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0448003520 con 0x7f04600cacb0 2023-08-25T10:01:28.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045e59c640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 0x7f04600c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:28.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069786350 0 0) 0x7f0448003520 con 0x7f04600cacb0 2023-08-25T10:01:28.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 msgr2=0x7f04600c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:28.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 0x7f04600c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 >> v1:172.21.15.164:6789/0 conn(0x7f046015ce50 legacy=0x7f04600cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460159a60 con 0x7f04600cacb0 2023-08-25T10:01:28.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0454001670 con 0x7f04600cacb0 2023-08-25T10:01:28.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 >> v1:172.21.15.71:6789/0 conn(0x7f04600cacb0 legacy=0x7f046015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0448003a10 0x7f0448003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 0x7f04480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f0448005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460159780 con 0x7f0448004d00 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044800bb20 con 0x7f0448003a10 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.319+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044800bce0 con 0x7f0460152d80 2023-08-25T10:01:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045ed9d640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f0448005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045dd9b640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 0x7f04480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045e59c640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0448003a10 0x7f0448003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045ed9d640 1 -- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 msgr2=0x7f04480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045ed9d640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460152d80 0x7f04480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045ed9d640 1 -- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0448003a10 msgr2=0x7f0448003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045ed9d640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0448003a10 0x7f0448003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.320+0000 7f045ed9d640 1 -- 172.21.15.71:0/2978544540 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448003520 con 0x7f0448004d00 2023-08-25T10:01:28.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.321+0000 7f045ed9d640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f0448005180 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0458001e20 tx=0x7f04480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39525349b2a5a41 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:28.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.322+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458003b70 con 0x7f0448004d00 2023-08-25T10:01:28.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.322+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0458003cd0 con 0x7f0448004d00 2023-08-25T10:01:28.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.322+0000 7f045d59a640 1 -- 172.21.15.71:0/2978544540 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458005310 con 0x7f0448004d00 2023-08-25T10:01:28.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.322+0000 7f0464845640 1 -- 172.21.15.71:0/2978544540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 msgr2=0x7f0448005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.322+0000 7f0464845640 1 --2- 172.21.15.71:0/2978544540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f0448005180 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0458001e20 tx=0x7f04480057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.322+0000 7f045ed9d640 1 -- 172.21.15.71:0/2978544540 reap_dead start 2023-08-25T10:01:28.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.323+0000 7f0464845640 1 -- 172.21.15.71:0/2978544540 shutdown_connections 2023-08-25T10:01:28.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.323+0000 7f0464845640 1 -- 172.21.15.71:0/2978544540 >> 172.21.15.71:0/2978544540 conn(0x7f046006b8f0 msgr2=0x7f04600654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:28.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.323+0000 7f0464845640 1 -- 172.21.15.71:0/2978544540 shutdown_connections 2023-08-25T10:01:28.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.323+0000 7f0464845640 1 -- 172.21.15.71:0/2978544540 wait complete. 2023-08-25T10:01:28.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.325+0000 7f0464845640 1 Processor -- start 2023-08-25T10:01:28.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.325+0000 7f0464845640 1 -- start start 2023-08-25T10:01:28.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f0464845640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f04600c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f0464845640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460144cd0 0x7f04601eea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f0464845640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04601eef50 0x7f0460141f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f0464845640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601442c0 con 0x7f0448004d00 2023-08-25T10:01:28.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f0464845640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601f2540 con 0x7f04601eef50 2023-08-25T10:01:28.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f0464845640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04601ee5c0 con 0x7f0460144cd0 2023-08-25T10:01:28.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f04600c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:28.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f04600c4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55626/0 (socket says 172.21.15.71:55626) 2023-08-25T10:01:28.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 -- 172.21.15.71:0/1566099439 learned_addr learned my addr 172.21.15.71:0/1566099439 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:28.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 -- 172.21.15.71:0/1566099439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460144cd0 msgr2=0x7f04601eea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460144cd0 0x7f04601eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 -- 172.21.15.71:0/1566099439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04601eef50 msgr2=0x7f0460141f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04601eef50 0x7f0460141f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.326+0000 7f045e59c640 1 -- 172.21.15.71:0/1566099439 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04601ef3d0 con 0x7f0448004d00 2023-08-25T10:01:28.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.327+0000 7f045e59c640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f04600c4990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0450011600 tx=0x7f0450011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:28.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.327+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450015870 con 0x7f0448004d00 2023-08-25T10:01:28.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.327+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04500159d0 con 0x7f0448004d00 2023-08-25T10:01:28.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.327+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450015d40 con 0x7f0448004d00 2023-08-25T10:01:28.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.327+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04601f2a30 con 0x7f0448004d00 2023-08-25T10:01:28.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.327+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04601f2750 con 0x7f0448004d00 2023-08-25T10:01:28.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.328+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0460143bb0 con 0x7f0448004d00 2023-08-25T10:01:28.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.330+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0450013430 con 0x7f0448004d00 2023-08-25T10:01:28.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.331+0000 7f044effd640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f044006a050 0x7f044006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:28.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.331+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f0450088f20 con 0x7f0448004d00 2023-08-25T10:01:28.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.331+0000 7f045dd9b640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f044006a050 0x7f044006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:28.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.332+0000 7f045dd9b640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f044006a050 0x7f044006c510 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0454001c90 tx=0x7f0454001b40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:28.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.336+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0450057b50 con 0x7f0448004d00 2023-08-25T10:01:28.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.601+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04600b1a60 con 0x7f0448004d00 2023-08-25T10:01:28.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.602+0000 7f044effd640 1 -- 172.21.15.71:0/1566099439 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f0450022020 con 0x7f0448004d00 2023-08-25T10:01:28.604 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:28.604 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:26.987746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:28.605 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 92 2023-08-25T10:01:28.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f044006a050 msgr2=0x7f044006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f044006a050 0x7f044006c510 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0454001c90 tx=0x7f0454001b40 comp rx=0 tx=0).stop 2023-08-25T10:01:28.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 msgr2=0x7f04600c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:28.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f04600c4990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0450011600 tx=0x7f0450011ac0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 shutdown_connections 2023-08-25T10:01:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04601eef50 0x7f0460141f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f044006a050 0x7f044006c510 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0460144cd0 0x7f04601eea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 --2- 172.21.15.71:0/1566099439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0448004d00 0x7f04600c4990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.605+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 >> 172.21.15.71:0/1566099439 conn(0x7f046006b8f0 msgr2=0x7f0460148220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.606+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 shutdown_connections 2023-08-25T10:01:28.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:28.606+0000 7f0464845640 1 -- 172.21.15.71:0/1566099439 wait complete. 2023-08-25T10:01:28.627 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:01:28.629 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 92, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 3 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:01:26.987746+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:01:30.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:30.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.840+0000 7feb4d657640 1 Processor -- start 2023-08-25T10:01:30.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.840+0000 7feb4d657640 1 -- start start 2023-08-25T10:01:30.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb4d657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 0x7feb480c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb4d657640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feb4806cc70 0x7feb4814f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb4d657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4806a960 con 0x7feb480c3940 2023-08-25T10:01:30.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb4d657640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4806ac20 con 0x7feb4806cc70 2023-08-25T10:01:30.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb4d657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4806af00 con 0x7feb481594d0 2023-08-25T10:01:30.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb477fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7feb480c3940 0x7feb481598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43478/0 (socket says 172.21.15.71:43478) 2023-08-25T10:01:30.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.842+0000 7feb477fe640 1 -- 172.21.15.71:0/472275937 learned_addr learned my addr 172.21.15.71:0/472275937 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:30.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb46ffd640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 0x7feb480c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb467fc640 1 --2- 172.21.15.71:0/472275937 >> v2:172.21.15.164:3300/0 conn(0x7feb4806cc70 0x7feb4814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076401217 0 0) 0x7feb4806a960 con 0x7feb480c3940 2023-08-25T10:01:30.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb30003520 con 0x7feb480c3940 2023-08-25T10:01:30.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3225585995 0 0) 0x7feb30003520 con 0x7feb480c3940 2023-08-25T10:01:30.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 msgr2=0x7feb480c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:30.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 0x7feb480c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 >> v2:172.21.15.164:3300/0 conn(0x7feb4806cc70 msgr2=0x7feb4814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:30.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 --2- 172.21.15.71:0/472275937 >> v2:172.21.15.164:3300/0 conn(0x7feb4806cc70 0x7feb4814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480c2e60 con 0x7feb480c3940 2023-08-25T10:01:30.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7feb400019f0 con 0x7feb480c3940 2023-08-25T10:01:30.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 >> v1:172.21.15.71:6789/0 conn(0x7feb480c3940 legacy=0x7feb481598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:30.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb300039e0 0x7feb30003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb300047c0 0x7feb30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 0x7feb30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4806ac20 con 0x7feb300039e0 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4806af00 con 0x7feb300047c0 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.843+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3000ba00 con 0x7feb481594d0 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb46ffd640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb300039e0 0x7feb30003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb467fc640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb300047c0 0x7feb30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb477fe640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 0x7feb30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb46ffd640 1 -- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 msgr2=0x7feb30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:30.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb46ffd640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb481594d0 0x7feb30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb46ffd640 1 -- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb300047c0 msgr2=0x7feb30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:30.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb46ffd640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb300047c0 0x7feb30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.844+0000 7feb46ffd640 1 -- 172.21.15.71:0/472275937 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30003520 con 0x7feb300039e0 2023-08-25T10:01:30.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.845+0000 7feb46ffd640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb300039e0 0x7feb30003e00 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7feb38009650 tx=0x7feb300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c6ea458d62f521 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:30.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3802e8c0 con 0x7feb300039e0 2023-08-25T10:01:30.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3802ea20 con 0x7feb300039e0 2023-08-25T10:01:30.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb45ffb640 1 -- 172.21.15.71:0/472275937 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3802edb0 con 0x7feb300039e0 2023-08-25T10:01:30.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb4d657640 1 -- 172.21.15.71:0/472275937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb300039e0 msgr2=0x7feb30003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:30.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb4d657640 1 --2- 172.21.15.71:0/472275937 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb300039e0 0x7feb30003e00 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7feb38009650 tx=0x7feb300057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb477fe640 1 -- 172.21.15.71:0/472275937 reap_dead start 2023-08-25T10:01:30.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb4d657640 1 -- 172.21.15.71:0/472275937 shutdown_connections 2023-08-25T10:01:30.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb4d657640 1 -- 172.21.15.71:0/472275937 >> 172.21.15.71:0/472275937 conn(0x7feb480654f0 msgr2=0x7feb48065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:30.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb4d657640 1 -- 172.21.15.71:0/472275937 shutdown_connections 2023-08-25T10:01:30.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.846+0000 7feb4d657640 1 -- 172.21.15.71:0/472275937 wait complete. 2023-08-25T10:01:30.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.848+0000 7feb4d657640 1 Processor -- start 2023-08-25T10:01:30.861 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.849+0000 7feb4d657640 1 -- start start 2023-08-25T10:01:30.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb4d657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb300039e0 0x7feb481f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb4d657640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 0x7feb481eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb4d657640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb481490a0 0x7feb48147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb4d657640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb481ee620 con 0x7feb48156940 2023-08-25T10:01:30.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb4d657640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb480c80f0 con 0x7feb481490a0 2023-08-25T10:01:30.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb4d657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480c8900 con 0x7feb300039e0 2023-08-25T10:01:30.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 0x7feb481eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 0x7feb481eea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41148/0 (socket says 172.21.15.71:41148) 2023-08-25T10:01:30.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 -- 172.21.15.71:0/1640290110 learned_addr learned my addr 172.21.15.71:0/1640290110 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:30.864 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb46ffd640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb300039e0 0x7feb481f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 -- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb300039e0 msgr2=0x7feb481f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:30.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb300039e0 0x7feb481f2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.865 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb477fe640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb481490a0 0x7feb48147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 -- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb481490a0 msgr2=0x7feb48147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:30.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb481490a0 0x7feb48147bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:30.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.850+0000 7feb467fc640 1 -- 172.21.15.71:0/1640290110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480c8340 con 0x7feb48156940 2023-08-25T10:01:30.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.851+0000 7feb467fc640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 0x7feb481eea40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00f600 tx=0x7feb3c00fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:30.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.851+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c014870 con 0x7feb48156940 2023-08-25T10:01:30.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.851+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c0149d0 con 0x7feb48156940 2023-08-25T10:01:30.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.851+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c014d40 con 0x7feb48156940 2023-08-25T10:01:30.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.851+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb480c8620 con 0x7feb48156940 2023-08-25T10:01:30.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.851+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb481eef80 con 0x7feb48156940 2023-08-25T10:01:30.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.852+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb481ee0a0 con 0x7feb48156940 2023-08-25T10:01:30.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.854+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7feb3c00d490 con 0x7feb48156940 2023-08-25T10:01:30.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.855+0000 7feb2f7fe640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb3406a050 0x7feb3406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:30.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.855+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7feb3c0894a0 con 0x7feb48156940 2023-08-25T10:01:30.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.855+0000 7feb46ffd640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb3406a050 0x7feb3406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:30.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.856+0000 7feb46ffd640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb3406a050 0x7feb3406c510 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feb3802e460 tx=0x7feb38032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:30.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:30.861+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7feb481ee0a0 con 0x7feb48156940 2023-08-25T10:01:31.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.116+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb481ee0a0 con 0x7feb48156940 2023-08-25T10:01:31.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.116+0000 7feb2f7fe640 1 -- 172.21.15.71:0/1640290110 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6527 (secure 0 0 0) 0x7feb481ee0a0 con 0x7feb48156940 2023-08-25T10:01:31.118 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:31.119 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:26.987746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:31.119 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 92 2023-08-25T10:01:31.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.119+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb3406a050 msgr2=0x7feb3406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:31.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.119+0000 7feb4d657640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb3406a050 0x7feb3406c510 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7feb3802e460 tx=0x7feb38032000 comp rx=0 tx=0).stop 2023-08-25T10:01:31.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.119+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 msgr2=0x7feb481eea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:31.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.119+0000 7feb4d657640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 0x7feb481eea40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00f600 tx=0x7feb3c00fac0 comp rx=0 tx=0).stop 2023-08-25T10:01:31.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 shutdown_connections 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb481490a0 0x7feb48147bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feb3406a050 0x7feb3406c510 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feb48156940 0x7feb481eea40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 --2- 172.21.15.71:0/1640290110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb300039e0 0x7feb481f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 >> 172.21.15.71:0/1640290110 conn(0x7feb480654f0 msgr2=0x7feb480c2020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 shutdown_connections 2023-08-25T10:01:31.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:31.120+0000 7feb4d657640 1 -- 172.21.15.71:0/1640290110 wait complete. 2023-08-25T10:01:31.140 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:33.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:33.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.361+0000 7f77e99cf640 1 Processor -- start 2023-08-25T10:01:33.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.361+0000 7f77e99cf640 1 -- start start 2023-08-25T10:01:33.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.363+0000 7f77e99cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e40cbdf0 0x7f77e406b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:33.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.363+0000 7f77e99cf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f77e406b8f0 0x7f77e415c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:33.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.363+0000 7f77e99cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e40c4ae0 con 0x7f77e4156640 2023-08-25T10:01:33.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.363+0000 7f77e99cf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40cad80 con 0x7f77e406b8f0 2023-08-25T10:01:33.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.363+0000 7f77e99cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c3c30 con 0x7f77e40cbdf0 2023-08-25T10:01:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f77e406b8f0 0x7f77e415c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:33.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e27fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e40cbdf0 0x7f77e406b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f77e406b8f0 0x7f77e415c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:46226/0 (socket says 172.21.15.71:46226) 2023-08-25T10:01:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e37fe640 1 -- 172.21.15.71:0/3981600567 learned_addr learned my addr 172.21.15.71:0/3981600567 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e27fc640 1 -- 172.21.15.71:0/3981600567 >> v2:172.21.15.164:3300/0 conn(0x7f77e406b8f0 msgr2=0x7f77e415c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e27fc640 1 --2- 172.21.15.71:0/3981600567 >> v2:172.21.15.164:3300/0 conn(0x7f77e406b8f0 0x7f77e415c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.364+0000 7f77e27fc640 1 -- 172.21.15.71:0/3981600567 >> v1:172.21.15.164:6789/0 conn(0x7f77e4156640 legacy=0x7f77e40c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:33.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.365+0000 7f77e27fc640 1 -- 172.21.15.71:0/3981600567 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40c3950 con 0x7f77e40cbdf0 2023-08-25T10:01:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.366+0000 7f77e27fc640 1 --2- 172.21.15.71:0/3981600567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e40cbdf0 0x7f77e406b3b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77d80097a0 tx=0x7f77d802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6d419e0c292e6e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.367+0000 7f77e1ffb640 1 -- 172.21.15.71:0/3981600567 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d802e070 con 0x7f77e40cbdf0 2023-08-25T10:01:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.367+0000 7f77e1ffb640 1 -- 172.21.15.71:0/3981600567 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d80315a0 con 0x7f77e40cbdf0 2023-08-25T10:01:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.367+0000 7f77e1ffb640 1 -- 172.21.15.71:0/3981600567 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8031930 con 0x7f77e40cbdf0 2023-08-25T10:01:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.367+0000 7f77e99cf640 1 -- 172.21.15.71:0/3981600567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e40cbdf0 msgr2=0x7f77e406b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.367+0000 7f77e99cf640 1 --2- 172.21.15.71:0/3981600567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e40cbdf0 0x7f77e406b3b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77d80097a0 tx=0x7f77d802b840 comp rx=0 tx=0).stop 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.368+0000 7f77e99cf640 1 -- 172.21.15.71:0/3981600567 shutdown_connections 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.368+0000 7f77e99cf640 1 --2- 172.21.15.71:0/3981600567 >> v2:172.21.15.164:3300/0 conn(0x7f77e406b8f0 0x7f77e415c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.368+0000 7f77e99cf640 1 --2- 172.21.15.71:0/3981600567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e40cbdf0 0x7f77e406b3b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.368+0000 7f77e99cf640 1 -- 172.21.15.71:0/3981600567 >> 172.21.15.71:0/3981600567 conn(0x7f77e406a5b0 msgr2=0x7f77e4070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.368+0000 7f77e99cf640 1 -- 172.21.15.71:0/3981600567 shutdown_connections 2023-08-25T10:01:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.368+0000 7f77e99cf640 1 -- 172.21.15.71:0/3981600567 wait complete. 2023-08-25T10:01:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.369+0000 7f77e99cf640 1 Processor -- start 2023-08-25T10:01:33.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.370+0000 7f77e99cf640 1 -- start start 2023-08-25T10:01:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.370+0000 7f77e99cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77e406b8f0 0x7f77e40c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:33.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e99cf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 0x7f77e4159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e99cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e4156640 0x7f77e414efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e99cf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e41591d0 con 0x7f77e40cbdf0 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e99cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c7910 con 0x7f77e406b8f0 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e99cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40c6690 con 0x7f77e4156640 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 0x7f77e4159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 0x7f77e4159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41160/0 (socket says 172.21.15.71:41160) 2023-08-25T10:01:33.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 -- 172.21.15.71:0/408616375 learned_addr learned my addr 172.21.15.71:0/408616375 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:33.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e37fe640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e4156640 0x7f77e414efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e2ffd640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77e406b8f0 0x7f77e40c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 -- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e4156640 msgr2=0x7f77e414efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e4156640 0x7f77e414efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 -- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77e406b8f0 msgr2=0x7f77e40c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77e406b8f0 0x7f77e40c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e27fc640 1 -- 172.21.15.71:0/408616375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8009400 con 0x7f77e40cbdf0 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.371+0000 7f77e2ffd640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77e406b8f0 0x7f77e40c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:33.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.372+0000 7f77e27fc640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 0x7f77e4159720 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f77d8034040 tx=0x7f77d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.372+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8032840 con 0x7f77e40cbdf0 2023-08-25T10:01:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.372+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d80329c0 con 0x7f77e40cbdf0 2023-08-25T10:01:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.372+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e4157b90 con 0x7f77e40cbdf0 2023-08-25T10:01:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.372+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8032d70 con 0x7f77e40cbdf0 2023-08-25T10:01:33.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.372+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e40b2880 con 0x7f77e40cbdf0 2023-08-25T10:01:33.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.374+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e4069160 con 0x7f77e40cbdf0 2023-08-25T10:01:33.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.375+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f77d804e430 con 0x7f77e40cbdf0 2023-08-25T10:01:33.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.375+0000 7f77cbfff640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77d006a050 0x7f77d006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:33.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.376+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f77d80aaa60 con 0x7f77e40cbdf0 2023-08-25T10:01:33.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.376+0000 7f77e2ffd640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77d006a050 0x7f77d006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:33.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.376+0000 7f77e2ffd640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77d006a050 0x7f77d006c510 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f77d4000f60 tx=0x7f77d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:33.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.381+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f77d8075c90 con 0x7f77e40cbdf0 2023-08-25T10:01:33.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.627+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77e40c9780 con 0x7f77e40cbdf0 2023-08-25T10:01:33.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.628+0000 7f77cbfff640 1 -- 172.21.15.71:0/408616375 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+6527 (secure 0 0 0) 0x7f77d80792b0 con 0x7f77e40cbdf0 2023-08-25T10:01:33.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:33.630 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:26.987746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0,3],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:33.630 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 92 2023-08-25T10:01:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77d006a050 msgr2=0x7f77d006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77d006a050 0x7f77d006c510 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f77d4000f60 tx=0x7f77d4009040 comp rx=0 tx=0).stop 2023-08-25T10:01:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 msgr2=0x7f77e4159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 0x7f77e4159720 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f77d8034040 tx=0x7f77d80046c0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 shutdown_connections 2023-08-25T10:01:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f77d006a050 0x7f77d006c510 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77e4156640 0x7f77e414efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77e40cbdf0 0x7f77e4159720 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 --2- 172.21.15.71:0/408616375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77e406b8f0 0x7f77e40c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:33.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 >> 172.21.15.71:0/408616375 conn(0x7f77e406a5b0 msgr2=0x7f77e41520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:33.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.631+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 shutdown_connections 2023-08-25T10:01:33.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:33.632+0000 7f77e99cf640 1 -- 172.21.15.71:0/408616375 wait complete. 2023-08-25T10:01:33.652 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:35.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:35.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.892+0000 7fc5d94ec640 1 Processor -- start 2023-08-25T10:01:35.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.892+0000 7fc5d94ec640 1 -- start start 2023-08-25T10:01:35.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d94ec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc5d41542d0 0x7fc5d41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d94ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d4069850 0x7fc5d40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d94ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5d40c75c0 0x7fc5d40c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d94ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d40c4b60 con 0x7fc5d40c75c0 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d94ec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d4157bf0 con 0x7fc5d41542d0 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d94ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d406a130 con 0x7fc5d4069850 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d8ceb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5d40c75c0 0x7fc5d40c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d8ceb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc5d40c75c0 0x7fc5d40c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41162/0 (socket says 172.21.15.71:41162) 2023-08-25T10:01:35.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d8ceb640 1 -- 172.21.15.71:0/2834976781 learned_addr learned my addr 172.21.15.71:0/2834976781 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/2834976781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d4069850 0x7fc5d40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d8ceb640 1 -- 172.21.15.71:0/2834976781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d4069850 msgr2=0x7fc5d40c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d8ceb640 1 --2- 172.21.15.71:0/2834976781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d4069850 0x7fc5d40c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d8ceb640 1 -- 172.21.15.71:0/2834976781 >> v2:172.21.15.164:3300/0 conn(0x7fc5d41542d0 msgr2=0x7fc5d41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.894+0000 7fc5d3fff640 1 --2- 172.21.15.71:0/2834976781 >> v2:172.21.15.164:3300/0 conn(0x7fc5d41542d0 0x7fc5d41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.895+0000 7fc5d8ceb640 1 --2- 172.21.15.71:0/2834976781 >> v2:172.21.15.164:3300/0 conn(0x7fc5d41542d0 0x7fc5d41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.895+0000 7fc5d8ceb640 1 -- 172.21.15.71:0/2834976781 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d41546b0 con 0x7fc5d40c75c0 2023-08-25T10:01:35.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.895+0000 7fc5d3fff640 1 --2- 172.21.15.71:0/2834976781 >> v2:172.21.15.164:3300/0 conn(0x7fc5d41542d0 0x7fc5d41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:35.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.896+0000 7fc5d8ceb640 1 --2- 172.21.15.71:0/2834976781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40c4fb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4069ee0 tx=0x7fc5cc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9015d4b7d22c0487 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:35.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.896+0000 7fc5d37fe640 1 -- 172.21.15.71:0/2834976781 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc02ca80 con 0x7fc5d40c75c0 2023-08-25T10:01:35.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.896+0000 7fc5d37fe640 1 -- 172.21.15.71:0/2834976781 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc02cbe0 con 0x7fc5d40c75c0 2023-08-25T10:01:35.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.896+0000 7fc5d37fe640 1 -- 172.21.15.71:0/2834976781 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc0343f0 con 0x7fc5d40c75c0 2023-08-25T10:01:35.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2834976781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d40c75c0 msgr2=0x7fc5d40c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:35.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2834976781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40c4fb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4069ee0 tx=0x7fc5cc02bee0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2834976781 shutdown_connections 2023-08-25T10:01:35.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2834976781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40c4fb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2834976781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d4069850 0x7fc5d40c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2834976781 >> v2:172.21.15.164:3300/0 conn(0x7fc5d41542d0 0x7fc5d41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2834976781 >> 172.21.15.71:0/2834976781 conn(0x7fc5d406c280 msgr2=0x7fc5d406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:35.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2834976781 shutdown_connections 2023-08-25T10:01:35.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.897+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2834976781 wait complete. 2023-08-25T10:01:35.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.899+0000 7fc5d94ec640 1 Processor -- start 2023-08-25T10:01:35.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.899+0000 7fc5d94ec640 1 -- start start 2023-08-25T10:01:35.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d94ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 0x7fc5d40bfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d94ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d94ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d41542d0 0x7fc5d4150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d94ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40c3740 con 0x7fc5d4069850 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d94ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4154ec0 con 0x7fc5d40c75c0 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d94ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d40bf5d0 con 0x7fc5d41542d0 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 0x7fc5d40bfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 0x7fc5d40bfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41166/0 (socket says 172.21.15.71:41166) 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5d3fff640 1 -- 172.21.15.71:0/2052305653 learned_addr learned my addr 172.21.15.71:0/2052305653 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.900+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d3fff640 1 -- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d41542d0 msgr2=0x7fc5d4150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d8ceb640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d41542d0 0x7fc5d4150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d3fff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d41542d0 0x7fc5d4150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d3fff640 1 -- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5d40c75c0 msgr2=0x7fc5d40bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d3fff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40bffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d3fff640 1 -- 172.21.15.71:0/2052305653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc009b30 con 0x7fc5d4069850 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d8ceb640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d41542d0 0x7fc5d4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:35.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d3fff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 0x7fc5d40bfa60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c7f0 tx=0x7fc5c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:35.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c40077c0 con 0x7fc5d4069850 2023-08-25T10:01:35.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4007920 con 0x7fc5d4069850 2023-08-25T10:01:35.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4007c90 con 0x7fc5d4069850 2023-08-25T10:01:35.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d41e2420 con 0x7fc5d4069850 2023-08-25T10:01:35.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.901+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d41e29e0 con 0x7fc5d4069850 2023-08-25T10:01:35.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.903+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d404e420 con 0x7fc5d4069850 2023-08-25T10:01:35.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.904+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc5c40113f0 con 0x7fc5d4069850 2023-08-25T10:01:35.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.905+0000 7fc5d17fa640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5b406a050 0x7fc5b406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:35.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.905+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc5c4088d70 con 0x7fc5d4069850 2023-08-25T10:01:35.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.905+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5b406a050 0x7fc5b406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:35.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.906+0000 7fc5cbfff640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5b406a050 0x7fc5b406c510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006fd0 tx=0x7fc5bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:35.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:35.909+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc5c4053f90 con 0x7fc5d4069850 2023-08-25T10:01:36.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.137+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5d4069f10 con 0x7fc5d4069850 2023-08-25T10:01:36.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.138+0000 7fc5d17fa640 1 -- 172.21.15.71:0/2052305653 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fc5c40575b0 con 0x7fc5d4069850 2023-08-25T10:01:36.140 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:36.141 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:36.141 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:36.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5b406a050 msgr2=0x7fc5b406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:36.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5b406a050 0x7fc5b406c510 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006fd0 tx=0x7fc5bc008040 comp rx=0 tx=0).stop 2023-08-25T10:01:36.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 msgr2=0x7fc5d40bfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:36.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 0x7fc5d40bfa60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400c7f0 tx=0x7fc5c400ccb0 comp rx=0 tx=0).stop 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 shutdown_connections 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5d40c75c0 0x7fc5d40bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc5b406a050 0x7fc5b406c510 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5d41542d0 0x7fc5d4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 --2- 172.21.15.71:0/2052305653 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5d4069850 0x7fc5d40bfa60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 >> 172.21.15.71:0/2052305653 conn(0x7fc5d406c280 msgr2=0x7fc5d41dccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:36.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 shutdown_connections 2023-08-25T10:01:36.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:36.141+0000 7fc5d94ec640 1 -- 172.21.15.71:0/2052305653 wait complete. 2023-08-25T10:01:36.161 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:38.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:38.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.380+0000 7f20d827c640 1 Processor -- start 2023-08-25T10:01:38.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.380+0000 7f20d827c640 1 -- start start 2023-08-25T10:01:38.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.381+0000 7f20d827c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f20d00c3a40 0x7f20d00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:38.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.381+0000 7f20d827c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f20d01509d0 0x7f20d0150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:38.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.381+0000 7f20d827c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d00c1dc0 con 0x7f20d00c3660 2023-08-25T10:01:38.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.381+0000 7f20d827c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d0157a80 con 0x7f20d00c3a40 2023-08-25T10:01:38.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.381+0000 7f20d827c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d0151320 con 0x7f20d01509d0 2023-08-25T10:01:38.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d6018640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f20d00c3660 0x7f20d006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43490/0 (socket says 172.21.15.71:43490) 2023-08-25T10:01:38.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d6018640 1 -- 172.21.15.71:0/2109193599 learned_addr learned my addr 172.21.15.71:0/2109193599 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:38.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d5817640 1 --2- 172.21.15.71:0/2109193599 >> v2:172.21.15.71:3300/0 conn(0x7f20d00c3a40 0x7f20d00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:38.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d6819640 1 --2- 172.21.15.71:0/2109193599 >> v2:172.21.15.164:3300/0 conn(0x7f20d01509d0 0x7f20d0150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:38.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d5817640 1 -- 172.21.15.71:0/2109193599 >> v2:172.21.15.164:3300/0 conn(0x7f20d01509d0 msgr2=0x7f20d0150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:38.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d5817640 1 --2- 172.21.15.71:0/2109193599 >> v2:172.21.15.164:3300/0 conn(0x7f20d01509d0 0x7f20d0150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d5817640 1 -- 172.21.15.71:0/2109193599 >> v1:172.21.15.71:6789/0 conn(0x7f20d00c3660 legacy=0x7f20d006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:38.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d5817640 1 -- 172.21.15.71:0/2109193599 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d0156e00 con 0x7f20d00c3a40 2023-08-25T10:01:38.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.382+0000 7f20d6819640 1 --2- 172.21.15.71:0/2109193599 >> v2:172.21.15.164:3300/0 conn(0x7f20d01509d0 0x7f20d0150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:38.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.383+0000 7f20d5817640 1 --2- 172.21.15.71:0/2109193599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d00c1710 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f20d00663f0 tx=0x7f20c002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f4adb925a3accc server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:38.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.383+0000 7f20d5016640 1 -- 172.21.15.71:0/2109193599 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0030610 con 0x7f20d00c3a40 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.383+0000 7f20d5016640 1 -- 172.21.15.71:0/2109193599 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0030770 con 0x7f20d00c3a40 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.383+0000 7f20d5016640 1 -- 172.21.15.71:0/2109193599 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0030b00 con 0x7f20d00c3a40 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 -- 172.21.15.71:0/2109193599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 msgr2=0x7f20d00c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 --2- 172.21.15.71:0/2109193599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d00c1710 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f20d00663f0 tx=0x7f20c002b960 comp rx=0 tx=0).stop 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 -- 172.21.15.71:0/2109193599 shutdown_connections 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 --2- 172.21.15.71:0/2109193599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d00c1710 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 --2- 172.21.15.71:0/2109193599 >> v2:172.21.15.164:3300/0 conn(0x7f20d01509d0 0x7f20d0150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 -- 172.21.15.71:0/2109193599 >> 172.21.15.71:0/2109193599 conn(0x7f20d00654f0 msgr2=0x7f20d0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:38.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 -- 172.21.15.71:0/2109193599 shutdown_connections 2023-08-25T10:01:38.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.384+0000 7f20d827c640 1 -- 172.21.15.71:0/2109193599 wait complete. 2023-08-25T10:01:38.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.386+0000 7f20d827c640 1 Processor -- start 2023-08-25T10:01:38.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.386+0000 7f20d827c640 1 -- start start 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d827c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20d00c3660 0x7f20d01525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d827c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d0069a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d827c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20d01509d0 0x7f20d0069f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d827c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d015aa70 con 0x7f20d00c3a40 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d827c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d0155ea0 con 0x7f20d00c3660 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d827c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d015acc0 con 0x7f20d01509d0 2023-08-25T10:01:38.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d5817640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d0069a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.387+0000 7f20d5817640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d0069a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:41190/0 (socket says 172.21.15.71:41190) 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 -- 172.21.15.71:0/881466402 learned_addr learned my addr 172.21.15.71:0/881466402 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d6018640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20d00c3660 0x7f20d01525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d6819640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20d01509d0 0x7f20d0069f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 -- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20d01509d0 msgr2=0x7f20d0069f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20d01509d0 0x7f20d0069f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 -- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20d00c3660 msgr2=0x7f20d01525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:38.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20d00c3660 0x7f20d01525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 -- 172.21.15.71:0/881466402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c0009400 con 0x7f20d00c3a40 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d6018640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20d00c3660 0x7f20d01525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.388+0000 7f20d5817640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d0069a30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f20c0009890 tx=0x7f20c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.389+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c003e040 con 0x7f20d00c3a40 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.389+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0039cb0 con 0x7f20d00c3a40 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.389+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0044630 con 0x7f20d00c3a40 2023-08-25T10:01:38.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.389+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d015afa0 con 0x7f20d00c3a40 2023-08-25T10:01:38.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.389+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d0159720 con 0x7f20d00c3a40 2023-08-25T10:01:38.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.390+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d00c29c0 con 0x7f20d00c3a40 2023-08-25T10:01:38.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.391+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f20c0044790 con 0x7f20d00c3a40 2023-08-25T10:01:38.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.392+0000 7f20c6ffd640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20b406a050 0x7f20b406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:38.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.392+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f20c00ab060 con 0x7f20d00c3a40 2023-08-25T10:01:38.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.392+0000 7f20d6018640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20b406a050 0x7f20b406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:38.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.393+0000 7f20d6018640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20b406a050 0x7f20b406c510 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f20cc005880 tx=0x7f20cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:38.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.397+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f20c0076210 con 0x7f20d00c3a40 2023-08-25T10:01:38.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.624+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20d01570b0 con 0x7f20d00c3a40 2023-08-25T10:01:38.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.625+0000 7f20c6ffd640 1 -- 172.21.15.71:0/881466402 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f20c0079830 con 0x7f20d00c3a40 2023-08-25T10:01:38.628 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:38.629 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:38.629 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:38.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.628+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20b406a050 msgr2=0x7f20b406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:38.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.628+0000 7f20d827c640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20b406a050 0x7f20b406c510 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f20cc005880 tx=0x7f20cc009040 comp rx=0 tx=0).stop 2023-08-25T10:01:38.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 msgr2=0x7f20d0069a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:38.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d0069a30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f20c0009890 tx=0x7f20c0004730 comp rx=0 tx=0).stop 2023-08-25T10:01:38.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 shutdown_connections 2023-08-25T10:01:38.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f20b406a050 0x7f20b406c510 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20d01509d0 0x7f20d0069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f20d00c3a40 0x7f20d0069a30 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 --2- 172.21.15.71:0/881466402 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20d00c3660 0x7f20d01525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:38.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 >> 172.21.15.71:0/881466402 conn(0x7f20d00654f0 msgr2=0x7f20d00c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:38.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 shutdown_connections 2023-08-25T10:01:38.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:38.629+0000 7f20d827c640 1 -- 172.21.15.71:0/881466402 wait complete. 2023-08-25T10:01:38.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:40.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:40.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.877+0000 7fe1baae0640 1 Processor -- start 2023-08-25T10:01:40.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.877+0000 7fe1baae0640 1 -- start start 2023-08-25T10:01:40.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.879+0000 7fe1baae0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 0x7fe1b40acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.879+0000 7fe1baae0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b40c1060 con 0x7fe1b40ad2f0 2023-08-25T10:01:40.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.879+0000 7fe1baae0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b40af730 con 0x7fe1b40c76d0 2023-08-25T10:01:40.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.879+0000 7fe1baae0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40adf70 con 0x7fe1b40c72f0 2023-08-25T10:01:40.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.879+0000 7fe1ba2df640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe1b40ad2f0 0x7fe1b4155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53436/0 (socket says 172.21.15.71:53436) 2023-08-25T10:01:40.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.879+0000 7fe1ba2df640 1 -- 172.21.15.71:0/3217797501 learned_addr learned my addr 172.21.15.71:0/3217797501 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:40.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b9ade640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 0x7fe1b40acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279387522 0 0) 0x7fe1b40c1060 con 0x7fe1b40ad2f0 2023-08-25T10:01:40.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1a0003520 con 0x7fe1b40ad2f0 2023-08-25T10:01:40.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 221196003 0 0) 0x7fe1a0003520 con 0x7fe1b40ad2f0 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 msgr2=0x7fe1b40acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 0x7fe1b40acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 >> v1:172.21.15.164:6789/0 conn(0x7fe1b40c76d0 legacy=0x7fe1b4152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b406b4a0 con 0x7fe1b40ad2f0 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe1ac0019f0 con 0x7fe1b40ad2f0 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.880+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 >> v1:172.21.15.71:6789/0 conn(0x7fe1b40ad2f0 legacy=0x7fe1b4155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b8adc640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 0x7fe1a0003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b8adc640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1a0003f90 0x7fe1a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b8adc640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40adf70 con 0x7fe1a0004d00 2023-08-25T10:01:40.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a000dd80 con 0x7fe1a0003f90 2023-08-25T10:01:40.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a000df40 con 0x7fe1b40c72f0 2023-08-25T10:01:40.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1ba2df640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b9ade640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 0x7fe1a0003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1ba2df640 1 -- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 msgr2=0x7fe1a0003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:40.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b92dd640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1a0003f90 0x7fe1a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1ba2df640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b40c72f0 0x7fe1a0003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:40.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1ba2df640 1 -- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1a0003f90 msgr2=0x7fe1a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:40.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1ba2df640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1a0003f90 0x7fe1a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:40.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1ba2df640 1 -- 172.21.15.71:0/3217797501 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a0003520 con 0x7fe1a0004d00 2023-08-25T10:01:40.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.881+0000 7fe1b92dd640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1a0003f90 0x7fe1a00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:40.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.882+0000 7fe1ba2df640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1a0005180 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00a800 tx=0x7fe1a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c9d88fcc2cadca server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:40.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0072d0 con 0x7fe1a0004d00 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac02f070 con 0x7fe1a0004d00 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1b8adc640 1 -- 172.21.15.71:0/3217797501 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0076f0 con 0x7fe1a0004d00 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1baae0640 1 -- 172.21.15.71:0/3217797501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 msgr2=0x7fe1a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1baae0640 1 --2- 172.21.15.71:0/3217797501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1a0005180 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00a800 tx=0x7fe1a00057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1ba2df640 1 -- 172.21.15.71:0/3217797501 reap_dead start 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1baae0640 1 -- 172.21.15.71:0/3217797501 shutdown_connections 2023-08-25T10:01:40.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.883+0000 7fe1baae0640 1 -- 172.21.15.71:0/3217797501 >> 172.21.15.71:0/3217797501 conn(0x7fe1b406c280 msgr2=0x7fe1b406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.884+0000 7fe1baae0640 1 -- 172.21.15.71:0/3217797501 shutdown_connections 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.884+0000 7fe1baae0640 1 -- 172.21.15.71:0/3217797501 wait complete. 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.885+0000 7fe1baae0640 1 Processor -- start 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.885+0000 7fe1baae0640 1 -- start start 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.886+0000 7fe1baae0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1b4070830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.886+0000 7fe1baae0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b4071910 0x7fe1b406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.886+0000 7fe1baae0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b4070d70 0x7fe1b406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.886+0000 7fe1baae0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40ac2a0 con 0x7fe1a0004d00 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.886+0000 7fe1baae0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40ab640 con 0x7fe1b4071910 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.886+0000 7fe1baae0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b406e370 con 0x7fe1b4070d70 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1b4070830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1b4070830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:44568/0 (socket says 172.21.15.71:44568) 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 -- 172.21.15.71:0/4225916726 learned_addr learned my addr 172.21.15.71:0/4225916726 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:40.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b92dd640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b4071910 0x7fe1b406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1ba2df640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b4070d70 0x7fe1b406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 -- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b4070d70 msgr2=0x7fe1b406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b4070d70 0x7fe1b406ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 -- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b4071910 msgr2=0x7fe1b406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b4071910 0x7fe1b406e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 -- 172.21.15.71:0/4225916726 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b40712f0 con 0x7fe1a0004d00 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1ba2df640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b4070d70 0x7fe1b406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:40.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.887+0000 7fe1b9ade640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1b4070830 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0011630 tx=0x7fe1b0011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:40.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.888+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b000f8c0 con 0x7fe1a0004d00 2023-08-25T10:01:40.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.888+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b000fa20 con 0x7fe1a0004d00 2023-08-25T10:01:40.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.888+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b000fd90 con 0x7fe1a0004d00 2023-08-25T10:01:40.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.888+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b40abfc0 con 0x7fe1a0004d00 2023-08-25T10:01:40.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.888+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b40aa530 con 0x7fe1a0004d00 2023-08-25T10:01:40.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.889+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b40aa0b0 con 0x7fe1a0004d00 2023-08-25T10:01:40.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.890+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe1b0015460 con 0x7fe1a0004d00 2023-08-25T10:01:40.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.891+0000 7fe1aa7fc640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe19806a080 0x7fe19806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:40.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.891+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe1b00893b0 con 0x7fe1a0004d00 2023-08-25T10:01:40.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.891+0000 7fe1b92dd640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe19806a080 0x7fe19806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:40.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.892+0000 7fe1b92dd640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe19806a080 0x7fe19806c540 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4156700 tx=0x7fe1a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:40.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:40.895+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe1b40aa0b0 con 0x7fe1a0004d00 2023-08-25T10:01:41.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.122+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1b40aa0b0 con 0x7fe1a0004d00 2023-08-25T10:01:41.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.123+0000 7fe1aa7fc640 1 -- 172.21.15.71:0/4225916726 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fe1b40aa0b0 con 0x7fe1a0004d00 2023-08-25T10:01:41.126 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:41.127 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:41.127 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:41.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.126+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe19806a080 msgr2=0x7fe19806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:41.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.126+0000 7fe1baae0640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe19806a080 0x7fe19806c540 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4156700 tx=0x7fe1a4009040 comp rx=0 tx=0).stop 2023-08-25T10:01:41.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.126+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 msgr2=0x7fe1b4070830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:41.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.126+0000 7fe1baae0640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1b4070830 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0011630 tx=0x7fe1b0011af0 comp rx=0 tx=0).stop 2023-08-25T10:01:41.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 shutdown_connections 2023-08-25T10:01:41.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe1b4071910 0x7fe1b406e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:41.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe19806a080 0x7fe19806c540 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:41.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe1b4070d70 0x7fe1b406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:41.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 --2- 172.21.15.71:0/4225916726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe1a0004d00 0x7fe1b4070830 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:41.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 >> 172.21.15.71:0/4225916726 conn(0x7fe1b406c280 msgr2=0x7fe1b4150e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:41.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 shutdown_connections 2023-08-25T10:01:41.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:41.127+0000 7fe1baae0640 1 -- 172.21.15.71:0/4225916726 wait complete. 2023-08-25T10:01:41.147 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:01:41.148 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:01:34.828305+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:01:43.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:43.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.374+0000 7f62bbc3f640 1 Processor -- start 2023-08-25T10:01:43.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.374+0000 7f62bbc3f640 1 -- start start 2023-08-25T10:01:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.376+0000 7f62bbc3f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f62b406eed0 0x7f62b415a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.376+0000 7f62bbc3f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b406a500 con 0x7f62b4065880 2023-08-25T10:01:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.376+0000 7f62bbc3f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b415a6d0 con 0x7f62b4157820 2023-08-25T10:01:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.376+0000 7f62bbc3f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b4071e60 con 0x7f62b406eed0 2023-08-25T10:01:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.376+0000 7f62b99db640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f62b406eed0 0x7f62b415a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.377+0000 7f62b91da640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f62b4065880 0x7f62b406f2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53452/0 (socket says 172.21.15.71:53452) 2023-08-25T10:01:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.377+0000 7f62b99db640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f62b406eed0 0x7f62b415a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:44576/0 (socket says 172.21.15.71:44576) 2023-08-25T10:01:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.377+0000 7f62b91da640 1 -- 172.21.15.71:0/2908127450 learned_addr learned my addr 172.21.15.71:0/2908127450 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:43.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.377+0000 7f62b99db640 1 -- 172.21.15.71:0/2908127450 >> v1:172.21.15.164:6789/0 conn(0x7f62b4157820 legacy=0x7f62b4161520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.377+0000 7f62b99db640 1 -- 172.21.15.71:0/2908127450 >> v1:172.21.15.71:6789/0 conn(0x7f62b4065880 legacy=0x7f62b406f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.377+0000 7f62b99db640 1 -- 172.21.15.71:0/2908127450 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b406a260 con 0x7f62b406eed0 2023-08-25T10:01:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.378+0000 7f62b99db640 1 --2- 172.21.15.71:0/2908127450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b415a190 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f62b00097a0 tx=0x7f62b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bdd206575965e0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62b89d9640 1 -- 172.21.15.71:0/2908127450 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0031830 con 0x7f62b406eed0 2023-08-25T10:01:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62b89d9640 1 -- 172.21.15.71:0/2908127450 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0031990 con 0x7f62b406eed0 2023-08-25T10:01:43.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62b89d9640 1 -- 172.21.15.71:0/2908127450 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0031d20 con 0x7f62b406eed0 2023-08-25T10:01:43.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62bbc3f640 1 -- 172.21.15.71:0/2908127450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 msgr2=0x7f62b415a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:43.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/2908127450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b415a190 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f62b00097a0 tx=0x7f62b002b8d0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62bbc3f640 1 -- 172.21.15.71:0/2908127450 shutdown_connections 2023-08-25T10:01:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/2908127450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b415a190 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.379+0000 7f62bbc3f640 1 -- 172.21.15.71:0/2908127450 >> 172.21.15.71:0/2908127450 conn(0x7f62b406d350 msgr2=0x7f62b40c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.380+0000 7f62bbc3f640 1 -- 172.21.15.71:0/2908127450 shutdown_connections 2023-08-25T10:01:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.380+0000 7f62bbc3f640 1 -- 172.21.15.71:0/2908127450 wait complete. 2023-08-25T10:01:43.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.381+0000 7f62bbc3f640 1 Processor -- start 2023-08-25T10:01:43.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.381+0000 7f62bbc3f640 1 -- start start 2023-08-25T10:01:43.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.382+0000 7f62bbc3f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62b4065880 0x7f62b4156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:43.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.382+0000 7f62bbc3f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b4153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.382+0000 7f62bbc3f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62b4157820 0x7f62b41598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.382+0000 7f62bbc3f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b4156b60 con 0x7f62b406eed0 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.382+0000 7f62bbc3f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b41ed010 con 0x7f62b4065880 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.382+0000 7f62bbc3f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b41ed2f0 con 0x7f62b4157820 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b4153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b4153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:44592/0 (socket says 172.21.15.71:44592) 2023-08-25T10:01:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 -- 172.21.15.71:0/1234060973 learned_addr learned my addr 172.21.15.71:0/1234060973 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b99db640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62b4065880 0x7f62b4156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 -- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62b4157820 msgr2=0x7f62b41598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62ba1dc640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62b4157820 0x7f62b41598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62b4157820 0x7f62b41598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 -- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62b4065880 msgr2=0x7f62b4156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62b4065880 0x7f62b4156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.383+0000 7f62b91da640 1 -- 172.21.15.71:0/1234060973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0009400 con 0x7f62b406eed0 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.384+0000 7f62b91da640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b4153590 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c480 tx=0x7f62a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.384+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a4005ce0 con 0x7f62b406eed0 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.384+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a4005e40 con 0x7f62b406eed0 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.384+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40072f0 con 0x7f62b406eed0 2023-08-25T10:01:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.384+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b41ed5d0 con 0x7f62b406eed0 2023-08-25T10:01:43.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.384+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b41edb90 con 0x7f62b406eed0 2023-08-25T10:01:43.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.385+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b414cba0 con 0x7f62b406eed0 2023-08-25T10:01:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.386+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f62a4007510 con 0x7f62b406eed0 2023-08-25T10:01:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.387+0000 7f62aa7fc640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f629806a050 0x7f629806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.387+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f62a4087ea0 con 0x7f62b406eed0 2023-08-25T10:01:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.387+0000 7f62b99db640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f629806a050 0x7f629806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.388+0000 7f62b99db640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f629806a050 0x7f629806c510 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f62b0036000 tx=0x7f62b003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:43.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.392+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f62a4053210 con 0x7f62b406eed0 2023-08-25T10:01:43.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.620+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62b40c6ae0 con 0x7f62b406eed0 2023-08-25T10:01:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.620+0000 7f62aa7fc640 1 -- 172.21.15.71:0/1234060973 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f62a4017090 con 0x7f62b406eed0 2023-08-25T10:01:43.623 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:43.624 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:43.624 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:43.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f629806a050 msgr2=0x7f629806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f629806a050 0x7f629806c510 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f62b0036000 tx=0x7f62b003c040 comp rx=0 tx=0).stop 2023-08-25T10:01:43.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 msgr2=0x7f62b4153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b4153590 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c480 tx=0x7f62a400c940 comp rx=0 tx=0).stop 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 shutdown_connections 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f629806a050 0x7f629806c510 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62b4157820 0x7f62b41598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f62b406eed0 0x7f62b4153590 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 --2- 172.21.15.71:0/1234060973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62b4065880 0x7f62b4156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:43.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 >> 172.21.15.71:0/1234060973 conn(0x7f62b406d350 msgr2=0x7f62b414c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:43.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.623+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 shutdown_connections 2023-08-25T10:01:43.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:43.624+0000 7f62bbc3f640 1 -- 172.21.15.71:0/1234060973 wait complete. 2023-08-25T10:01:43.644 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:45.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:45.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.873+0000 7fa56246a640 1 Processor -- start 2023-08-25T10:01:45.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.873+0000 7fa56246a640 1 -- start start 2023-08-25T10:01:45.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.875+0000 7fa56246a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa55c159fc0 0x7fa55c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.875+0000 7fa56246a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55c06a960 con 0x7fa55c06cc70 2023-08-25T10:01:45.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.875+0000 7fa56246a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55c06ac20 con 0x7fa55c1594d0 2023-08-25T10:01:45.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.875+0000 7fa56246a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa55c06af00 con 0x7fa55c159fc0 2023-08-25T10:01:45.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.875+0000 7fa55b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa55c06cc70 0x7fa55c1598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53468/0 (socket says 172.21.15.71:53468) 2023-08-25T10:01:45.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.875+0000 7fa55b7fe640 1 -- 172.21.15.71:0/4083823928 learned_addr learned my addr 172.21.15.71:0/4083823928 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:45.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577133065 0 0) 0x7fa55c06a960 con 0x7fa55c06cc70 2023-08-25T10:01:45.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa560a07640 1 --2- 172.21.15.71:0/4083823928 >> v2:172.21.15.164:3300/0 conn(0x7fa55c159fc0 0x7fa55c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:45.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa544003520 con 0x7fa55c06cc70 2023-08-25T10:01:45.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895428356 0 0) 0x7fa544003520 con 0x7fa55c06cc70 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 >> v2:172.21.15.164:3300/0 conn(0x7fa55c159fc0 msgr2=0x7fa55c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 --2- 172.21.15.71:0/4083823928 >> v2:172.21.15.164:3300/0 conn(0x7fa55c159fc0 0x7fa55c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 >> v1:172.21.15.164:6789/0 conn(0x7fa55c1594d0 legacy=0x7fa55c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c0c2e60 con 0x7fa55c06cc70 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fa550001670 con 0x7fa55c06cc70 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 >> v1:172.21.15.71:6789/0 conn(0x7fa55c06cc70 legacy=0x7fa55c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5440039e0 0x7fa544003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5440047c0 0x7fa544004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa544005180 0x7fa54400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c06af00 con 0x7fa5440039e0 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54400c090 con 0x7fa5440047c0 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.876+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54400c250 con 0x7fa544005180 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa55bfff640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5440039e0 0x7fa544003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa55bfff640 1 -- 172.21.15.71:0/4083823928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa544005180 msgr2=0x7fa54400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:45.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa55bfff640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa544005180 0x7fa54400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa55bfff640 1 -- 172.21.15.71:0/4083823928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5440047c0 msgr2=0x7fa544004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:45.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa55bfff640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa5440047c0 0x7fa544004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa55bfff640 1 -- 172.21.15.71:0/4083823928 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa544003520 con 0x7fa5440039e0 2023-08-25T10:01:45.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.877+0000 7fa560a07640 1 -- 172.21.15.71:0/4083823928 reap_dead start 2023-08-25T10:01:45.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.878+0000 7fa55bfff640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5440039e0 0x7fa544003e00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa54c001750 tx=0x7fa5440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53a09353580231d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:45.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.878+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0052d0 con 0x7fa5440039e0 2023-08-25T10:01:45.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.878+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54c02e050 con 0x7fa5440039e0 2023-08-25T10:01:45.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.878+0000 7fa55affd640 1 -- 172.21.15.71:0/4083823928 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c003830 con 0x7fa5440039e0 2023-08-25T10:01:45.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 -- 172.21.15.71:0/4083823928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5440039e0 msgr2=0x7fa544003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:45.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5440039e0 0x7fa544003e00 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa54c001750 tx=0x7fa5440057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 -- 172.21.15.71:0/4083823928 shutdown_connections 2023-08-25T10:01:45.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 --2- 172.21.15.71:0/4083823928 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5440039e0 0x7fa544003e00 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 -- 172.21.15.71:0/4083823928 >> 172.21.15.71:0/4083823928 conn(0x7fa55c0654f0 msgr2=0x7fa55c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:45.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 -- 172.21.15.71:0/4083823928 shutdown_connections 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.879+0000 7fa56246a640 1 -- 172.21.15.71:0/4083823928 wait complete. 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.881+0000 7fa56246a640 1 Processor -- start 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.881+0000 7fa56246a640 1 -- start start 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa56246a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5440039e0 0x7fa55c143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa56246a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 0x7fa55c144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa56246a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa55c1ef790 0x7fa55c1efc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa56246a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c06e930 con 0x7fa55c1e0e20 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa56246a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c0c59b0 con 0x7fa55c1ef790 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa56246a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c06e5c0 con 0x7fa5440039e0 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa55b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 0x7fa55c144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa55b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 0x7fa55c144180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:44612/0 (socket says 172.21.15.71:44612) 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa55b7fe640 1 -- 172.21.15.71:0/2256968615 learned_addr learned my addr 172.21.15.71:0/2256968615 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.882+0000 7fa55bfff640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5440039e0 0x7fa55c143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:45.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa560a07640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa55c1ef790 0x7fa55c1efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa55b7fe640 1 -- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5440039e0 msgr2=0x7fa55c143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa55b7fe640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5440039e0 0x7fa55c143c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa55b7fe640 1 -- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa55c1ef790 msgr2=0x7fa55c1efc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa55b7fe640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa55c1ef790 0x7fa55c1efc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa55b7fe640 1 -- 172.21.15.71:0/2256968615 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c149ae0 con 0x7fa55c1e0e20 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa55b7fe640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 0x7fa55c144180 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c730 tx=0x7fa55000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5500072d0 con 0x7fa55c1e0e20 2023-08-25T10:01:45.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa550007430 con 0x7fa55c1e0e20 2023-08-25T10:01:45.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550004a20 con 0x7fa55c1e0e20 2023-08-25T10:01:45.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c149800 con 0x7fa55c1e0e20 2023-08-25T10:01:45.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.883+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55c156e30 con 0x7fa55c1e0e20 2023-08-25T10:01:45.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.885+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55c06e700 con 0x7fa55c1e0e20 2023-08-25T10:01:45.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.886+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa55000a5e0 con 0x7fa55c1e0e20 2023-08-25T10:01:45.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.887+0000 7fa558ff9640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa54806a050 0x7fa54806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:45.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.887+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fa550088100 con 0x7fa55c1e0e20 2023-08-25T10:01:45.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.887+0000 7fa55bfff640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa54806a050 0x7fa54806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:45.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.888+0000 7fa55bfff640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa54806a050 0x7fa54806c510 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa54c033000 tx=0x7fa54c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:45.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:45.891+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa55c06e700 con 0x7fa55c1e0e20 2023-08-25T10:01:46.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.122+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa55c06e700 con 0x7fa55c1e0e20 2023-08-25T10:01:46.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.124+0000 7fa558ff9640 1 -- 172.21.15.71:0/2256968615 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fa55c06e700 con 0x7fa55c1e0e20 2023-08-25T10:01:46.127 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:46.127 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:46.128 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:46.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.126+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa54806a050 msgr2=0x7fa54806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.126+0000 7fa56246a640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa54806a050 0x7fa54806c510 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa54c033000 tx=0x7fa54c009030 comp rx=0 tx=0).stop 2023-08-25T10:01:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.126+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 msgr2=0x7fa55c144180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:46.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.126+0000 7fa56246a640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 0x7fa55c144180 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c730 tx=0x7fa55000cbf0 comp rx=0 tx=0).stop 2023-08-25T10:01:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 shutdown_connections 2023-08-25T10:01:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa55c1ef790 0x7fa55c1efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa54806a050 0x7fa54806c510 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa55c1e0e20 0x7fa55c144180 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 --2- 172.21.15.71:0/2256968615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa5440039e0 0x7fa55c143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:46.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 >> 172.21.15.71:0/2256968615 conn(0x7fa55c0654f0 msgr2=0x7fa55c145f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:46.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 shutdown_connections 2023-08-25T10:01:46.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:46.127+0000 7fa56246a640 1 -- 172.21.15.71:0/2256968615 wait complete. 2023-08-25T10:01:46.148 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:48.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:48.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.372+0000 7f386a1f5640 1 Processor -- start 2023-08-25T10:01:48.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.372+0000 7f386a1f5640 1 -- start start 2023-08-25T10:01:48.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.373+0000 7f386a1f5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f38640c76d0 0x7f3864152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:48.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.373+0000 7f386a1f5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386406aed0 0x7f38641532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.373+0000 7f386a1f5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38640c1060 con 0x7f38640c72f0 2023-08-25T10:01:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.373+0000 7f386a1f5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38640adf70 con 0x7f38640c76d0 2023-08-25T10:01:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.373+0000 7f386a1f5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640af730 con 0x7f386406aed0 2023-08-25T10:01:48.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38691f3640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f38640c72f0 0x7f38640acdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:53478/0 (socket says 172.21.15.71:53478) 2023-08-25T10:01:48.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38691f3640 1 -- 172.21.15.71:0/1489117069 learned_addr learned my addr 172.21.15.71:0/1489117069 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:48.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38689f2640 1 --2- 172.21.15.71:0/1489117069 >> v2:172.21.15.71:3300/0 conn(0x7f38640c76d0 0x7f3864152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:48.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38689f2640 1 -- 172.21.15.71:0/1489117069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386406aed0 msgr2=0x7f38641532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:48.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38689f2640 1 --2- 172.21.15.71:0/1489117069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386406aed0 0x7f38641532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38689f2640 1 -- 172.21.15.71:0/1489117069 >> v1:172.21.15.71:6789/0 conn(0x7f38640c72f0 legacy=0x7f38640acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:48.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.374+0000 7f38689f2640 1 -- 172.21.15.71:0/1489117069 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386406b4a0 con 0x7f38640c76d0 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.375+0000 7f38689f2640 1 --2- 172.21.15.71:0/1489117069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38640c76d0 0x7f3864152da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f38540097a0 tx=0x7f385402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f97bf4ce8e9a3a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f385bfff640 1 -- 172.21.15.71:0/1489117069 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854030800 con 0x7f38640c76d0 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f385bfff640 1 -- 172.21.15.71:0/1489117069 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3854030960 con 0x7f38640c76d0 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f385bfff640 1 -- 172.21.15.71:0/1489117069 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854030cf0 con 0x7f38640c76d0 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f386a1f5640 1 -- 172.21.15.71:0/1489117069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38640c76d0 msgr2=0x7f3864152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f386a1f5640 1 --2- 172.21.15.71:0/1489117069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38640c76d0 0x7f3864152da0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f38540097a0 tx=0x7f385402b940 comp rx=0 tx=0).stop 2023-08-25T10:01:48.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f386a1f5640 1 -- 172.21.15.71:0/1489117069 shutdown_connections 2023-08-25T10:01:48.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f386a1f5640 1 --2- 172.21.15.71:0/1489117069 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386406aed0 0x7f38641532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f386a1f5640 1 --2- 172.21.15.71:0/1489117069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38640c76d0 0x7f3864152da0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.376+0000 7f386a1f5640 1 -- 172.21.15.71:0/1489117069 >> 172.21.15.71:0/1489117069 conn(0x7f386406c280 msgr2=0x7f386406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:48.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.377+0000 7f386a1f5640 1 -- 172.21.15.71:0/1489117069 shutdown_connections 2023-08-25T10:01:48.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.377+0000 7f386a1f5640 1 -- 172.21.15.71:0/1489117069 wait complete. 2023-08-25T10:01:48.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.379+0000 7f386a1f5640 1 Processor -- start 2023-08-25T10:01:48.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.379+0000 7f386a1f5640 1 -- start start 2023-08-25T10:01:48.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f386a1f5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 0x7f38640a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f386a1f5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38640c72f0 0x7f38640bf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f386a1f5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38640c76d0 0x7f38641eddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f386a1f5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386406f020 con 0x7f386406aed0 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f386a1f5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640aa4c0 con 0x7f38640c76d0 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f386a1f5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640aa710 con 0x7f38640c72f0 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f38691f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 0x7f38640a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.380+0000 7f38691f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 0x7f38640a9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:44628/0 (socket says 172.21.15.71:44628) 2023-08-25T10:01:48.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 -- 172.21.15.71:0/2521653882 learned_addr learned my addr 172.21.15.71:0/2521653882 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38699f4640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38640c76d0 0x7f38641eddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38689f2640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38640c72f0 0x7f38640bf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 -- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38640c72f0 msgr2=0x7f38640bf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38640c72f0 0x7f38640bf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 -- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38640c76d0 msgr2=0x7f38641eddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38640c76d0 0x7f38641eddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 -- 172.21.15.71:0/2521653882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854009400 con 0x7f386406aed0 2023-08-25T10:01:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.381+0000 7f38691f3640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 0x7f38640a9f80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f386000c4a0 tx=0x7f386000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.382+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860005ce0 con 0x7f386406aed0 2023-08-25T10:01:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.382+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3860005e40 con 0x7f386406aed0 2023-08-25T10:01:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.382+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860007270 con 0x7f386406aed0 2023-08-25T10:01:48.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.382+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38641565a0 con 0x7f386406aed0 2023-08-25T10:01:48.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.382+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f386406e6f0 con 0x7f386406aed0 2023-08-25T10:01:48.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.383+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386404e420 con 0x7f386406aed0 2023-08-25T10:01:48.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.384+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f38600073d0 con 0x7f386406aed0 2023-08-25T10:01:48.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.385+0000 7f3859ffb640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f384806a080 0x7f384806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:48.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.385+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f3860087f00 con 0x7f386406aed0 2023-08-25T10:01:48.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.385+0000 7f38689f2640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f384806a080 0x7f384806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:48.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.386+0000 7f38689f2640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f384806a080 0x7f384806c540 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3854036000 tx=0x7f385403b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:48.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.389+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f38600531c0 con 0x7f386406aed0 2023-08-25T10:01:48.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.616+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38640734c0 con 0x7f386406aed0 2023-08-25T10:01:48.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.617+0000 7f3859ffb640 1 -- 172.21.15.71:0/2521653882 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f3860017090 con 0x7f386406aed0 2023-08-25T10:01:48.621 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:48.621 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:48.621 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:48.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f384806a080 msgr2=0x7f384806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:48.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f384806a080 0x7f384806c540 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3854036000 tx=0x7f385403b040 comp rx=0 tx=0).stop 2023-08-25T10:01:48.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 msgr2=0x7f38640a9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:48.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 0x7f38640a9f80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f386000c4a0 tx=0x7f386000c960 comp rx=0 tx=0).stop 2023-08-25T10:01:48.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 shutdown_connections 2023-08-25T10:01:48.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38640c76d0 0x7f38641eddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f384806a080 0x7f384806c540 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38640c72f0 0x7f38640bf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 --2- 172.21.15.71:0/2521653882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f386406aed0 0x7f38640a9f80 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:48.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 >> 172.21.15.71:0/2521653882 conn(0x7f386406c280 msgr2=0x7f3864072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:48.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 shutdown_connections 2023-08-25T10:01:48.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:48.620+0000 7f386a1f5640 1 -- 172.21.15.71:0/2521653882 wait complete. 2023-08-25T10:01:48.642 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:50.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:50.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.874+0000 7f89f64e0640 1 Processor -- start 2023-08-25T10:01:50.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.874+0000 7f89f64e0640 1 -- start start 2023-08-25T10:01:50.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.875+0000 7f89f64e0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f89f0069850 0x7f89f01576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:50.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f64e0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f89f00c4fb0 0x7f89f00c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:50.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f64e0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f0157bf0 con 0x7f89f01542d0 2023-08-25T10:01:50.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f64e0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f00c4b60 con 0x7f89f00c4fb0 2023-08-25T10:01:50.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f64e0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f006a130 con 0x7f89f0069850 2023-08-25T10:01:50.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f5cdf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f89f00c4fb0 0x7f89f00c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f5cdf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f89f00c4fb0 0x7f89f00c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36654/0 (socket says 172.21.15.71:36654) 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f5cdf640 1 -- 172.21.15.71:0/514838102 learned_addr learned my addr 172.21.15.71:0/514838102 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f4cdd640 1 --2- 172.21.15.71:0/514838102 >> v2:172.21.15.164:3300/0 conn(0x7f89f0069850 0x7f89f01576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f5cdf640 1 -- 172.21.15.71:0/514838102 >> v2:172.21.15.164:3300/0 conn(0x7f89f0069850 msgr2=0x7f89f01576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f5cdf640 1 --2- 172.21.15.71:0/514838102 >> v2:172.21.15.164:3300/0 conn(0x7f89f0069850 0x7f89f01576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.876+0000 7f89f5cdf640 1 -- 172.21.15.71:0/514838102 >> v1:172.21.15.164:6789/0 conn(0x7f89f01542d0 legacy=0x7f89f00c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.877+0000 7f89f5cdf640 1 -- 172.21.15.71:0/514838102 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f00c5980 con 0x7f89f00c4fb0 2023-08-25T10:01:50.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.878+0000 7f89f5cdf640 1 --2- 172.21.15.71:0/514838102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f00c4fb0 0x7f89f00c5440 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f89f00c13c0 tx=0x7f89e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f56cdf19a7ad02a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.878+0000 7f89e7fff640 1 -- 172.21.15.71:0/514838102 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e802ca80 con 0x7f89f00c4fb0 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.878+0000 7f89e7fff640 1 -- 172.21.15.71:0/514838102 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e802cbe0 con 0x7f89f00c4fb0 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.878+0000 7f89e7fff640 1 -- 172.21.15.71:0/514838102 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e80343f0 con 0x7f89f00c4fb0 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 -- 172.21.15.71:0/514838102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f00c4fb0 msgr2=0x7f89f00c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 --2- 172.21.15.71:0/514838102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f00c4fb0 0x7f89f00c5440 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f89f00c13c0 tx=0x7f89e802bee0 comp rx=0 tx=0).stop 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 -- 172.21.15.71:0/514838102 shutdown_connections 2023-08-25T10:01:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 --2- 172.21.15.71:0/514838102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f00c4fb0 0x7f89f00c5440 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 --2- 172.21.15.71:0/514838102 >> v2:172.21.15.164:3300/0 conn(0x7f89f0069850 0x7f89f01576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 -- 172.21.15.71:0/514838102 >> 172.21.15.71:0/514838102 conn(0x7f89f006c280 msgr2=0x7f89f006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 -- 172.21.15.71:0/514838102 shutdown_connections 2023-08-25T10:01:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.879+0000 7f89f64e0640 1 -- 172.21.15.71:0/514838102 wait complete. 2023-08-25T10:01:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.881+0000 7f89f64e0640 1 Processor -- start 2023-08-25T10:01:50.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.881+0000 7f89f64e0640 1 -- start start 2023-08-25T10:01:50.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.882+0000 7f89f64e0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 0x7f89f015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:50.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.882+0000 7f89f64e0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89f00c4fb0 0x7f89f014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:50.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.882+0000 7f89f64e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89f01542d0 0x7f89f00c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.882+0000 7f89f64e0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f01585c0 con 0x7f89f0069850 2023-08-25T10:01:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.882+0000 7f89f64e0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f01f1be0 con 0x7f89f00c4fb0 2023-08-25T10:01:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.882+0000 7f89f64e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f01edce0 con 0x7f89f01542d0 2023-08-25T10:01:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 0x7f89f015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 0x7f89f015ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36656/0 (socket says 172.21.15.71:36656) 2023-08-25T10:01:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 -- 172.21.15.71:0/251513702 learned_addr learned my addr 172.21.15.71:0/251513702 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:50.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f4cdd640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89f00c4fb0 0x7f89f014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:50.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f5cdf640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89f01542d0 0x7f89f00c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:50.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 -- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89f01542d0 msgr2=0x7f89f00c2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:50.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89f01542d0 0x7f89f00c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:50.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 -- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89f00c4fb0 msgr2=0x7f89f014ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:50.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89f00c4fb0 0x7f89f014ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.883+0000 7f89f54de640 1 -- 172.21.15.71:0/251513702 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e8009b30 con 0x7f89f0069850 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.884+0000 7f89f54de640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 0x7f89f015ab30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c460 tx=0x7f89ec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.884+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec012830 con 0x7f89f0069850 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.884+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec0129d0 con 0x7f89f0069850 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.884+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec012d80 con 0x7f89f0069850 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.884+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f015b070 con 0x7f89f0069850 2023-08-25T10:01:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.884+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f01edf60 con 0x7f89f0069850 2023-08-25T10:01:50.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.885+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f004e420 con 0x7f89f0069850 2023-08-25T10:01:50.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.886+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f89ec01a3b0 con 0x7f89f0069850 2023-08-25T10:01:50.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.887+0000 7f89e5ffb640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89d406a050 0x7f89d406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:50.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.887+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f89ec0889b0 con 0x7f89f0069850 2023-08-25T10:01:50.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.888+0000 7f89f4cdd640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89d406a050 0x7f89d406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:50.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.888+0000 7f89f4cdd640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89d406a050 0x7f89d406c510 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f89e0006fd0 tx=0x7f89e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:50.893+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f89ec053be0 con 0x7f89f0069850 2023-08-25T10:01:51.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.123+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f0069f10 con 0x7f89f0069850 2023-08-25T10:01:51.127 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.124+0000 7f89e5ffb640 1 -- 172.21.15.71:0/251513702 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f89ec057200 con 0x7f89f0069850 2023-08-25T10:01:51.127 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:51.127 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:51.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89d406a050 msgr2=0x7f89d406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:51.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89d406a050 0x7f89d406c510 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f89e0006fd0 tx=0x7f89e0008040 comp rx=0 tx=0).stop 2023-08-25T10:01:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 msgr2=0x7f89f015ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 0x7f89f015ab30 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c460 tx=0x7f89ec00c920 comp rx=0 tx=0).stop 2023-08-25T10:01:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 shutdown_connections 2023-08-25T10:01:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f89f00c4fb0 0x7f89f014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f89d406a050 0x7f89d406c510 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89f01542d0 0x7f89f00c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:51.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.126+0000 7f89f64e0640 1 --2- 172.21.15.71:0/251513702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f89f0069850 0x7f89f015ab30 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:51.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.127+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 >> 172.21.15.71:0/251513702 conn(0x7f89f006c280 msgr2=0x7f89f00ad8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:51.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.127+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 shutdown_connections 2023-08-25T10:01:51.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:51.127+0000 7f89f64e0640 1 -- 172.21.15.71:0/251513702 wait complete. 2023-08-25T10:01:51.148 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:53.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:53.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.387+0000 7fdfa2f9b640 1 Processor -- start 2023-08-25T10:01:53.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.387+0000 7fdfa2f9b640 1 -- start start 2023-08-25T10:01:53.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.389+0000 7fdfa2f9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdf9c071be0 0x7fdf9c164520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.389+0000 7fdfa2f9b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0c5260 con 0x7fdf9c0c60e0 2023-08-25T10:01:53.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.389+0000 7fdfa2f9b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf9c0c6580 con 0x7fdf9c152db0 2023-08-25T10:01:53.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.389+0000 7fdfa2f9b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf9c158b90 con 0x7fdf9c071be0 2023-08-25T10:01:53.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa1798640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fdf9c0c60e0 0x7fdf9c158fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42246/0 (socket says 172.21.15.71:42246) 2023-08-25T10:01:53.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa1798640 1 -- 172.21.15.71:0/4045504136 learned_addr learned my addr 172.21.15.71:0/4045504136 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:53.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa279a640 1 --2- 172.21.15.71:0/4045504136 >> v2:172.21.15.164:3300/0 conn(0x7fdf9c071be0 0x7fdf9c164520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198157452 0 0) 0x7fdf9c0c5260 con 0x7fdf9c0c60e0 2023-08-25T10:01:53.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf88003520 con 0x7fdf9c0c60e0 2023-08-25T10:01:53.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 769106388 0 0) 0x7fdf88003520 con 0x7fdf9c0c60e0 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 >> v2:172.21.15.164:3300/0 conn(0x7fdf9c071be0 msgr2=0x7fdf9c164520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 --2- 172.21.15.71:0/4045504136 >> v2:172.21.15.164:3300/0 conn(0x7fdf9c071be0 0x7fdf9c164520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 >> v1:172.21.15.164:6789/0 conn(0x7fdf9c152db0 legacy=0x7fdf9c0c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c15cc30 con 0x7fdf9c0c60e0 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fdf8c001670 con 0x7fdf9c0c60e0 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 >> v1:172.21.15.71:6789/0 conn(0x7fdf9c0c60e0 legacy=0x7fdf9c158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf880039e0 0x7fdf88003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf88005180 0x7fdf8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c158b90 con 0x7fdf88005180 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8800c0b0 con 0x7fdf880039e0 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8800c270 con 0x7fdf880047c0 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.391+0000 7fdfa279a640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf88005180 0x7fdf8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa1798640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa1f99640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf880039e0 0x7fdf88003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa279a640 1 -- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf880047c0 msgr2=0x7fdf88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa279a640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa279a640 1 -- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf880039e0 msgr2=0x7fdf88003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa279a640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf880039e0 0x7fdf88003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa279a640 1 -- 172.21.15.71:0/4045504136 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88003520 con 0x7fdf88005180 2023-08-25T10:01:53.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.392+0000 7fdfa1f99640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf880039e0 0x7fdf88003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:53.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.393+0000 7fdfa279a640 1 -- 172.21.15.71:0/4045504136 reap_dead start 2023-08-25T10:01:53.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.393+0000 7fdfa279a640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf88005180 0x7fdf8800b980 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009ac0 tx=0x7fdf880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226477ab5a49cc4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:53.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.393+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94004020 con 0x7fdf88005180 2023-08-25T10:01:53.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.393+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf94004180 con 0x7fdf88005180 2023-08-25T10:01:53.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.393+0000 7fdfa0f97640 1 -- 172.21.15.71:0/4045504136 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9402c710 con 0x7fdf88005180 2023-08-25T10:01:53.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/4045504136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf88005180 msgr2=0x7fdf8800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf88005180 0x7fdf8800b980 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009ac0 tx=0x7fdf880057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/4045504136 shutdown_connections 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/4045504136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf88005180 0x7fdf8800b980 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/4045504136 >> 172.21.15.71:0/4045504136 conn(0x7fdf9c06d0d0 msgr2=0x7fdf9c0ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/4045504136 shutdown_connections 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.394+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/4045504136 wait complete. 2023-08-25T10:01:53.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.396+0000 7fdfa2f9b640 1 Processor -- start 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.396+0000 7fdfa2f9b640 1 -- start start 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.397+0000 7fdfa2f9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf88005180 0x7fdf9c14c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.397+0000 7fdfa2f9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 0x7fdf9c1ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa2f9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf9c0706a0 0x7fdf9c06fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa2f9b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c15c480 con 0x7fdf9c159d50 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa2f9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c06a1c0 con 0x7fdf88005180 2023-08-25T10:01:53.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa2f9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9c1eddd0 con 0x7fdf9c0706a0 2023-08-25T10:01:53.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 0x7fdf9c1ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 0x7fdf9c1ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36666/0 (socket says 172.21.15.71:36666) 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 -- 172.21.15.71:0/3939305916 learned_addr learned my addr 172.21.15.71:0/3939305916 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1f99640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf88005180 0x7fdf9c14c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 -- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf9c0706a0 msgr2=0x7fdf9c06fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf9c0706a0 0x7fdf9c06fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 -- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf88005180 msgr2=0x7fdf9c14c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf88005180 0x7fdf9c14c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1798640 1 -- 172.21.15.71:0/3939305916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c157660 con 0x7fdf9c159d50 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.398+0000 7fdfa1f99640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf88005180 0x7fdf9c14c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.399+0000 7fdfa1798640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 0x7fdf9c1ee890 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00e730 tx=0x7fdf8c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.399+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00c6c0 con 0x7fdf9c159d50 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.399+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c00c840 con 0x7fdf9c159d50 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.399+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00cbf0 con 0x7fdf9c159d50 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.399+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf9c15c720 con 0x7fdf9c159d50 2023-08-25T10:01:53.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.399+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf9c1ee370 con 0x7fdf9c159d50 2023-08-25T10:01:53.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.401+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9c04e420 con 0x7fdf9c159d50 2023-08-25T10:01:53.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.402+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fdf8c01d460 con 0x7fdf9c159d50 2023-08-25T10:01:53.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.403+0000 7fdf927fc640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdf8006a080 0x7fdf8006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:53.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.403+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fdf8c088b20 con 0x7fdf9c159d50 2023-08-25T10:01:53.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.404+0000 7fdfa1f99640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdf8006a080 0x7fdf8006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:53.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.404+0000 7fdfa1f99640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdf8006a080 0x7fdf8006c540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf98001d40 tx=0x7fdf98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:53.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.408+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fdf8c053d40 con 0x7fdf9c159d50 2023-08-25T10:01:53.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.673+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf9c0c5c00 con 0x7fdf9c159d50 2023-08-25T10:01:53.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.674+0000 7fdf927fc640 1 -- 172.21.15.71:0/3939305916 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7fdf8c057360 con 0x7fdf9c159d50 2023-08-25T10:01:53.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:53.678 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:53.679 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:53.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.677+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdf8006a080 msgr2=0x7fdf8006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.677+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdf8006a080 0x7fdf8006c540 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf98001d40 tx=0x7fdf98009040 comp rx=0 tx=0).stop 2023-08-25T10:01:53.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.677+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 msgr2=0x7fdf9c1ee890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:53.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.677+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 0x7fdf9c1ee890 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00e730 tx=0x7fdf8c00ebf0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 shutdown_connections 2023-08-25T10:01:53.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdf8006a080 0x7fdf8006c540 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf9c0706a0 0x7fdf9c06fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdf9c159d50 0x7fdf9c1ee890 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 --2- 172.21.15.71:0/3939305916 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf88005180 0x7fdf9c14c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:53.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 >> 172.21.15.71:0/3939305916 conn(0x7fdf9c06d0d0 msgr2=0x7fdf9c156990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:53.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 shutdown_connections 2023-08-25T10:01:53.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:53.678+0000 7fdfa2f9b640 1 -- 172.21.15.71:0/3939305916 wait complete. 2023-08-25T10:01:53.702 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:01:53.703 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:01:34.828305+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:01:55.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:55.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:01:55.925 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:01:55.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.952+0000 7ff602408640 1 Processor -- start 2023-08-25T10:01:55.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.952+0000 7ff602408640 1 -- start start 2023-08-25T10:01:55.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff602408640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb570 0x7ff5fc0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:55.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff602408640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff5fc15cd00 0x7ff5fc158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:55.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff602408640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0b1af0 con 0x7ff5fc0cb190 2023-08-25T10:01:55.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff602408640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc0c98f0 con 0x7ff5fc15cd00 2023-08-25T10:01:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff602408640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc15d190 con 0x7ff5fc0cb570 2023-08-25T10:01:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff601406640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff5fc0cb190 0x7ff5fc0c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:39616/0 (socket says 172.21.15.71:39616) 2023-08-25T10:01:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff601406640 1 -- 172.21.15.71:0/2755461681 learned_addr learned my addr 172.21.15.71:0/2755461681 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff600c05640 1 --2- 172.21.15.71:0/2755461681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb570 0x7ff5fc0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.954+0000 7ff601c07640 1 --2- 172.21.15.71:0/2755461681 >> v2:172.21.15.71:3300/0 conn(0x7ff5fc15cd00 0x7ff5fc158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.955+0000 7ff5f3fff640 1 -- 172.21.15.71:0/2755461681 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015014356 0 0) 0x7ff5fc0b1af0 con 0x7ff5fc0cb190 2023-08-25T10:01:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.955+0000 7ff600c05640 1 -- 172.21.15.71:0/2755461681 >> v2:172.21.15.71:3300/0 conn(0x7ff5fc15cd00 msgr2=0x7ff5fc158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.955+0000 7ff600c05640 1 --2- 172.21.15.71:0/2755461681 >> v2:172.21.15.71:3300/0 conn(0x7ff5fc15cd00 0x7ff5fc158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.955+0000 7ff600c05640 1 -- 172.21.15.71:0/2755461681 >> v1:172.21.15.164:6789/0 conn(0x7ff5fc0cb190 legacy=0x7ff5fc0c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.955+0000 7ff600c05640 1 -- 172.21.15.71:0/2755461681 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0b0fb0 con 0x7ff5fc0cb570 2023-08-25T10:01:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.957+0000 7ff600c05640 1 --2- 172.21.15.71:0/2755461681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb570 0x7ff5fc0ca910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0097a0 tx=0x7ff5ec02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ad64e16cbe9695d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.957+0000 7ff5f3fff640 1 -- 172.21.15.71:0/2755461681 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec030800 con 0x7ff5fc0cb570 2023-08-25T10:01:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.957+0000 7ff5f3fff640 1 -- 172.21.15.71:0/2755461681 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec030960 con 0x7ff5fc0cb570 2023-08-25T10:01:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.957+0000 7ff5f3fff640 1 -- 172.21.15.71:0/2755461681 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec030cf0 con 0x7ff5fc0cb570 2023-08-25T10:01:55.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 -- 172.21.15.71:0/2755461681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb570 msgr2=0x7ff5fc0ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 --2- 172.21.15.71:0/2755461681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb570 0x7ff5fc0ca910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0097a0 tx=0x7ff5ec02b940 comp rx=0 tx=0).stop 2023-08-25T10:01:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 -- 172.21.15.71:0/2755461681 shutdown_connections 2023-08-25T10:01:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 --2- 172.21.15.71:0/2755461681 >> v2:172.21.15.71:3300/0 conn(0x7ff5fc15cd00 0x7ff5fc158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 --2- 172.21.15.71:0/2755461681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb570 0x7ff5fc0ca910 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 -- 172.21.15.71:0/2755461681 >> 172.21.15.71:0/2755461681 conn(0x7ff5fc06ae20 msgr2=0x7ff5fc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.958+0000 7ff602408640 1 -- 172.21.15.71:0/2755461681 shutdown_connections 2023-08-25T10:01:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.959+0000 7ff602408640 1 -- 172.21.15.71:0/2755461681 wait complete. 2023-08-25T10:01:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.961+0000 7ff602408640 1 Processor -- start 2023-08-25T10:01:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.961+0000 7ff602408640 1 -- start start 2023-08-25T10:01:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff602408640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb190 0x7ff5fc14cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff602408640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5fc0cb570 0x7ff5fc1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff602408640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 0x7ff5fc155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff602408640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14caa0 con 0x7ff5fc15cd00 2023-08-25T10:01:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff602408640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc1eef20 con 0x7ff5fc0cb570 2023-08-25T10:01:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff602408640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc14c450 con 0x7ff5fc0cb190 2023-08-25T10:01:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff601c07640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 0x7ff5fc155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff601c07640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 0x7ff5fc155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36684/0 (socket says 172.21.15.71:36684) 2023-08-25T10:01:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff601c07640 1 -- 172.21.15.71:0/3255429346 learned_addr learned my addr 172.21.15.71:0/3255429346 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:55.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff601406640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb190 0x7ff5fc14cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:55.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.963+0000 7ff600c05640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5fc0cb570 0x7ff5fc1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:55.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff601c07640 1 -- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb190 msgr2=0x7ff5fc14cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:55.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff601c07640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb190 0x7ff5fc14cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:55.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff601c07640 1 -- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5fc0cb570 msgr2=0x7ff5fc1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:55.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff601c07640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5fc0cb570 0x7ff5fc1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff601c07640 1 -- 172.21.15.71:0/3255429346 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec009400 con 0x7ff5fc15cd00 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff601c07640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 0x7ff5fc155b80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400efc0 tx=0x7ff5f400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.964+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4014480 con 0x7ff5fc15cd00 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.965+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f400f040 con 0x7ff5fc15cd00 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.965+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc14c730 con 0x7ff5fc15cd00 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.965+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4014820 con 0x7ff5fc15cd00 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.965+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc1ef1e0 con 0x7ff5fc15cd00 2023-08-25T10:01:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.966+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc14cc50 con 0x7ff5fc15cd00 2023-08-25T10:01:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.967+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff5f4014980 con 0x7ff5fc15cd00 2023-08-25T10:01:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.968+0000 7ff5f1ffb640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff5e006a050 0x7ff5e006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.968+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff5f4089420 con 0x7ff5fc15cd00 2023-08-25T10:01:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.969+0000 7ff601406640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff5e006a050 0x7ff5e006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.970+0000 7ff601406640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff5e006a050 0x7ff5e006c510 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8000f60 tx=0x7ff5f8007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:55.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:55.975+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff5fc14cc50 con 0x7ff5fc15cd00 2023-08-25T10:01:56.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.231+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5fc156840 con 0x7ff5fc15cd00 2023-08-25T10:01:56.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.231+0000 7ff5f1ffb640 1 -- 172.21.15.71:0/3255429346 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7ff5fc156840 con 0x7ff5fc15cd00 2023-08-25T10:01:56.234 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:56.234 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:56.234 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff5e006a050 msgr2=0x7ff5e006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff5e006a050 0x7ff5e006c510 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8000f60 tx=0x7ff5f8007e40 comp rx=0 tx=0).stop 2023-08-25T10:01:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 msgr2=0x7ff5fc155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 0x7ff5fc155b80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400efc0 tx=0x7ff5f400c490 comp rx=0 tx=0).stop 2023-08-25T10:01:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 shutdown_connections 2023-08-25T10:01:56.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5fc0cb570 0x7ff5fc1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff5e006a050 0x7ff5e006c510 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff5fc15cd00 0x7ff5fc155b80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 --2- 172.21.15.71:0/3255429346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5fc0cb190 0x7ff5fc14cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:56.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 >> 172.21.15.71:0/3255429346 conn(0x7ff5fc06ae20 msgr2=0x7ff5fc0c7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 shutdown_connections 2023-08-25T10:01:56.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:56.234+0000 7ff602408640 1 -- 172.21.15.71:0/3255429346 wait complete. 2023-08-25T10:01:56.255 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:01:58.256 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:01:58.484 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.483+0000 7f4d77270640 1 Processor -- start 2023-08-25T10:01:58.485 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.483+0000 7f4d77270640 1 -- start start 2023-08-25T10:01:58.486 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.484+0000 7f4d77270640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 0x7f4d700c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.484+0000 7f4d77270640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4d700c3a40 0x7f4d7006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.484+0000 7f4d77270640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d700c1dc0 con 0x7f4d701509d0 2023-08-25T10:01:58.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.484+0000 7f4d77270640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d70157a80 con 0x7f4d700c3a40 2023-08-25T10:01:58.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.484+0000 7f4d77270640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7006cf70 con 0x7f4d700c3660 2023-08-25T10:01:58.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.485+0000 7f4d7580d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4d701509d0 0x7f4d70150e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42260/0 (socket says 172.21.15.71:42260) 2023-08-25T10:01:58.487 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.485+0000 7f4d7580d640 1 -- 172.21.15.71:0/3052659343 learned_addr learned my addr 172.21.15.71:0/3052659343 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:01:58.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.485+0000 7f4d7500c640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 0x7f4d700c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.485+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146987326 0 0) 0x7f4d700c1dc0 con 0x7f4d701509d0 2023-08-25T10:01:58.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.485+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d5c003520 con 0x7f4d701509d0 2023-08-25T10:01:58.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.485+0000 7f4d7480b640 1 --2- 172.21.15.71:0/3052659343 >> v2:172.21.15.164:3300/0 conn(0x7f4d700c3a40 0x7f4d7006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 713934661 0 0) 0x7f4d5c003520 con 0x7f4d701509d0 2023-08-25T10:01:58.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 msgr2=0x7f4d700c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:58.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 0x7f4d700c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 >> v2:172.21.15.164:3300/0 conn(0x7f4d700c3a40 msgr2=0x7f4d7006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:58.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 --2- 172.21.15.71:0/3052659343 >> v2:172.21.15.164:3300/0 conn(0x7f4d700c3a40 0x7f4d7006ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70156e00 con 0x7f4d701509d0 2023-08-25T10:01:58.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4d680019f0 con 0x7f4d701509d0 2023-08-25T10:01:58.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 >> v1:172.21.15.71:6789/0 conn(0x7f4d701509d0 legacy=0x7f4d70150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:58.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d5c0039e0 0x7f4d5c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d5c0047c0 0x7f4d5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 0x7f4d5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d70157a80 con 0x7f4d5c0047c0 2023-08-25T10:01:58.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7006cf70 con 0x7f4d5c0039e0 2023-08-25T10:01:58.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5c00ba90 con 0x7f4d700c3660 2023-08-25T10:01:58.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.486+0000 7f4d7480b640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d5c0047c0 0x7f4d5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.487+0000 7f4d7480b640 1 -- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 msgr2=0x7f4d5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:58.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.487+0000 7f4d7480b640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d700c3660 0x7f4d5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.487+0000 7f4d7480b640 1 -- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d5c0039e0 msgr2=0x7f4d5c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:01:58.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.487+0000 7f4d7480b640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d5c0039e0 0x7f4d5c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.487+0000 7f4d7480b640 1 -- 172.21.15.71:0/3052659343 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c003520 con 0x7f4d5c0047c0 2023-08-25T10:01:58.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.488+0000 7f4d7480b640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d5c0047c0 0x7f4d5c004c40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000ba30 tx=0x7f4d5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e829b4ba777798 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:58.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.488+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60032680 con 0x7f4d5c0047c0 2023-08-25T10:01:58.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.488+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d600327e0 con 0x7f4d5c0047c0 2023-08-25T10:01:58.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.488+0000 7f4d67fff640 1 -- 172.21.15.71:0/3052659343 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60032b70 con 0x7f4d5c0047c0 2023-08-25T10:01:58.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d77270640 1 -- 172.21.15.71:0/3052659343 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d5c0047c0 msgr2=0x7f4d5c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:58.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d77270640 1 --2- 172.21.15.71:0/3052659343 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d5c0047c0 0x7f4d5c004c40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000ba30 tx=0x7f4d5c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d7580d640 1 -- 172.21.15.71:0/3052659343 reap_dead start 2023-08-25T10:01:58.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d77270640 1 -- 172.21.15.71:0/3052659343 shutdown_connections 2023-08-25T10:01:58.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d77270640 1 -- 172.21.15.71:0/3052659343 >> 172.21.15.71:0/3052659343 conn(0x7f4d700654f0 msgr2=0x7f4d70065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:58.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d77270640 1 -- 172.21.15.71:0/3052659343 shutdown_connections 2023-08-25T10:01:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.489+0000 7f4d77270640 1 -- 172.21.15.71:0/3052659343 wait complete. 2023-08-25T10:01:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.491+0000 7f4d77270640 1 Processor -- start 2023-08-25T10:01:58.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.492+0000 7f4d77270640 1 -- start start 2023-08-25T10:01:58.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.492+0000 7f4d77270640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d5c0047c0 0x7f4d701ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.492+0000 7f4d77270640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 0x7f4d7014b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.492+0000 7f4d77270640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d70149730 0x7f4d70149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d77270640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701ef5c0 con 0x7f4d700c9950 2023-08-25T10:01:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d77270640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7014ca70 con 0x7f4d70149730 2023-08-25T10:01:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d77270640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7014cd50 con 0x7f4d5c0047c0 2023-08-25T10:01:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 0x7f4d7014b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 0x7f4d7014b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36698/0 (socket says 172.21.15.71:36698) 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 -- 172.21.15.71:0/182780112 learned_addr learned my addr 172.21.15.71:0/182780112 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7500c640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d5c0047c0 0x7f4d701ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7580d640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d70149730 0x7f4d70149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 -- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d5c0047c0 msgr2=0x7f4d701ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d5c0047c0 0x7f4d701ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 -- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d70149730 msgr2=0x7f4d70149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d70149730 0x7f4d70149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.493+0000 7f4d7480b640 1 -- 172.21.15.71:0/182780112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7014ac50 con 0x7f4d700c9950 2023-08-25T10:01:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.494+0000 7f4d7480b640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 0x7f4d7014b730 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4d6002bf00 tx=0x7f4d60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.494+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60030860 con 0x7f4d700c9950 2023-08-25T10:01:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.494+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d600309e0 con 0x7f4d700c9950 2023-08-25T10:01:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.494+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d60030d90 con 0x7f4d700c9950 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.494+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7014af30 con 0x7f4d700c9950 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.494+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d701efa70 con 0x7f4d700c9950 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.496+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70069160 con 0x7f4d700c9950 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.497+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4d6003b420 con 0x7f4d700c9950 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.498+0000 7f4d65ffb640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d5406a080 0x7f4d5406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.498+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4d600aad80 con 0x7f4d700c9950 2023-08-25T10:01:58.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.498+0000 7f4d7500c640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d5406a080 0x7f4d5406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:01:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.499+0000 7f4d7500c640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d5406a080 0x7f4d5406c540 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009970 tx=0x7f4d6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:01:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.503+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4d60075fc0 con 0x7f4d700c9950 2023-08-25T10:01:58.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.764+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d70154680 con 0x7f4d700c9950 2023-08-25T10:01:58.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.765+0000 7f4d65ffb640 1 -- 172.21.15.71:0/182780112 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f4d60036090 con 0x7f4d700c9950 2023-08-25T10:01:58.767 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:01:58.768 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:01:58.768 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:01:58.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d5406a080 msgr2=0x7f4d5406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:58.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d5406a080 0x7f4d5406c540 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009970 tx=0x7f4d6c008040 comp rx=0 tx=0).stop 2023-08-25T10:01:58.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 msgr2=0x7f4d7014b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:01:58.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 0x7f4d7014b730 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4d6002bf00 tx=0x7f4d60004730 comp rx=0 tx=0).stop 2023-08-25T10:01:58.770 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 shutdown_connections 2023-08-25T10:01:58.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d70149730 0x7f4d70149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4d5406a080 0x7f4d5406c540 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d700c9950 0x7f4d7014b730 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 --2- 172.21.15.71:0/182780112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4d5c0047c0 0x7f4d701ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:01:58.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 >> 172.21.15.71:0/182780112 conn(0x7f4d700654f0 msgr2=0x7f4d70155900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:01:58.771 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 shutdown_connections 2023-08-25T10:01:58.772 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:01:58.768+0000 7f4d77270640 1 -- 172.21.15.71:0/182780112 wait complete. 2023-08-25T10:01:58.790 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:00.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:01.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.009+0000 7f6f94e52640 1 Processor -- start 2023-08-25T10:02:01.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.010+0000 7f6f94e52640 1 -- start start 2023-08-25T10:02:01.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.012+0000 7f6f94e52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f901576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.012+0000 7f6f94e52640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f90157bf0 con 0x7f6f901542d0 2023-08-25T10:02:01.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.012+0000 7f6f94e52640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9006a130 con 0x7f6f900c4fb0 2023-08-25T10:02:01.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.012+0000 7f6f94e52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900c4b60 con 0x7f6f90069850 2023-08-25T10:02:01.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.012+0000 7f6f8f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6f901542d0 0x7f6f900c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54808/0 (socket says 172.21.15.71:54808) 2023-08-25T10:02:01.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.012+0000 7f6f8f7fe640 1 -- 172.21.15.71:0/2698528054 learned_addr learned my addr 172.21.15.71:0/2698528054 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:01.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f901576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822116669 0 0) 0x7f6f80001110 con 0x7f6f901542d0 2023-08-25T10:02:01.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f78003520 con 0x7f6f901542d0 2023-08-25T10:02:01.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2270054102 0 0) 0x7f6f78003520 con 0x7f6f901542d0 2023-08-25T10:02:01.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 msgr2=0x7f6f901576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:01.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f901576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 >> v1:172.21.15.164:6789/0 conn(0x7f6f900c4fb0 legacy=0x7f6f900c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f901546b0 con 0x7f6f901542d0 2023-08-25T10:02:01.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f6f80001800 con 0x7f6f901542d0 2023-08-25T10:02:01.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 >> v1:172.21.15.71:6789/0 conn(0x7f6f901542d0 legacy=0x7f6f900c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f78003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f78004d00 0x7f6f78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900c4b60 con 0x7f6f78003f90 2023-08-25T10:02:01.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7800ba90 con 0x7f6f78004d00 2023-08-25T10:02:01.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.013+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7800bc50 con 0x7f6f90069850 2023-08-25T10:02:01.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8f7fe640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f78003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8ffff640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f78004d00 0x7f6f78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8effd640 1 -- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 msgr2=0x7f6f78003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:01.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f78003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8effd640 1 -- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f78004d00 msgr2=0x7f6f78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:01.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f78004d00 0x7f6f78005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8effd640 1 -- 172.21.15.71:0/2698528054 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f901546b0 con 0x7f6f78003f90 2023-08-25T10:02:01.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8ffff640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f78004d00 0x7f6f78005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:01.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.014+0000 7f6f8f7fe640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f90069850 0x7f6f78003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:01.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.016+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f780047c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009a90 tx=0x7f6f780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2722509fe01e64 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:01.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.016+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8402c680 con 0x7f6f78003f90 2023-08-25T10:02:01.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.016+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8402c7e0 con 0x7f6f78003f90 2023-08-25T10:02:01.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.016+0000 7f6f8e7fc640 1 -- 172.21.15.71:0/2698528054 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8402cb70 con 0x7f6f78003f90 2023-08-25T10:02:01.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f94e52640 1 -- 172.21.15.71:0/2698528054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 msgr2=0x7f6f780047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2698528054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f780047c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009a90 tx=0x7f6f780057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f8ffff640 1 -- 172.21.15.71:0/2698528054 reap_dead start 2023-08-25T10:02:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f94e52640 1 -- 172.21.15.71:0/2698528054 shutdown_connections 2023-08-25T10:02:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f94e52640 1 -- 172.21.15.71:0/2698528054 >> 172.21.15.71:0/2698528054 conn(0x7f6f9006c280 msgr2=0x7f6f9006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f94e52640 1 -- 172.21.15.71:0/2698528054 shutdown_connections 2023-08-25T10:02:01.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.017+0000 7f6f94e52640 1 -- 172.21.15.71:0/2698528054 wait complete. 2023-08-25T10:02:01.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.020+0000 7f6f94e52640 1 Processor -- start 2023-08-25T10:02:01.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.020+0000 7f6f94e52640 1 -- start start 2023-08-25T10:02:01.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f94e52640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f900c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f94e52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f900adf80 0x7f6f90157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f94e52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f901554b0 0x7f6f90147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f94e52640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90156df0 con 0x7f6f78003f90 2023-08-25T10:02:01.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f94e52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90156930 con 0x7f6f901554b0 2023-08-25T10:02:01.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f94e52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90147cb0 con 0x7f6f900adf80 2023-08-25T10:02:01.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f8f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f900c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f8f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f900c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60644/0 (socket says 172.21.15.71:60644) 2023-08-25T10:02:01.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f8f7fe640 1 -- 172.21.15.71:0/2397733167 learned_addr learned my addr 172.21.15.71:0/2397733167 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:01.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.021+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f900adf80 0x7f6f90157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8ffff640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f901554b0 0x7f6f90147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8f7fe640 1 -- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f900adf80 msgr2=0x7f6f90157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8f7fe640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f900adf80 0x7f6f90157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8f7fe640 1 -- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f901554b0 msgr2=0x7f6f90147090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8f7fe640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f901554b0 0x7f6f90147090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8f7fe640 1 -- 172.21.15.71:0/2397733167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900c1840 con 0x7f6f78003f90 2023-08-25T10:02:01.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f900adf80 0x7f6f90157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:01.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8ffff640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f901554b0 0x7f6f90147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:01.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f8f7fe640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f900c8ac0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6f80000fb0 tx=0x7f6f800077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:01.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f800017a0 con 0x7f6f78003f90 2023-08-25T10:02:01.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8000e950 con 0x7f6f78003f90 2023-08-25T10:02:01.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f900c1b20 con 0x7f6f78003f90 2023-08-25T10:02:01.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8000ed00 con 0x7f6f78003f90 2023-08-25T10:02:01.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.022+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f900c1e00 con 0x7f6f78003f90 2023-08-25T10:02:01.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.024+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9004e420 con 0x7f6f78003f90 2023-08-25T10:02:01.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.025+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6f8001d460 con 0x7f6f78003f90 2023-08-25T10:02:01.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.026+0000 7f6f77fff640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6f7c06a080 0x7f6f7c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:01.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.026+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6f80088b00 con 0x7f6f78003f90 2023-08-25T10:02:01.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.026+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6f7c06a080 0x7f6f7c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:01.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.027+0000 7f6f8effd640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6f7c06a080 0x7f6f7c06c540 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6f84036000 tx=0x7f6f84030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:01.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.032+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6f80053ca0 con 0x7f6f78003f90 2023-08-25T10:02:01.296 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.293+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f90157bf0 con 0x7f6f78003f90 2023-08-25T10:02:01.297 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.294+0000 7f6f77fff640 1 -- 172.21.15.71:0/2397733167 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f6f90157bf0 con 0x7f6f78003f90 2023-08-25T10:02:01.297 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:01.297 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:01.298 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:02:01.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6f7c06a080 msgr2=0x7f6f7c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6f7c06a080 0x7f6f7c06c540 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6f84036000 tx=0x7f6f84030000 comp rx=0 tx=0).stop 2023-08-25T10:02:01.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 msgr2=0x7f6f900c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:01.300 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f900c8ac0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f6f80000fb0 tx=0x7f6f800077e0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 shutdown_connections 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f901554b0 0x7f6f90147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6f7c06a080 0x7f6f7c06c540 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f900adf80 0x7f6f90157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 --2- 172.21.15.71:0/2397733167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6f78003f90 0x7f6f900c8ac0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.297+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 >> 172.21.15.71:0/2397733167 conn(0x7f6f9006c280 msgr2=0x7f6f900c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.298+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 shutdown_connections 2023-08-25T10:02:01.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:01.298+0000 7f6f94e52640 1 -- 172.21.15.71:0/2397733167 wait complete. 2023-08-25T10:02:01.318 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:03.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.552+0000 7f2b6d50c640 1 Processor -- start 2023-08-25T10:02:03.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.552+0000 7f2b6d50c640 1 -- start start 2023-08-25T10:02:03.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.554+0000 7f2b6d50c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2b680c3a40 0x7f2b680c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.555+0000 7f2b6d50c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b680c1dc0 con 0x7f2b681509d0 2023-08-25T10:02:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.555+0000 7f2b6d50c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b68156e00 con 0x7f2b680c3660 2023-08-25T10:02:03.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.555+0000 7f2b6d50c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b68157a80 con 0x7f2b680c3a40 2023-08-25T10:02:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.555+0000 7f2b667fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2b680c3a40 0x7f2b680c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.555+0000 7f2b667fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2b680c3a40 0x7f2b680c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60658/0 (socket says 172.21.15.71:60658) 2023-08-25T10:02:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.555+0000 7f2b667fc640 1 -- 172.21.15.71:0/4208856808 learned_addr learned my addr 172.21.15.71:0/4208856808 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:03.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.556+0000 7f2b667fc640 1 -- 172.21.15.71:0/4208856808 >> v1:172.21.15.164:6789/0 conn(0x7f2b680c3660 legacy=0x7f2b6806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.556+0000 7f2b667fc640 1 -- 172.21.15.71:0/4208856808 >> v1:172.21.15.71:6789/0 conn(0x7f2b681509d0 legacy=0x7f2b68150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.556+0000 7f2b667fc640 1 -- 172.21.15.71:0/4208856808 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b681570e0 con 0x7f2b680c3a40 2023-08-25T10:02:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.557+0000 7f2b667fc640 1 --2- 172.21.15.71:0/4208856808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3a40 0x7f2b680c1710 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009870 tx=0x7f2b5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9c88b2ff35dafd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:03.564 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.557+0000 7f2b65ffb640 1 -- 172.21.15.71:0/4208856808 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0317d0 con 0x7f2b680c3a40 2023-08-25T10:02:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.557+0000 7f2b65ffb640 1 -- 172.21.15.71:0/4208856808 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c031930 con 0x7f2b680c3a40 2023-08-25T10:02:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b65ffb640 1 -- 172.21.15.71:0/4208856808 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c031cc0 con 0x7f2b680c3a40 2023-08-25T10:02:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 -- 172.21.15.71:0/4208856808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3a40 msgr2=0x7f2b680c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/4208856808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3a40 0x7f2b680c1710 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009870 tx=0x7f2b5c02b8f0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 -- 172.21.15.71:0/4208856808 shutdown_connections 2023-08-25T10:02:03.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/4208856808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3a40 0x7f2b680c1710 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 -- 172.21.15.71:0/4208856808 >> 172.21.15.71:0/4208856808 conn(0x7f2b680654f0 msgr2=0x7f2b68065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 -- 172.21.15.71:0/4208856808 shutdown_connections 2023-08-25T10:02:03.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.558+0000 7f2b6d50c640 1 -- 172.21.15.71:0/4208856808 wait complete. 2023-08-25T10:02:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.561+0000 7f2b6d50c640 1 Processor -- start 2023-08-25T10:02:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.561+0000 7f2b6d50c640 1 -- start start 2023-08-25T10:02:03.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b6d50c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 0x7f2b681eca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b6d50c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b680c3a40 0x7f2b681538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b6d50c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b681509d0 0x7f2b681e37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b6d50c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6814da10 con 0x7f2b680c3660 2023-08-25T10:02:03.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b6d50c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68143ad0 con 0x7f2b680c3a40 2023-08-25T10:02:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b6d50c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680c8990 con 0x7f2b681509d0 2023-08-25T10:02:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b66ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 0x7f2b681eca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b66ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 0x7f2b681eca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60666/0 (socket says 172.21.15.71:60666) 2023-08-25T10:02:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.562+0000 7f2b66ffd640 1 -- 172.21.15.71:0/413093921 learned_addr learned my addr 172.21.15.71:0/413093921 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:03.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b667fc640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b680c3a40 0x7f2b681538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b677fe640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b681509d0 0x7f2b681e37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b66ffd640 1 -- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b681509d0 msgr2=0x7f2b681e37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b66ffd640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b681509d0 0x7f2b681e37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b66ffd640 1 -- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b680c3a40 msgr2=0x7f2b681538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b66ffd640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b680c3a40 0x7f2b681538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b66ffd640 1 -- 172.21.15.71:0/413093921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0094d0 con 0x7f2b680c3660 2023-08-25T10:02:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b677fe640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b681509d0 0x7f2b681e37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b667fc640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b680c3a40 0x7f2b681538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b66ffd640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 0x7f2b681eca50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800c400 tx=0x7f2b5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:03.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580052d0 con 0x7f2b680c3660 2023-08-25T10:02:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b58005430 con 0x7f2b680c3660 2023-08-25T10:02:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b681472c0 con 0x7f2b680c3660 2023-08-25T10:02:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.563+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58003830 con 0x7f2b680c3660 2023-08-25T10:02:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.564+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b68147510 con 0x7f2b680c3660 2023-08-25T10:02:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.565+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68069160 con 0x7f2b680c3660 2023-08-25T10:02:03.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.566+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2b58003990 con 0x7f2b680c3660 2023-08-25T10:02:03.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.567+0000 7f2b4f7fe640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b5406a050 0x7f2b5406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:03.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.567+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f2b58088110 con 0x7f2b680c3660 2023-08-25T10:02:03.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.567+0000 7f2b667fc640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b5406a050 0x7f2b5406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:03.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.568+0000 7f2b667fc640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b5406a050 0x7f2b5406c510 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c036000 tx=0x7f2b5c03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:03.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.572+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2b58053210 con 0x7f2b680c3660 2023-08-25T10:02:03.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.840+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b68154680 con 0x7f2b680c3660 2023-08-25T10:02:03.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.841+0000 7f2b4f7fe640 1 -- 172.21.15.71:0/413093921 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f2b58017090 con 0x7f2b680c3660 2023-08-25T10:02:03.843 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:03.843 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:03.844 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:02:03.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b5406a050 msgr2=0x7f2b5406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b5406a050 0x7f2b5406c510 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c036000 tx=0x7f2b5c03d040 comp rx=0 tx=0).stop 2023-08-25T10:02:03.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 msgr2=0x7f2b681eca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:03.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 0x7f2b681eca50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f2b5800c400 tx=0x7f2b5800c8c0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 shutdown_connections 2023-08-25T10:02:03.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b680c3a40 0x7f2b681538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2b5406a050 0x7f2b5406c510 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b681509d0 0x7f2b681e37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 --2- 172.21.15.71:0/413093921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2b680c3660 0x7f2b681eca50 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:03.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 >> 172.21.15.71:0/413093921 conn(0x7f2b680654f0 msgr2=0x7f2b680658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:03.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 shutdown_connections 2023-08-25T10:02:03.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:03.844+0000 7f2b6d50c640 1 -- 172.21.15.71:0/413093921 wait complete. 2023-08-25T10:02:03.864 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:05.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:06.101 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.099+0000 7f63333b5640 1 Processor -- start 2023-08-25T10:02:06.101 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.099+0000 7f63333b5640 1 -- start start 2023-08-25T10:02:06.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f63333b5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f632c153310 0x7f632c06a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f63333b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f632c0cec00 0x7f632c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f63333b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0c3e00 0x7f632c0c4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f63333b5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c0c4790 con 0x7f632c153310 2023-08-25T10:02:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f63333b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c1590c0 con 0x7f632c0cec00 2023-08-25T10:02:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f63333b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c159380 con 0x7f632c0c3e00 2023-08-25T10:02:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.101+0000 7f6331151640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f632c153310 0x7f632c06a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f632c153310 0x7f632c06a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60680/0 (socket says 172.21.15.71:60680) 2023-08-25T10:02:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 -- 172.21.15.71:0/1417486655 learned_addr learned my addr 172.21.15.71:0/1417486655 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:06.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6330950640 1 --2- 172.21.15.71:0/1417486655 >> v2:172.21.15.164:3300/0 conn(0x7f632c0cec00 0x7f632c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331952640 1 --2- 172.21.15.71:0/1417486655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0c3e00 0x7f632c0c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 -- 172.21.15.71:0/1417486655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0c3e00 msgr2=0x7f632c0c4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 --2- 172.21.15.71:0/1417486655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0c3e00 0x7f632c0c4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 -- 172.21.15.71:0/1417486655 >> v2:172.21.15.164:3300/0 conn(0x7f632c0cec00 msgr2=0x7f632c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 --2- 172.21.15.71:0/1417486655 >> v2:172.21.15.164:3300/0 conn(0x7f632c0cec00 0x7f632c0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.102+0000 7f6331151640 1 -- 172.21.15.71:0/1417486655 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c159660 con 0x7f632c153310 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.103+0000 7f6331151640 1 --2- 172.21.15.71:0/1417486655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c06a620 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f632c1536f0 tx=0x7f632802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b81725850eec55c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f6323fff640 1 -- 172.21.15.71:0/1417486655 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328030850 con 0x7f632c153310 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f6323fff640 1 -- 172.21.15.71:0/1417486655 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63280309b0 con 0x7f632c153310 2023-08-25T10:02:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f6323fff640 1 -- 172.21.15.71:0/1417486655 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328030d40 con 0x7f632c153310 2023-08-25T10:02:06.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 -- 172.21.15.71:0/1417486655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 msgr2=0x7f632c06a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:06.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 --2- 172.21.15.71:0/1417486655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c06a620 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f632c1536f0 tx=0x7f632802b990 comp rx=0 tx=0).stop 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 -- 172.21.15.71:0/1417486655 shutdown_connections 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 --2- 172.21.15.71:0/1417486655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0c3e00 0x7f632c0c4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 --2- 172.21.15.71:0/1417486655 >> v2:172.21.15.164:3300/0 conn(0x7f632c0cec00 0x7f632c0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 --2- 172.21.15.71:0/1417486655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c06a620 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.104+0000 7f63333b5640 1 -- 172.21.15.71:0/1417486655 >> 172.21.15.71:0/1417486655 conn(0x7f632c065910 msgr2=0x7f632c065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.105+0000 7f63333b5640 1 -- 172.21.15.71:0/1417486655 shutdown_connections 2023-08-25T10:02:06.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.105+0000 7f63333b5640 1 -- 172.21.15.71:0/1417486655 wait complete. 2023-08-25T10:02:06.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.106+0000 7f63333b5640 1 Processor -- start 2023-08-25T10:02:06.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.107+0000 7f63333b5640 1 -- start start 2023-08-25T10:02:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f63333b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f632c0c3e00 0x7f632c14fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f63333b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0cec00 0x7f632c1f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f63333b5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f63333b5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1f1ab0 con 0x7f632c153310 2023-08-25T10:02:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f63333b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c156c20 con 0x7f632c0c3e00 2023-08-25T10:02:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f63333b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0ce450 con 0x7f632c0cec00 2023-08-25T10:02:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f6331952640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f6331952640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60686/0 (socket says 172.21.15.71:60686) 2023-08-25T10:02:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f6331952640 1 -- 172.21.15.71:0/2367346655 learned_addr learned my addr 172.21.15.71:0/2367346655 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f6330950640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0cec00 0x7f632c1f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.108+0000 7f6331151640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f632c0c3e00 0x7f632c14fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6331952640 1 -- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0cec00 msgr2=0x7f632c1f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:06.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6331952640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0cec00 0x7f632c1f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6331952640 1 -- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f632c0c3e00 msgr2=0x7f632c14fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:06.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6331952640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f632c0c3e00 0x7f632c14fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6331952640 1 -- 172.21.15.71:0/2367346655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63280094d0 con 0x7f632c153310 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6331952640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c143310 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6324007d50 tx=0x7f632400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324014420 con 0x7f632c153310 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63240145a0 con 0x7f632c153310 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324014950 con 0x7f632c153310 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c153b70 con 0x7f632c153310 2023-08-25T10:02:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.109+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c0b39e0 con 0x7f632c153310 2023-08-25T10:02:06.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.111+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c1f1810 con 0x7f632c153310 2023-08-25T10:02:06.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.112+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6324014ab0 con 0x7f632c153310 2023-08-25T10:02:06.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.113+0000 7f6321ffb640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f631006a050 0x7f631006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.113+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f63240892c0 con 0x7f632c153310 2023-08-25T10:02:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.113+0000 7f6331151640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f631006a050 0x7f631006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.114+0000 7f6331151640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f631006a050 0x7f631006c510 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6328036000 tx=0x7f6328008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:06.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.118+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f632c1f1810 con 0x7f632c153310 2023-08-25T10:02:06.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.381+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f632c1f1810 con 0x7f632c153310 2023-08-25T10:02:06.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.382+0000 7f6321ffb640 1 -- 172.21.15.71:0/2367346655 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f632c1f1810 con 0x7f632c153310 2023-08-25T10:02:06.384 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:06.385 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:06.385 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:02:06.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f631006a050 msgr2=0x7f631006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:06.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f631006a050 0x7f631006c510 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6328036000 tx=0x7f6328008030 comp rx=0 tx=0).stop 2023-08-25T10:02:06.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 msgr2=0x7f632c143310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:06.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c143310 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6324007d50 tx=0x7f632400e4b0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 shutdown_connections 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f631006a050 0x7f631006c510 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f632c153310 0x7f632c143310 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f632c0cec00 0x7f632c1f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 --2- 172.21.15.71:0/2367346655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f632c0c3e00 0x7f632c14fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 >> 172.21.15.71:0/2367346655 conn(0x7f632c065910 msgr2=0x7f632c1525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 shutdown_connections 2023-08-25T10:02:06.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:06.385+0000 7f63333b5640 1 -- 172.21.15.71:0/2367346655 wait complete. 2023-08-25T10:02:06.404 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:02:06.405 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:01:34.828305+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:02:08.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:08.648 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.647+0000 7f9f05f53640 1 Processor -- start 2023-08-25T10:02:08.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.647+0000 7f9f05f53640 1 -- start start 2023-08-25T10:02:08.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.648+0000 7f9f05f53640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9f00071760 0x7f9f0006a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:08.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.648+0000 7f9f05f53640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f000c0480 con 0x7f9f0006ad90 2023-08-25T10:02:08.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9f05f53640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f00069850 con 0x7f9f000af290 2023-08-25T10:02:08.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9f05f53640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0006b1a0 con 0x7f9f00071760 2023-08-25T10:02:08.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9efffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9f00071760 0x7f9f0006a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:08.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9f05752640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9f0006ad90 0x7f9f0015c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:54826/0 (socket says 172.21.15.71:54826) 2023-08-25T10:02:08.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9f05752640 1 -- 172.21.15.71:0/1493072995 learned_addr learned my addr 172.21.15.71:0/1493072995 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:08.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9efffff640 1 -- 172.21.15.71:0/1493072995 >> v1:172.21.15.164:6789/0 conn(0x7f9f000af290 legacy=0x7f9f0006b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.649+0000 7f9efffff640 1 -- 172.21.15.71:0/1493072995 >> v1:172.21.15.71:6789/0 conn(0x7f9f0006ad90 legacy=0x7f9f0015c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.650+0000 7f9efffff640 1 -- 172.21.15.71:0/1493072995 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00069ad0 con 0x7f9f00071760 2023-08-25T10:02:08.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.650+0000 7f9eff7fe640 1 -- 172.21.15.71:0/1493072995 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605129725 0 0) 0x7f9f000c0480 con 0x7f9f0006ad90 2023-08-25T10:02:08.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.651+0000 7f9efffff640 1 --2- 172.21.15.71:0/1493072995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f00071760 0x7f9f0006a850 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0097a0 tx=0x7f9eec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2ed0757035d71 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:08.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.651+0000 7f9eff7fe640 1 -- 172.21.15.71:0/1493072995 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec030610 con 0x7f9f00071760 2023-08-25T10:02:08.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.651+0000 7f9eff7fe640 1 -- 172.21.15.71:0/1493072995 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec030770 con 0x7f9f00071760 2023-08-25T10:02:08.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9eff7fe640 1 -- 172.21.15.71:0/1493072995 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec030b00 con 0x7f9f00071760 2023-08-25T10:02:08.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 -- 172.21.15.71:0/1493072995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f00071760 msgr2=0x7f9f0006a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 --2- 172.21.15.71:0/1493072995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f00071760 0x7f9f0006a850 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0097a0 tx=0x7f9eec02b960 comp rx=0 tx=0).stop 2023-08-25T10:02:08.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 -- 172.21.15.71:0/1493072995 shutdown_connections 2023-08-25T10:02:08.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 --2- 172.21.15.71:0/1493072995 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f00071760 0x7f9f0006a850 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 -- 172.21.15.71:0/1493072995 >> 172.21.15.71:0/1493072995 conn(0x7f9f0006c280 msgr2=0x7f9f0006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:08.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 -- 172.21.15.71:0/1493072995 shutdown_connections 2023-08-25T10:02:08.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.652+0000 7f9f05f53640 1 -- 172.21.15.71:0/1493072995 wait complete. 2023-08-25T10:02:08.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.654+0000 7f9f05f53640 1 Processor -- start 2023-08-25T10:02:08.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.655+0000 7f9f05f53640 1 -- start start 2023-08-25T10:02:08.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05f53640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0006ad90 0x7f9f001ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:08.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05f53640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f00071760 0x7f9f00153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:08.658 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05f53640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 0x7f9f0006eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:08.658 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05f53640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0006f760 con 0x7f9f000af290 2023-08-25T10:02:08.658 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05f53640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f000ab400 con 0x7f9f00071760 2023-08-25T10:02:08.658 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05f53640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f00070a50 con 0x7f9f0006ad90 2023-08-25T10:02:08.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05752640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 0x7f9f0006eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:08.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05752640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 0x7f9f0006eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60708/0 (socket says 172.21.15.71:60708) 2023-08-25T10:02:08.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f05752640 1 -- 172.21.15.71:0/340249414 learned_addr learned my addr 172.21.15.71:0/340249414 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:08.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9efffff640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f00071760 0x7f9f00153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:08.660 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.656+0000 7f9f04f51640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0006ad90 0x7f9f001ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:08.661 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f05752640 1 -- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0006ad90 msgr2=0x7f9f001ee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f05752640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0006ad90 0x7f9f001ee620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.662 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f05752640 1 -- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f00071760 msgr2=0x7f9f00153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f05752640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f00071760 0x7f9f00153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.663 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f05752640 1 -- 172.21.15.71:0/340249414 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec009400 con 0x7f9f000af290 2023-08-25T10:02:08.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f04f51640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0006ad90 0x7f9f001ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:08.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.657+0000 7f9f05752640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 0x7f9f0006eae0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c7e0 tx=0x7f9ef800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:08.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.658+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef800a640 con 0x7f9f000af290 2023-08-25T10:02:08.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.658+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef800a7c0 con 0x7f9f000af290 2023-08-25T10:02:08.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.658+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f00154190 con 0x7f9f000af290 2023-08-25T10:02:08.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.658+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef800ab70 con 0x7f9f000af290 2023-08-25T10:02:08.666 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.658+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f00070ff0 con 0x7f9f000af290 2023-08-25T10:02:08.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.659+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0004e420 con 0x7f9f000af290 2023-08-25T10:02:08.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.660+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9ef800acd0 con 0x7f9f000af290 2023-08-25T10:02:08.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.662+0000 7f9efd7fa640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9ee406a050 0x7f9ee406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:08.668 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.662+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9ef8087f50 con 0x7f9f000af290 2023-08-25T10:02:08.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.662+0000 7f9f04f51640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9ee406a050 0x7f9ee406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:08.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.663+0000 7f9f04f51640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9ee406a050 0x7f9ee406c510 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4007fd0 tx=0x7f9ef4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:08.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.666+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9ef8053230 con 0x7f9f000af290 2023-08-25T10:02:08.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.911+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f00154910 con 0x7f9f000af290 2023-08-25T10:02:08.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.912+0000 7f9efd7fa640 1 -- 172.21.15.71:0/340249414 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f9ef8018020 con 0x7f9f000af290 2023-08-25T10:02:08.914 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:08.914 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:08.915 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:02:08.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9ee406a050 msgr2=0x7f9ee406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9ee406a050 0x7f9ee406c510 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4007fd0 tx=0x7f9ef4009040 comp rx=0 tx=0).stop 2023-08-25T10:02:08.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 msgr2=0x7f9f0006eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:08.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 0x7f9f0006eae0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c7e0 tx=0x7f9ef800cca0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 shutdown_connections 2023-08-25T10:02:08.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f00071760 0x7f9f00153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9ee406a050 0x7f9ee406c510 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f000af290 0x7f9f0006eae0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 --2- 172.21.15.71:0/340249414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f0006ad90 0x7f9f001ee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 >> 172.21.15.71:0/340249414 conn(0x7f9f0006c280 msgr2=0x7f9f001e4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 shutdown_connections 2023-08-25T10:02:08.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:08.915+0000 7f9f05f53640 1 -- 172.21.15.71:0/340249414 wait complete. 2023-08-25T10:02:08.935 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:10.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:11.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.175+0000 7f597e8c5640 1 Processor -- start 2023-08-25T10:02:11.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.175+0000 7f597e8c5640 1 -- start start 2023-08-25T10:02:11.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.176+0000 7f597e8c5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f59780c3660 0x7f59780c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:11.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.176+0000 7f597e8c5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f59780c3a40 0x7f597806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:11.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.176+0000 7f597e8c5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59780c1dc0 con 0x7f59781509d0 2023-08-25T10:02:11.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.176+0000 7f597e8c5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5978157a80 con 0x7f59780c3660 2023-08-25T10:02:11.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.176+0000 7f597e8c5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597806cf70 con 0x7f59780c3a40 2023-08-25T10:02:11.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f59780c3660 0x7f59780c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:11.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f59780c3660 0x7f59780c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48866/0 (socket says 172.21.15.71:48866) 2023-08-25T10:02:11.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 -- 172.21.15.71:0/3009299837 learned_addr learned my addr 172.21.15.71:0/3009299837 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:11.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 -- 172.21.15.71:0/3009299837 >> v2:172.21.15.164:3300/0 conn(0x7f59780c3a40 msgr2=0x7f597806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:11.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 --2- 172.21.15.71:0/3009299837 >> v2:172.21.15.164:3300/0 conn(0x7f59780c3a40 0x7f597806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 -- 172.21.15.71:0/3009299837 >> v1:172.21.15.164:6789/0 conn(0x7f59781509d0 legacy=0x7f5978150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:11.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.177+0000 7f5977fff640 1 -- 172.21.15.71:0/3009299837 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978156e00 con 0x7f59780c3660 2023-08-25T10:02:11.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.178+0000 7f5977fff640 1 --2- 172.21.15.71:0/3009299837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3660 0x7f59780c1710 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5978071cc0 tx=0x7f596802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5efc41c08ac7007 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:11.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.179+0000 7f59777fe640 1 -- 172.21.15.71:0/3009299837 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968030860 con 0x7f59780c3660 2023-08-25T10:02:11.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.179+0000 7f59777fe640 1 -- 172.21.15.71:0/3009299837 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59680309c0 con 0x7f59780c3660 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.179+0000 7f59777fe640 1 -- 172.21.15.71:0/3009299837 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968030d50 con 0x7f59780c3660 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.179+0000 7f597e8c5640 1 -- 172.21.15.71:0/3009299837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3660 msgr2=0x7f59780c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.179+0000 7f597e8c5640 1 --2- 172.21.15.71:0/3009299837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3660 0x7f59780c1710 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5978071cc0 tx=0x7f596802b9a0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.180+0000 7f597e8c5640 1 -- 172.21.15.71:0/3009299837 shutdown_connections 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.180+0000 7f597e8c5640 1 --2- 172.21.15.71:0/3009299837 >> v2:172.21.15.164:3300/0 conn(0x7f59780c3a40 0x7f597806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.180+0000 7f597e8c5640 1 --2- 172.21.15.71:0/3009299837 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3660 0x7f59780c1710 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.180+0000 7f597e8c5640 1 -- 172.21.15.71:0/3009299837 >> 172.21.15.71:0/3009299837 conn(0x7f59780654f0 msgr2=0x7f5978065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:11.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.180+0000 7f597e8c5640 1 -- 172.21.15.71:0/3009299837 shutdown_connections 2023-08-25T10:02:11.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.180+0000 7f597e8c5640 1 -- 172.21.15.71:0/3009299837 wait complete. 2023-08-25T10:02:11.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.181+0000 7f597e8c5640 1 Processor -- start 2023-08-25T10:02:11.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.182+0000 7f597e8c5640 1 -- start start 2023-08-25T10:02:11.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597e8c5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59780c3660 0x7f59781f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:11.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597e8c5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 0x7f5978153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:11.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597e8c5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59781509d0 0x7f59781525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:11.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597e8c5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59781f2b70 con 0x7f59780c3a40 2023-08-25T10:02:11.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597e8c5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59781ef140 con 0x7f59780c3660 2023-08-25T10:02:11.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597e8c5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59781ef5d0 con 0x7f59781509d0 2023-08-25T10:02:11.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f596ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 0x7f5978153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:11.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f596ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 0x7f5978153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48876/0 (socket says 172.21.15.71:48876) 2023-08-25T10:02:11.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f596ffff640 1 -- 172.21.15.71:0/1253628254 learned_addr learned my addr 172.21.15.71:0/1253628254 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:11.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f597ce62640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59781509d0 0x7f59781525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:11.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.183+0000 7f5977fff640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59780c3660 0x7f59781f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:11.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f596ffff640 1 -- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59781509d0 msgr2=0x7f59781525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f596ffff640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59781509d0 0x7f59781525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f596ffff640 1 -- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59780c3660 msgr2=0x7f59781f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f596ffff640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59780c3660 0x7f59781f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f596ffff640 1 -- 172.21.15.71:0/1253628254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59680094d0 con 0x7f59780c3a40 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f597ce62640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59781509d0 0x7f59781525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f596ffff640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 0x7f5978153820 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f596000b450 tx=0x7f596000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960011890 con 0x7f59780c3a40 2023-08-25T10:02:11.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59600119f0 con 0x7f59780c3a40 2023-08-25T10:02:11.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.184+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960011da0 con 0x7f59780c3a40 2023-08-25T10:02:11.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.185+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5978154490 con 0x7f59780c3a40 2023-08-25T10:02:11.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.185+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59781541b0 con 0x7f59780c3a40 2023-08-25T10:02:11.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.186+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59780c4990 con 0x7f59780c3a40 2023-08-25T10:02:11.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.187+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f596001a400 con 0x7f59780c3a40 2023-08-25T10:02:11.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.188+0000 7f59757fa640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f595806a050 0x7f595806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:11.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.188+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f5960088fa0 con 0x7f59780c3a40 2023-08-25T10:02:11.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.189+0000 7f5977fff640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f595806a050 0x7f595806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:11.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.189+0000 7f5977fff640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f595806a050 0x7f595806c510 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5968036000 tx=0x7f5968008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:11.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.194+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5960054130 con 0x7f59780c3a40 2023-08-25T10:02:11.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.470+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5978069160 con 0x7f59780c3a40 2023-08-25T10:02:11.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.471+0000 7f59757fa640 1 -- 172.21.15.71:0/1253628254 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f5960015090 con 0x7f59780c3a40 2023-08-25T10:02:11.473 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:11.474 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:11.474 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:02:11.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f595806a050 msgr2=0x7f595806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f595806a050 0x7f595806c510 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5968036000 tx=0x7f5968008030 comp rx=0 tx=0).stop 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 msgr2=0x7f5978153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 0x7f5978153820 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f596000b450 tx=0x7f596000b910 comp rx=0 tx=0).stop 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 shutdown_connections 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f595806a050 0x7f595806c510 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59781509d0 0x7f59781525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f59780c3a40 0x7f5978153820 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 --2- 172.21.15.71:0/1253628254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59780c3660 0x7f59781f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:11.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 >> 172.21.15.71:0/1253628254 conn(0x7f59780654f0 msgr2=0x7f59780c2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:11.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 shutdown_connections 2023-08-25T10:02:11.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:11.474+0000 7f597e8c5640 1 -- 172.21.15.71:0/1253628254 wait complete. 2023-08-25T10:02:11.496 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:13.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:13.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.739+0000 7f9bf359e640 1 Processor -- start 2023-08-25T10:02:13.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.739+0000 7f9bf359e640 1 -- start start 2023-08-25T10:02:13.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.740+0000 7f9bf359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9bf40cb190 0x7f9bf40ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf359e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf40b1af0 con 0x7f9bf415cd00 2023-08-25T10:02:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf359e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf40b0fb0 con 0x7f9bf40cb570 2023-08-25T10:02:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf359e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf40c98f0 con 0x7f9bf40cb190 2023-08-25T10:02:13.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf2d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9bf415cd00 0x7f9bf4158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47384/0 (socket says 172.21.15.71:47384) 2023-08-25T10:02:13.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf2d9d640 1 -- 172.21.15.71:0/4079705590 learned_addr learned my addr 172.21.15.71:0/4079705590 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:13.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf259c640 1 --2- 172.21.15.71:0/4079705590 >> v2:172.21.15.164:3300/0 conn(0x7f9bf40cb190 0x7f9bf40ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680932693 0 0) 0x7f9bf40b1af0 con 0x7f9bf415cd00 2023-08-25T10:02:13.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.741+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bdc003520 con 0x7f9bf415cd00 2023-08-25T10:02:13.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3127296522 0 0) 0x7f9bdc003520 con 0x7f9bf415cd00 2023-08-25T10:02:13.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 >> v2:172.21.15.164:3300/0 conn(0x7f9bf40cb190 msgr2=0x7f9bf40ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:13.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 --2- 172.21.15.71:0/4079705590 >> v2:172.21.15.164:3300/0 conn(0x7f9bf40cb190 0x7f9bf40ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 >> v1:172.21.15.164:6789/0 conn(0x7f9bf40cb570 legacy=0x7f9bf40c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:13.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf415d130 con 0x7f9bf415cd00 2023-08-25T10:02:13.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f9bec0019f0 con 0x7f9bf415cd00 2023-08-25T10:02:13.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 >> v1:172.21.15.71:6789/0 conn(0x7f9bf415cd00 legacy=0x7f9bf4158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:13.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0039e0 0x7f9bdc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc005180 0x7f9bdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40c98f0 con 0x7f9bdc0039e0 2023-08-25T10:02:13.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc00c090 con 0x7f9bdc0047c0 2023-08-25T10:02:13.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc00c250 con 0x7f9bdc005180 2023-08-25T10:02:13.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.742+0000 7f9bf259c640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0039e0 0x7f9bdc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf1d9b640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf2d9d640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc005180 0x7f9bdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf259c640 1 -- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc005180 msgr2=0x7f9bdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:13.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf259c640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc005180 0x7f9bdc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf259c640 1 -- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0047c0 msgr2=0x7f9bdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:13.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf259c640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf259c640 1 -- 172.21.15.71:0/4079705590 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc003520 con 0x7f9bdc0039e0 2023-08-25T10:02:13.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf1d9b640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0047c0 0x7f9bdc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:13.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf2d9d640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc005180 0x7f9bdc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:13.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.743+0000 7f9bf2d9d640 1 -- 172.21.15.71:0/4079705590 reap_dead start 2023-08-25T10:02:13.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.744+0000 7f9bf259c640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0039e0 0x7f9bdc003e00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009ac0 tx=0x7f9bdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf5850ad80dc692 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:13.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.744+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be402c6b0 con 0x7f9bdc0039e0 2023-08-25T10:02:13.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.744+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be402c810 con 0x7f9bdc0039e0 2023-08-25T10:02:13.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf159a640 1 -- 172.21.15.71:0/4079705590 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be402cba0 con 0x7f9bdc0039e0 2023-08-25T10:02:13.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 -- 172.21.15.71:0/4079705590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0039e0 msgr2=0x7f9bdc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:13.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0039e0 0x7f9bdc003e00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009ac0 tx=0x7f9bdc0057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 -- 172.21.15.71:0/4079705590 shutdown_connections 2023-08-25T10:02:13.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 --2- 172.21.15.71:0/4079705590 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bdc0039e0 0x7f9bdc003e00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 -- 172.21.15.71:0/4079705590 >> 172.21.15.71:0/4079705590 conn(0x7f9bf406ae20 msgr2=0x7f9bf406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:13.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 -- 172.21.15.71:0/4079705590 shutdown_connections 2023-08-25T10:02:13.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.745+0000 7f9bf359e640 1 -- 172.21.15.71:0/4079705590 wait complete. 2023-08-25T10:02:13.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.747+0000 7f9bf359e640 1 Processor -- start 2023-08-25T10:02:13.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.747+0000 7f9bf359e640 1 -- start start 2023-08-25T10:02:13.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0039e0 0x7f9bf4158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 0x7f9bf4158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4157520 0x7f9bf4156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf414e8b0 con 0x7f9bf4154f10 2023-08-25T10:02:13.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40c7dc0 con 0x7f9bdc0039e0 2023-08-25T10:02:13.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4156a80 con 0x7f9bf4157520 2023-08-25T10:02:13.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 0x7f9bf4158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 0x7f9bf4158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48894/0 (socket says 172.21.15.71:48894) 2023-08-25T10:02:13.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 -- 172.21.15.71:0/1509988037 learned_addr learned my addr 172.21.15.71:0/1509988037 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:13.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf2d9d640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4157520 0x7f9bf4156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf259c640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0039e0 0x7f9bf4158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 -- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4157520 msgr2=0x7f9bf4156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:13.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4157520 0x7f9bf4156160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 -- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0039e0 msgr2=0x7f9bf4158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:13.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0039e0 0x7f9bf4158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:13.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf1d9b640 1 -- 172.21.15.71:0/1509988037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf40c8030 con 0x7f9bf4154f10 2023-08-25T10:02:13.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.749+0000 7f9bf259c640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0039e0 0x7f9bf4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:13.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.750+0000 7f9bf1d9b640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 0x7f9bf4158f50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9be800d770 tx=0x7f9be800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:13.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.750+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8005270 con 0x7f9bf4154f10 2023-08-25T10:02:13.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.750+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be80053d0 con 0x7f9bf4154f10 2023-08-25T10:02:13.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.750+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf4156d20 con 0x7f9bf4154f10 2023-08-25T10:02:13.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.750+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be800b8d0 con 0x7f9bf4154f10 2023-08-25T10:02:13.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.750+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf41566a0 con 0x7f9bf4154f10 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.751+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf404e420 con 0x7f9bf4154f10 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.752+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9be800ba30 con 0x7f9bf4154f10 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.753+0000 7f9be2ffd640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd406a080 0x7f9bd406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.753+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9be8088ad0 con 0x7f9bf4154f10 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.753+0000 7f9bf259c640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd406a080 0x7f9bd406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.754+0000 7f9bf259c640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd406a080 0x7f9bd406c540 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9be4036000 tx=0x7f9be4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:13.758+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9be8053bd0 con 0x7f9bf4154f10 2023-08-25T10:02:14.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.023+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bf40b2f30 con 0x7f9bf4154f10 2023-08-25T10:02:14.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.024+0000 7f9be2ffd640 1 -- 172.21.15.71:0/1509988037 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+6525 (secure 0 0 0) 0x7f9be8019310 con 0x7f9bf4154f10 2023-08-25T10:02:14.027 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:14.027 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:01:34.828305+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:14.028 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 93 2023-08-25T10:02:14.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd406a080 msgr2=0x7f9bd406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:14.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd406a080 0x7f9bd406c540 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9be4036000 tx=0x7f9be4030000 comp rx=0 tx=0).stop 2023-08-25T10:02:14.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 msgr2=0x7f9bf4158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:14.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 0x7f9bf4158f50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9be800d770 tx=0x7f9be800dc30 comp rx=0 tx=0).stop 2023-08-25T10:02:14.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 shutdown_connections 2023-08-25T10:02:14.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9bd406a080 0x7f9bd406c540 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:14.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bf4157520 0x7f9bf4156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:14.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9bf4154f10 0x7f9bf4158f50 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:14.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 --2- 172.21.15.71:0/1509988037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc0039e0 0x7f9bf4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:14.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 >> 172.21.15.71:0/1509988037 conn(0x7f9bf406ae20 msgr2=0x7f9bf40cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:14.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 shutdown_connections 2023-08-25T10:02:14.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:14.027+0000 7f9bf359e640 1 -- 172.21.15.71:0/1509988037 wait complete. 2023-08-25T10:02:14.047 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:16.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:16.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.301+0000 7f4fef59e640 1 Processor -- start 2023-08-25T10:02:16.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.302+0000 7f4fef59e640 1 -- start start 2023-08-25T10:02:16.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.303+0000 7f4fef59e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4ff00c72f0 0x7f4ff00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:16.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.303+0000 7f4fef59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00c76d0 0x7f4ff0152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:16.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.303+0000 7f4fef59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00c1060 con 0x7f4ff00ad2f0 2023-08-25T10:02:16.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.303+0000 7f4fef59e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff00adf70 con 0x7f4ff00c72f0 2023-08-25T10:02:16.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.303+0000 7f4fef59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00af730 con 0x7f4ff00c76d0 2023-08-25T10:02:16.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4feed9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4ff00ad2f0 0x7f4ff0155c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47394/0 (socket says 172.21.15.71:47394) 2023-08-25T10:02:16.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4feed9d640 1 -- 172.21.15.71:0/1096253199 learned_addr learned my addr 172.21.15.71:0/1096253199 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:16.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fee59c640 1 --2- 172.21.15.71:0/1096253199 >> v2:172.21.15.71:3300/0 conn(0x7f4ff00c72f0 0x7f4ff00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:16.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fedd9b640 1 --2- 172.21.15.71:0/1096253199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00c76d0 0x7f4ff0152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:16.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fed59a640 1 -- 172.21.15.71:0/1096253199 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720759490 0 0) 0x7f4ff00c1060 con 0x7f4ff00ad2f0 2023-08-25T10:02:16.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fee59c640 1 -- 172.21.15.71:0/1096253199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00c76d0 msgr2=0x7f4ff0152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:16.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fee59c640 1 --2- 172.21.15.71:0/1096253199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00c76d0 0x7f4ff0152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fee59c640 1 -- 172.21.15.71:0/1096253199 >> v1:172.21.15.71:6789/0 conn(0x7f4ff00ad2f0 legacy=0x7f4ff0155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:16.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.304+0000 7f4fee59c640 1 -- 172.21.15.71:0/1096253199 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff006b4a0 con 0x7f4ff00c72f0 2023-08-25T10:02:16.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.306+0000 7f4fee59c640 1 --2- 172.21.15.71:0/1096253199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff00acdb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00097a0 tx=0x7f4fe002db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a970b65fa8b6b3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:16.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.306+0000 7f4fed59a640 1 -- 172.21.15.71:0/1096253199 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe002f910 con 0x7f4ff00c72f0 2023-08-25T10:02:16.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.306+0000 7f4fed59a640 1 -- 172.21.15.71:0/1096253199 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe002fa70 con 0x7f4ff00c72f0 2023-08-25T10:02:16.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.306+0000 7f4fed59a640 1 -- 172.21.15.71:0/1096253199 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe002fe00 con 0x7f4ff00c72f0 2023-08-25T10:02:16.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 -- 172.21.15.71:0/1096253199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 msgr2=0x7f4ff00acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:16.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 --2- 172.21.15.71:0/1096253199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff00acdb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00097a0 tx=0x7f4fe002db50 comp rx=0 tx=0).stop 2023-08-25T10:02:16.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 -- 172.21.15.71:0/1096253199 shutdown_connections 2023-08-25T10:02:16.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 --2- 172.21.15.71:0/1096253199 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00c76d0 0x7f4ff0152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 --2- 172.21.15.71:0/1096253199 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff00acdb0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 -- 172.21.15.71:0/1096253199 >> 172.21.15.71:0/1096253199 conn(0x7f4ff006c280 msgr2=0x7f4ff006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:16.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 -- 172.21.15.71:0/1096253199 shutdown_connections 2023-08-25T10:02:16.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.307+0000 7f4fef59e640 1 -- 172.21.15.71:0/1096253199 wait complete. 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.309+0000 7f4fef59e640 1 Processor -- start 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.309+0000 7f4fef59e640 1 -- start start 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.310+0000 7f4fef59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00ad2f0 0x7f4ff014f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.310+0000 7f4fef59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff01ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.310+0000 7f4fef59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ff00c76d0 0x7f4ff0155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.310+0000 7f4fef59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01de570 con 0x7f4ff00c72f0 2023-08-25T10:02:16.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.310+0000 7f4fef59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01dd890 con 0x7f4ff00c76d0 2023-08-25T10:02:16.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.310+0000 7f4fef59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff01de7a0 con 0x7f4ff00ad2f0 2023-08-25T10:02:16.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff01ddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:16.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff01ddd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48902/0 (socket says 172.21.15.71:48902) 2023-08-25T10:02:16.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 -- 172.21.15.71:0/3945535157 learned_addr learned my addr 172.21.15.71:0/3945535157 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:16.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fee59c640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00ad2f0 0x7f4ff014f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:16.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4feed9d640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ff00c76d0 0x7f4ff0155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:16.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 -- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00ad2f0 msgr2=0x7f4ff014f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:16.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00ad2f0 0x7f4ff014f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 -- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ff00c76d0 msgr2=0x7f4ff0155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:16.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ff00c76d0 0x7f4ff0155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 -- 172.21.15.71:0/3945535157 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0009400 con 0x7f4ff00c72f0 2023-08-25T10:02:16.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4feed9d640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ff00c76d0 0x7f4ff0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:16.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.311+0000 7f4fedd9b640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff01ddd20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400d5f0 tx=0x7f4fe400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:16.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.312+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40125f0 con 0x7f4ff00c72f0 2023-08-25T10:02:16.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.312+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4012750 con 0x7f4ff00c72f0 2023-08-25T10:02:16.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.312+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4012b00 con 0x7f4ff00c72f0 2023-08-25T10:02:16.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.312+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff01dea80 con 0x7f4ff00c72f0 2023-08-25T10:02:16.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.312+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff01dd520 con 0x7f4ff00c72f0 2023-08-25T10:02:16.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.313+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff01dd6e0 con 0x7f4ff00c72f0 2023-08-25T10:02:16.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.314+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4fe4012c60 con 0x7f4ff00c72f0 2023-08-25T10:02:16.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.315+0000 7f4fdeffd640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fd006a080 0x7f4fd006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:16.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.315+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4fe4088ee0 con 0x7f4ff00c72f0 2023-08-25T10:02:16.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.316+0000 7f4fee59c640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fd006a080 0x7f4fd006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:16.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.316+0000 7f4fee59c640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fd006a080 0x7f4fd006c540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0038000 tx=0x7f4fe0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:16.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.320+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4fe4054100 con 0x7f4ff00c72f0 2023-08-25T10:02:16.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.592+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ff01dda00 con 0x7f4ff00c72f0 2023-08-25T10:02:16.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.593+0000 7f4fdeffd640 1 -- 172.21.15.71:0/3945535157 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4fe4088070 con 0x7f4ff00c72f0 2023-08-25T10:02:16.595 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:16.596 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:16.596 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:16.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fd006a080 msgr2=0x7f4fd006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:16.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fd006a080 0x7f4fd006c540 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0038000 tx=0x7f4fe0008030 comp rx=0 tx=0).stop 2023-08-25T10:02:16.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 msgr2=0x7f4ff01ddd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:16.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff01ddd20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400d5f0 tx=0x7f4fe400dab0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 shutdown_connections 2023-08-25T10:02:16.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ff00c76d0 0x7f4ff0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4fd006a080 0x7f4fd006c540 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ff00c72f0 0x7f4ff01ddd20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 --2- 172.21.15.71:0/3945535157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ff00ad2f0 0x7f4ff014f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:16.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 >> 172.21.15.71:0/3945535157 conn(0x7f4ff006c280 msgr2=0x7f4ff014dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:16.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 shutdown_connections 2023-08-25T10:02:16.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:16.596+0000 7f4fef59e640 1 -- 172.21.15.71:0/3945535157 wait complete. 2023-08-25T10:02:16.617 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:18.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:18.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.842+0000 7f77c0c46640 1 Processor -- start 2023-08-25T10:02:18.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.842+0000 7f77c0c46640 1 -- start start 2023-08-25T10:02:18.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77c0c46640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f77bc152d80 0x7f77bc0c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:18.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77c0c46640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:18.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77c0c46640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77bc1591c0 con 0x7f77bc0cacb0 2023-08-25T10:02:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77c0c46640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77bc1594a0 con 0x7f77bc152d80 2023-08-25T10:02:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77c0c46640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc159780 con 0x7f77bc15ce50 2023-08-25T10:02:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77ba59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f77bc152d80 0x7f77bc0c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77ba59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f77bc152d80 0x7f77bc0c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48908/0 (socket says 172.21.15.71:48908) 2023-08-25T10:02:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77ba59c640 1 -- 172.21.15.71:0/3653536683 learned_addr learned my addr 172.21.15.71:0/3653536683 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.844+0000 7f77b9d9b640 1 --1- 172.21.15.71:0/3653536683 >> v1:172.21.15.71:6789/0 conn(0x7f77bc0cacb0 0x7f77bc15c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47410/0 (socket says 172.21.15.71:47410) 2023-08-25T10:02:18.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.845+0000 7f77bad9d640 1 --2- 172.21.15.71:0/3653536683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:18.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.845+0000 7f77ba59c640 1 -- 172.21.15.71:0/3653536683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 msgr2=0x7f77bc0c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:18.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.845+0000 7f77ba59c640 1 --2- 172.21.15.71:0/3653536683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc0c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:18.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.845+0000 7f77ba59c640 1 -- 172.21.15.71:0/3653536683 >> v1:172.21.15.71:6789/0 conn(0x7f77bc0cacb0 legacy=0x7f77bc15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:18.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.845+0000 7f77ba59c640 1 -- 172.21.15.71:0/3653536683 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc159a60 con 0x7f77bc152d80 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.845+0000 7f77b959a640 1 -- 172.21.15.71:0/3653536683 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359253221 0 0) 0x7f77bc1591c0 con 0x7f77bc0cacb0 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.846+0000 7f77ba59c640 1 --2- 172.21.15.71:0/3653536683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc152d80 0x7f77bc0c9360 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009870 tx=0x7f77ac02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369459ed30e74270 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.846+0000 7f77b959a640 1 -- 172.21.15.71:0/3653536683 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac02f840 con 0x7f77bc152d80 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.846+0000 7f77b959a640 1 -- 172.21.15.71:0/3653536683 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac02f9a0 con 0x7f77bc152d80 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.846+0000 7f77b959a640 1 -- 172.21.15.71:0/3653536683 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac02fd30 con 0x7f77bc152d80 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 -- 172.21.15.71:0/3653536683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc152d80 msgr2=0x7f77bc0c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 --2- 172.21.15.71:0/3653536683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc152d80 0x7f77bc0c9360 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009870 tx=0x7f77ac02ba20 comp rx=0 tx=0).stop 2023-08-25T10:02:18.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 -- 172.21.15.71:0/3653536683 shutdown_connections 2023-08-25T10:02:18.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 --2- 172.21.15.71:0/3653536683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:18.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 --2- 172.21.15.71:0/3653536683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc152d80 0x7f77bc0c9360 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:18.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 -- 172.21.15.71:0/3653536683 >> 172.21.15.71:0/3653536683 conn(0x7f77bc06b8f0 msgr2=0x7f77bc0654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:18.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 -- 172.21.15.71:0/3653536683 shutdown_connections 2023-08-25T10:02:18.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.847+0000 7f77c0c46640 1 -- 172.21.15.71:0/3653536683 wait complete. 2023-08-25T10:02:18.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.849+0000 7f77c0c46640 1 Processor -- start 2023-08-25T10:02:18.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.849+0000 7f77c0c46640 1 -- start start 2023-08-25T10:02:18.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77c0c46640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 0x7f77bc15b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:18.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77c0c46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77bc152d80 0x7f77bc1eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:18.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77c0c46640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc1eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:18.854 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77c0c46640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1f1fb0 con 0x7f77bc0cacb0 2023-08-25T10:02:18.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77c0c46640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1f2290 con 0x7f77bc152d80 2023-08-25T10:02:18.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77c0c46640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1f2570 con 0x7f77bc15ce50 2023-08-25T10:02:18.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.850+0000 7f77ba59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 0x7f77bc15b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:18.855 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 0x7f77bc15b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48916/0 (socket says 172.21.15.71:48916) 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 -- 172.21.15.71:0/406576237 learned_addr learned my addr 172.21.15.71:0/406576237 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77bad9d640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc1eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77b9d9b640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77bc152d80 0x7f77bc1eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 -- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 msgr2=0x7f77bc1eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc1eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 -- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77bc152d80 msgr2=0x7f77bc1eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:18.856 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77bc152d80 0x7f77bc1eeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:18.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 -- 172.21.15.71:0/406576237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0094d0 con 0x7f77bc0cacb0 2023-08-25T10:02:18.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77bad9d640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc1eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:18.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77b9d9b640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77bc152d80 0x7f77bc1eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:18.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.851+0000 7f77ba59c640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 0x7f77bc15b4e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f77ac000c00 tx=0x7f77ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:18.857 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.852+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac037c40 con 0x7f77bc0cacb0 2023-08-25T10:02:18.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.852+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac037da0 con 0x7f77bc0cacb0 2023-08-25T10:02:18.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.852+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac041640 con 0x7f77bc0cacb0 2023-08-25T10:02:18.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.852+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77bc1f2850 con 0x7f77bc0cacb0 2023-08-25T10:02:18.858 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.852+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77bc1f1c80 con 0x7f77bc0cacb0 2023-08-25T10:02:18.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.853+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77bc0b1a60 con 0x7f77bc0cacb0 2023-08-25T10:02:18.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.854+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f77ac0417a0 con 0x7f77bc0cacb0 2023-08-25T10:02:18.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.856+0000 7f77aaffd640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f779c06a050 0x7f779c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:18.859 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.856+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f77ac03c070 con 0x7f77bc0cacb0 2023-08-25T10:02:18.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.856+0000 7f77b9d9b640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f779c06a050 0x7f779c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:18.860 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.857+0000 7f77b9d9b640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f779c06a050 0x7f779c06c510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f77bc150530 tx=0x7f77b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:18.863 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:18.862+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f77ac076c50 con 0x7f77bc0cacb0 2023-08-25T10:02:19.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.140+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77bc069160 con 0x7f77bc0cacb0 2023-08-25T10:02:19.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.141+0000 7f77aaffd640 1 -- 172.21.15.71:0/406576237 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f77ac07a270 con 0x7f77bc0cacb0 2023-08-25T10:02:19.143 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:19.143 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:19.144 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:19.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f779c06a050 msgr2=0x7f779c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:19.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f779c06a050 0x7f779c06c510 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f77bc150530 tx=0x7f77b0009040 comp rx=0 tx=0).stop 2023-08-25T10:02:19.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 msgr2=0x7f77bc15b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:19.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 0x7f77bc15b4e0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f77ac000c00 tx=0x7f77ac004730 comp rx=0 tx=0).stop 2023-08-25T10:02:19.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 shutdown_connections 2023-08-25T10:02:19.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77bc152d80 0x7f77bc1eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:19.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f779c06a050 0x7f779c06c510 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:19.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77bc15ce50 0x7f77bc1eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:19.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 --2- 172.21.15.71:0/406576237 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f77bc0cacb0 0x7f77bc15b4e0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:19.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 >> 172.21.15.71:0/406576237 conn(0x7f77bc06b8f0 msgr2=0x7f77bc0cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:19.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 shutdown_connections 2023-08-25T10:02:19.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:19.144+0000 7f77c0c46640 1 -- 172.21.15.71:0/406576237 wait complete. 2023-08-25T10:02:19.165 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:02:19.166 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:02:21.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:21.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.387+0000 7f84d1821640 1 Processor -- start 2023-08-25T10:02:21.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.387+0000 7f84d1821640 1 -- start start 2023-08-25T10:02:21.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.388+0000 7f84d1821640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84cc0cb570 0x7f84cc0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.388+0000 7f84d1821640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc15cd00 0x7f84cc158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.388+0000 7f84d1821640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84cc0b1af0 con 0x7f84cc0cb190 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.388+0000 7f84d1821640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84cc0c98f0 con 0x7f84cc0cb570 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.388+0000 7f84d1821640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc15d170 con 0x7f84cc15cd00 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84cc0cb570 0x7f84cc0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84cc0cb570 0x7f84cc0ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42248/0 (socket says 172.21.15.71:42248) 2023-08-25T10:02:21.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 -- 172.21.15.71:0/2261269801 learned_addr learned my addr 172.21.15.71:0/2261269801 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:21.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84d1020640 1 --2- 172.21.15.71:0/2261269801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc15cd00 0x7f84cc158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:21.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 -- 172.21.15.71:0/2261269801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc15cd00 msgr2=0x7f84cc158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:21.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 --2- 172.21.15.71:0/2261269801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc15cd00 0x7f84cc158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 -- 172.21.15.71:0/2261269801 >> v1:172.21.15.164:6789/0 conn(0x7f84cc0cb190 legacy=0x7f84cc0c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:21.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.389+0000 7f84cbfff640 1 -- 172.21.15.71:0/2261269801 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc0b0fb0 con 0x7f84cc0cb570 2023-08-25T10:02:21.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.391+0000 7f84cbfff640 1 --2- 172.21.15.71:0/2261269801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc0ca910 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009870 tx=0x7f84b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d392651712290c0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:21.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.391+0000 7f84cb7fe640 1 -- 172.21.15.71:0/2261269801 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8030840 con 0x7f84cc0cb570 2023-08-25T10:02:21.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.391+0000 7f84cb7fe640 1 -- 172.21.15.71:0/2261269801 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b80309a0 con 0x7f84cc0cb570 2023-08-25T10:02:21.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.391+0000 7f84cb7fe640 1 -- 172.21.15.71:0/2261269801 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8030d30 con 0x7f84cc0cb570 2023-08-25T10:02:21.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 -- 172.21.15.71:0/2261269801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 msgr2=0x7f84cc0ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:21.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 --2- 172.21.15.71:0/2261269801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc0ca910 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009870 tx=0x7f84b802b980 comp rx=0 tx=0).stop 2023-08-25T10:02:21.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 -- 172.21.15.71:0/2261269801 shutdown_connections 2023-08-25T10:02:21.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 --2- 172.21.15.71:0/2261269801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc15cd00 0x7f84cc158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 --2- 172.21.15.71:0/2261269801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc0ca910 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 -- 172.21.15.71:0/2261269801 >> 172.21.15.71:0/2261269801 conn(0x7f84cc06ae20 msgr2=0x7f84cc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:21.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 -- 172.21.15.71:0/2261269801 shutdown_connections 2023-08-25T10:02:21.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.392+0000 7f84d1821640 1 -- 172.21.15.71:0/2261269801 wait complete. 2023-08-25T10:02:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.394+0000 7f84d1821640 1 Processor -- start 2023-08-25T10:02:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.395+0000 7f84d1821640 1 -- start start 2023-08-25T10:02:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1821640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc0cb190 0x7f84cc14dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1821640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:21.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1821640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84cc15cd00 0x7f84cc1549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1821640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc14dbd0 con 0x7f84cc0cb570 2023-08-25T10:02:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1821640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc156230 con 0x7f84cc15cd00 2023-08-25T10:02:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1821640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c5140 con 0x7f84cc0cb190 2023-08-25T10:02:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84cbfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42262/0 (socket says 172.21.15.71:42262) 2023-08-25T10:02:21.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84cbfff640 1 -- 172.21.15.71:0/4243490306 learned_addr learned my addr 172.21.15.71:0/4243490306 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d081f640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc0cb190 0x7f84cc14dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.396+0000 7f84d1020640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84cc15cd00 0x7f84cc1549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84cbfff640 1 -- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc0cb190 msgr2=0x7f84cc14dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84cbfff640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc0cb190 0x7f84cc14dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84cbfff640 1 -- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84cc15cd00 msgr2=0x7f84cc1549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84cbfff640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84cc15cd00 0x7f84cc1549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84cbfff640 1 -- 172.21.15.71:0/4243490306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b80094d0 con 0x7f84cc0cb570 2023-08-25T10:02:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84d081f640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc0cb190 0x7f84cc14dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84d1020640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84cc15cd00 0x7f84cc1549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84cbfff640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc155cf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009870 tx=0x7f84b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8039c90 con 0x7f84cc0cb570 2023-08-25T10:02:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b8039df0 con 0x7f84cc0cb570 2023-08-25T10:02:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8043620 con 0x7f84cc0cb570 2023-08-25T10:02:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.397+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84cc0c5370 con 0x7f84cc0cb570 2023-08-25T10:02:21.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.398+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc0ce710 con 0x7f84cc0cb570 2023-08-25T10:02:21.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.399+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc04e420 con 0x7f84cc0cb570 2023-08-25T10:02:21.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.400+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f84b8043780 con 0x7f84cc0cb570 2023-08-25T10:02:21.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.401+0000 7f84c97fa640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84b006a080 0x7f84b006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:21.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.401+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f84b803e070 con 0x7f84cc0cb570 2023-08-25T10:02:21.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.401+0000 7f84d081f640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84b006a080 0x7f84b006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:21.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.402+0000 7f84d081f640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84b006a080 0x7f84b006c540 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f84c0003eb0 tx=0x7f84c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:21.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.406+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f84b8076d20 con 0x7f84cc0cb570 2023-08-25T10:02:21.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.672+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84cc0c6110 con 0x7f84cc0cb570 2023-08-25T10:02:21.675 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.673+0000 7f84c97fa640 1 -- 172.21.15.71:0/4243490306 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f84b8034090 con 0x7f84cc0cb570 2023-08-25T10:02:21.675 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:21.676 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:21.676 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:21.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84b006a080 msgr2=0x7f84b006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:21.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84b006a080 0x7f84b006c540 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f84c0003eb0 tx=0x7f84c0009040 comp rx=0 tx=0).stop 2023-08-25T10:02:21.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 msgr2=0x7f84cc155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:21.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc155cf0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009870 tx=0x7f84b8004730 comp rx=0 tx=0).stop 2023-08-25T10:02:21.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 shutdown_connections 2023-08-25T10:02:21.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84cc15cd00 0x7f84cc1549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84b006a080 0x7f84b006c540 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84cc0cb570 0x7f84cc155cf0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 --2- 172.21.15.71:0/4243490306 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84cc0cb190 0x7f84cc14dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:21.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 >> 172.21.15.71:0/4243490306 conn(0x7f84cc06ae20 msgr2=0x7f84cc0c6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:21.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 shutdown_connections 2023-08-25T10:02:21.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:21.677+0000 7f84d1821640 1 -- 172.21.15.71:0/4243490306 wait complete. 2023-08-25T10:02:21.699 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:23.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:23.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.925+0000 7f4eaf59e640 1 Processor -- start 2023-08-25T10:02:23.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.925+0000 7f4eaf59e640 1 -- start start 2023-08-25T10:02:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.927+0000 7f4eaf59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb00c72f0 0x7f4eb00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.927+0000 7f4eaf59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4eb00ad2f0 0x7f4eb0155c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.927+0000 7f4eaf59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00c1060 con 0x7f4eb00c76d0 2023-08-25T10:02:23.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4eaf59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb00adf70 con 0x7f4eb00ad2f0 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4eaf59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00af730 con 0x7f4eb00c72f0 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4eadd9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4eb00c76d0 0x7f4eb0152da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60176/0 (socket says 172.21.15.71:60176) 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4eadd9b640 1 -- 172.21.15.71:0/1568128364 learned_addr learned my addr 172.21.15.71:0/1568128364 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4eaed9d640 1 --2- 172.21.15.71:0/1568128364 >> v2:172.21.15.164:3300/0 conn(0x7f4eb00ad2f0 0x7f4eb0155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4eae59c640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb00c72f0 0x7f4eb00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082149143 0 0) 0x7f4eb00c1060 con 0x7f4eb00c76d0 2023-08-25T10:02:23.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.928+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98003520 con 0x7f4eb00c76d0 2023-08-25T10:02:23.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1942002757 0 0) 0x7f4e98003520 con 0x7f4eb00c76d0 2023-08-25T10:02:23.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb00c72f0 msgr2=0x7f4eb00acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:23.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb00c72f0 0x7f4eb00acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 >> v2:172.21.15.164:3300/0 conn(0x7f4eb00ad2f0 msgr2=0x7f4eb0155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 --2- 172.21.15.71:0/1568128364 >> v2:172.21.15.164:3300/0 conn(0x7f4eb00ad2f0 0x7f4eb0155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb006b4a0 con 0x7f4eb00c76d0 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4ea4001670 con 0x7f4eb00c76d0 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 >> v1:172.21.15.71:6789/0 conn(0x7f4eb00c76d0 legacy=0x7f4eb0152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e980039e0 0x7f4e98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980047c0 0x7f4e98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e98005180 0x7f4e9800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00adf70 con 0x7f4e98005180 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00af730 con 0x7f4e980047c0 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9800e180 con 0x7f4e980039e0 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.929+0000 7f4eae59c640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4eb00c72f0 0x7f4eb00acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eaed9d640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e98005180 0x7f4e9800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eadd9b640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980047c0 0x7f4e98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eae59c640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e980039e0 0x7f4e98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eaed9d640 1 -- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e980039e0 msgr2=0x7f4e98003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:23.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eaed9d640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e980039e0 0x7f4e98003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eaed9d640 1 -- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980047c0 msgr2=0x7f4e98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:23.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eaed9d640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980047c0 0x7f4e98004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.930+0000 7f4eaed9d640 1 -- 172.21.15.71:0/1568128364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98003520 con 0x7f4e98005180 2023-08-25T10:02:23.936 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.931+0000 7f4eaed9d640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e98005180 0x7f4e9800db90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009f90 tx=0x7f4e980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a80ffc2ac5a01e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:23.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea802cc20 con 0x7f4e98005180 2023-08-25T10:02:23.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea800f040 con 0x7f4e98005180 2023-08-25T10:02:23.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4ead59a640 1 -- 172.21.15.71:0/1568128364 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8034440 con 0x7f4e98005180 2023-08-25T10:02:23.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4eaf59e640 1 -- 172.21.15.71:0/1568128364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e98005180 msgr2=0x7f4e9800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:23.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/1568128364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e98005180 0x7f4e9800db90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009f90 tx=0x7f4e980057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4eaed9d640 1 -- 172.21.15.71:0/1568128364 reap_dead start 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4eaf59e640 1 -- 172.21.15.71:0/1568128364 shutdown_connections 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.932+0000 7f4eaf59e640 1 -- 172.21.15.71:0/1568128364 >> 172.21.15.71:0/1568128364 conn(0x7f4eb006c280 msgr2=0x7f4eb006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.933+0000 7f4eaf59e640 1 -- 172.21.15.71:0/1568128364 shutdown_connections 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.933+0000 7f4eaf59e640 1 -- 172.21.15.71:0/1568128364 wait complete. 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.935+0000 7f4eaf59e640 1 Processor -- start 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.935+0000 7f4eaf59e640 1 -- start start 2023-08-25T10:02:23.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaf59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e98005180 0x7f4eb0154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaf59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 0x7f4eb01e35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaf59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb0150da0 0x7f4eb014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaf59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01e1a90 con 0x7f4eb01e1630 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaf59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01e1fc0 con 0x7f4eb0150da0 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaf59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01eeec0 con 0x7f4e98005180 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eadd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 0x7f4eb01e35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eadd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 0x7f4eb01e35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42274/0 (socket says 172.21.15.71:42274) 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eadd9b640 1 -- 172.21.15.71:0/2554986746 learned_addr learned my addr 172.21.15.71:0/2554986746 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eae59c640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e98005180 0x7f4eb0154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.936+0000 7f4eaed9d640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb0150da0 0x7f4eb014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eadd9b640 1 -- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e98005180 msgr2=0x7f4eb0154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eadd9b640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e98005180 0x7f4eb0154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eadd9b640 1 -- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb0150da0 msgr2=0x7f4eb014e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eadd9b640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb0150da0 0x7f4eb014e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:23.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eadd9b640 1 -- 172.21.15.71:0/2554986746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb01e1ce0 con 0x7f4eb01e1630 2023-08-25T10:02:23.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eae59c640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e98005180 0x7f4eb0154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:23.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eadd9b640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 0x7f4eb01e35c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400e7b0 tx=0x7f4ea400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:23.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400c8d0 con 0x7f4eb01e1630 2023-08-25T10:02:23.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea400ca50 con 0x7f4eb01e1630 2023-08-25T10:02:23.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb01ee100 con 0x7f4eb01e1630 2023-08-25T10:02:23.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.937+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400ce00 con 0x7f4eb01e1630 2023-08-25T10:02:23.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.938+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb01ee6c0 con 0x7f4eb01e1630 2023-08-25T10:02:23.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.939+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb004e420 con 0x7f4eb01e1630 2023-08-25T10:02:23.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.940+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4ea401d460 con 0x7f4eb01e1630 2023-08-25T10:02:23.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.941+0000 7f4e9effd640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e9006a050 0x7f4e9006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:23.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.941+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4ea4088a90 con 0x7f4eb01e1630 2023-08-25T10:02:23.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.942+0000 7f4eae59c640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e9006a050 0x7f4e9006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:23.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.942+0000 7f4eae59c640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e9006a050 0x7f4e9006c510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0006fd0 tx=0x7f4ea0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:23.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:23.947+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4ea4053cc0 con 0x7f4eb01e1630 2023-08-25T10:02:24.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.203+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eb006b1e0 con 0x7f4eb01e1630 2023-08-25T10:02:24.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.204+0000 7f4e9effd640 1 -- 172.21.15.71:0/2554986746 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4ea40572e0 con 0x7f4eb01e1630 2023-08-25T10:02:24.207 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:24.207 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:24.207 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:24.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.206+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e9006a050 msgr2=0x7f4e9006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:24.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.206+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e9006a050 0x7f4e9006c510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0006fd0 tx=0x7f4ea0008040 comp rx=0 tx=0).stop 2023-08-25T10:02:24.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.206+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 msgr2=0x7f4eb01e35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:24.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.206+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 0x7f4eb01e35c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400e7b0 tx=0x7f4ea400ec70 comp rx=0 tx=0).stop 2023-08-25T10:02:24.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 shutdown_connections 2023-08-25T10:02:24.209 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4eb0150da0 0x7f4eb014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:24.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e9006a050 0x7f4e9006c510 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:24.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4eb01e1630 0x7f4eb01e35c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:24.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 --2- 172.21.15.71:0/2554986746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e98005180 0x7f4eb0154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:24.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 >> 172.21.15.71:0/2554986746 conn(0x7f4eb006c280 msgr2=0x7f4eb0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:24.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 shutdown_connections 2023-08-25T10:02:24.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:24.207+0000 7f4eaf59e640 1 -- 172.21.15.71:0/2554986746 wait complete. 2023-08-25T10:02:24.230 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:26.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:02:26.084 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:02:26.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:26.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.441+0000 7f9cb64c2640 1 Processor -- start 2023-08-25T10:02:26.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.441+0000 7f9cb64c2640 1 -- start start 2023-08-25T10:02:26.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb64c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb006d3b0 0x7f9cb00c7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb64c2640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9cb006d790 0x7f9cb00b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb64c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9cb00c60b0 0x7f9cb00c6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb64c2640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb0159c90 con 0x7f9cb006d790 2023-08-25T10:02:26.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb64c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb00aee20 con 0x7f9cb00c60b0 2023-08-25T10:02:26.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb64c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb0158c60 con 0x7f9cb006d3b0 2023-08-25T10:02:26.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb4cbf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9cb006d790 0x7f9cb00b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb4cbf640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9cb006d790 0x7f9cb00b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42288/0 (socket says 172.21.15.71:42288) 2023-08-25T10:02:26.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.443+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/2540025735 learned_addr learned my addr 172.21.15.71:0/2540025735 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:26.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb5cc1640 1 --2- 172.21.15.71:0/2540025735 >> v2:172.21.15.164:3300/0 conn(0x7f9cb00c60b0 0x7f9cb00c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb54c0640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb006d3b0 0x7f9cb00c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/2540025735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb006d3b0 msgr2=0x7f9cb00c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:26.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb4cbf640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb006d3b0 0x7f9cb00c7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/2540025735 >> v2:172.21.15.164:3300/0 conn(0x7f9cb00c60b0 msgr2=0x7f9cb00c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb4cbf640 1 --2- 172.21.15.71:0/2540025735 >> v2:172.21.15.164:3300/0 conn(0x7f9cb00c60b0 0x7f9cb00c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/2540025735 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb00c7b70 con 0x7f9cb006d790 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb5cc1640 1 --2- 172.21.15.71:0/2540025735 >> v2:172.21.15.164:3300/0 conn(0x7f9cb00c60b0 0x7f9cb00c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.444+0000 7f9cb54c0640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb006d3b0 0x7f9cb00c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.445+0000 7f9cb4cbf640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb00b02d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9cb00c0c60 tx=0x7f9ca002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be86592ad279980d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:26.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.445+0000 7f9ca7fff640 1 -- 172.21.15.71:0/2540025735 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0030850 con 0x7f9cb006d790 2023-08-25T10:02:26.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9ca7fff640 1 -- 172.21.15.71:0/2540025735 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca00309b0 con 0x7f9cb006d790 2023-08-25T10:02:26.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9ca7fff640 1 -- 172.21.15.71:0/2540025735 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0030d40 con 0x7f9cb006d790 2023-08-25T10:02:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 -- 172.21.15.71:0/2540025735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 msgr2=0x7f9cb00b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb00b02d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9cb00c0c60 tx=0x7f9ca002b990 comp rx=0 tx=0).stop 2023-08-25T10:02:26.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 -- 172.21.15.71:0/2540025735 shutdown_connections 2023-08-25T10:02:26.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/2540025735 >> v2:172.21.15.164:3300/0 conn(0x7f9cb00c60b0 0x7f9cb00c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb00b02d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/2540025735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb006d3b0 0x7f9cb00c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 -- 172.21.15.71:0/2540025735 >> 172.21.15.71:0/2540025735 conn(0x7f9cb00cb4b0 msgr2=0x7f9cb00cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:26.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.446+0000 7f9cb64c2640 1 -- 172.21.15.71:0/2540025735 shutdown_connections 2023-08-25T10:02:26.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.447+0000 7f9cb64c2640 1 -- 172.21.15.71:0/2540025735 wait complete. 2023-08-25T10:02:26.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.448+0000 7f9cb64c2640 1 Processor -- start 2023-08-25T10:02:26.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.449+0000 7f9cb64c2640 1 -- start start 2023-08-25T10:02:26.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb64c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb006d3b0 0x7f9cb0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb64c2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb006c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb64c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb00c60b0 0x7f9cb006c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb64c2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb00c55d0 con 0x7f9cb006d790 2023-08-25T10:02:26.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb64c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb0156180 con 0x7f9cb006d3b0 2023-08-25T10:02:26.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb64c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb00c3100 con 0x7f9cb00c60b0 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb4cbf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb006c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb4cbf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb006c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42296/0 (socket says 172.21.15.71:42296) 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/3412595973 learned_addr learned my addr 172.21.15.71:0/3412595973 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.450+0000 7f9cb54c0640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb006d3b0 0x7f9cb0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb5cc1640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb00c60b0 0x7f9cb006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb00c60b0 msgr2=0x7f9cb006c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb4cbf640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb00c60b0 0x7f9cb006c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb006d3b0 msgr2=0x7f9cb0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb4cbf640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb006d3b0 0x7f9cb0071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb4cbf640 1 -- 172.21.15.71:0/3412595973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca00094d0 con 0x7f9cb006d790 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb5cc1640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb00c60b0 0x7f9cb006c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb4cbf640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb006c1d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0000c00 tx=0x7f9ca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0030820 con 0x7f9cb006d790 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca0043400 con 0x7f9cb006d790 2023-08-25T10:02:26.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00437d0 con 0x7f9cb006d790 2023-08-25T10:02:26.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.451+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb00ae330 con 0x7f9cb006d790 2023-08-25T10:02:26.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.452+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb01eeaf0 con 0x7f9cb006d790 2023-08-25T10:02:26.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.453+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb004e420 con 0x7f9cb006d790 2023-08-25T10:02:26.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.454+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9ca002e070 con 0x7f9cb006d790 2023-08-25T10:02:26.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.455+0000 7f9ca5ffb640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c9406a050 0x7f9c9406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:26.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.455+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9ca003e070 con 0x7f9cb006d790 2023-08-25T10:02:26.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.455+0000 7f9cb54c0640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c9406a050 0x7f9c9406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:26.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.456+0000 7f9cb54c0640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c9406a050 0x7f9c9406c510 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00a950 tx=0x7f9cac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:26.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.461+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9ca0076cc0 con 0x7f9cb006d790 2023-08-25T10:02:26.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.705+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb00c0780 con 0x7f9cb006d790 2023-08-25T10:02:26.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.706+0000 7f9ca5ffb640 1 -- 172.21.15.71:0/3412595973 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f9ca0034020 con 0x7f9cb006d790 2023-08-25T10:02:26.708 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:26.709 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:26.709 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:26.711 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c9406a050 msgr2=0x7f9c9406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:26.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c9406a050 0x7f9c9406c510 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00a950 tx=0x7f9cac008040 comp rx=0 tx=0).stop 2023-08-25T10:02:26.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 msgr2=0x7f9cb006c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:26.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb006c1d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0000c00 tx=0x7f9ca0004730 comp rx=0 tx=0).stop 2023-08-25T10:02:26.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 shutdown_connections 2023-08-25T10:02:26.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9c9406a050 0x7f9c9406c510 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9cb00c60b0 0x7f9cb006c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9cb006d790 0x7f9cb006c1d0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 --2- 172.21.15.71:0/3412595973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cb006d3b0 0x7f9cb0071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:26.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.709+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 >> 172.21.15.71:0/3412595973 conn(0x7f9cb00cb4b0 msgr2=0x7f9cb00ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:26.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.710+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 shutdown_connections 2023-08-25T10:02:26.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:26.710+0000 7f9cb64c2640 1 -- 172.21.15.71:0/3412595973 wait complete. 2023-08-25T10:02:26.734 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:28.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:28.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.962+0000 7fead4d01640 1 Processor -- start 2023-08-25T10:02:28.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.962+0000 7fead4d01640 1 -- start start 2023-08-25T10:02:28.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7fead4d01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 0x7fead00cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7fead4d01640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fead00c3e00 0x7fead00c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7fead4d01640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead00c47b0 con 0x7fead0153310 2023-08-25T10:02:28.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7fead4d01640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead01590c0 con 0x7fead00c3e00 2023-08-25T10:02:28.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7fead4d01640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead0159380 con 0x7fead00cec00 2023-08-25T10:02:28.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7feace59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fead0153310 0x7fead006a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60188/0 (socket says 172.21.15.71:60188) 2023-08-25T10:02:28.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.964+0000 7feace59c640 1 -- 172.21.15.71:0/1247588949 learned_addr learned my addr 172.21.15.71:0/1247588949 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:28.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacdd9b640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 0x7fead00cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feaced9d640 1 --2- 172.21.15.71:0/1247588949 >> v2:172.21.15.164:3300/0 conn(0x7fead00c3e00 0x7fead00c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846676599 0 0) 0x7fead00c47b0 con 0x7fead0153310 2023-08-25T10:02:28.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab8003520 con 0x7fead0153310 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 444923223 0 0) 0x7feab8003520 con 0x7fead0153310 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 msgr2=0x7fead00cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 0x7fead00cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 >> v2:172.21.15.164:3300/0 conn(0x7fead00c3e00 msgr2=0x7fead00c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 --2- 172.21.15.71:0/1247588949 >> v2:172.21.15.164:3300/0 conn(0x7fead00c3e00 0x7fead00c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0159660 con 0x7fead0153310 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7feac0001670 con 0x7fead0153310 2023-08-25T10:02:28.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 >> v1:172.21.15.71:6789/0 conn(0x7fead0153310 legacy=0x7fead006a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 0x7feab80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8003f60 0x7feab80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab8004d00 0x7feab8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead01590c0 con 0x7feab8004d00 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead0159380 con 0x7feab8003f60 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.965+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab800ba30 con 0x7fead00cec00 2023-08-25T10:02:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feaced9d640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab8004d00 0x7feab8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feace59c640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 0x7feab80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feacdd9b640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8003f60 0x7feab80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feaced9d640 1 -- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 msgr2=0x7feab80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feaced9d640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead00cec00 0x7feab80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feaced9d640 1 -- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8003f60 msgr2=0x7feab80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feaced9d640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8003f60 0x7feab80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.966+0000 7feaced9d640 1 -- 172.21.15.71:0/1247588949 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8003520 con 0x7feab8004d00 2023-08-25T10:02:28.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.968+0000 7feaced9d640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab8004d00 0x7feab8005180 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feac8009fc0 tx=0x7feab80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab06fc6c034ea9e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:28.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.968+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac802cc60 con 0x7feab8004d00 2023-08-25T10:02:28.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.968+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac800f040 con 0x7feab8004d00 2023-08-25T10:02:28.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.968+0000 7feacd59a640 1 -- 172.21.15.71:0/1247588949 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac8034440 con 0x7feab8004d00 2023-08-25T10:02:28.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7fead4d01640 1 -- 172.21.15.71:0/1247588949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab8004d00 msgr2=0x7feab8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:28.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7fead4d01640 1 --2- 172.21.15.71:0/1247588949 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feab8004d00 0x7feab8005180 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7feac8009fc0 tx=0x7feab80057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7feaced9d640 1 -- 172.21.15.71:0/1247588949 reap_dead start 2023-08-25T10:02:28.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7fead4d01640 1 -- 172.21.15.71:0/1247588949 shutdown_connections 2023-08-25T10:02:28.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7fead4d01640 1 -- 172.21.15.71:0/1247588949 >> 172.21.15.71:0/1247588949 conn(0x7fead0065910 msgr2=0x7fead0065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:28.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7fead4d01640 1 -- 172.21.15.71:0/1247588949 shutdown_connections 2023-08-25T10:02:28.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.969+0000 7fead4d01640 1 -- 172.21.15.71:0/1247588949 wait complete. 2023-08-25T10:02:28.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.971+0000 7fead4d01640 1 Processor -- start 2023-08-25T10:02:28.974 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.971+0000 7fead4d01640 1 -- start start 2023-08-25T10:02:28.975 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7fead4d01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8004d00 0x7fead01eea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7fead4d01640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 0x7fead01eda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7fead4d01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead01eef80 0x7fead00c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7fead4d01640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead01ec2f0 con 0x7fead014c3c0 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7fead4d01640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead01f2820 con 0x7feab8004d00 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7fead4d01640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead0153910 con 0x7fead01eef80 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 0x7fead01eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.976 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 0x7fead01eda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:42314/0 (socket says 172.21.15.71:42314) 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 -- 172.21.15.71:0/568862485 learned_addr learned my addr 172.21.15.71:0/568862485 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feace59c640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8004d00 0x7fead01eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feaced9d640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead01eef80 0x7fead00c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 -- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead01eef80 msgr2=0x7fead00c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead01eef80 0x7fead00c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 -- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8004d00 msgr2=0x7fead01eea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:28.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8004d00 0x7fead01eea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feacdd9b640 1 -- 172.21.15.71:0/568862485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0149100 con 0x7fead014c3c0 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.972+0000 7feace59c640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8004d00 0x7fead01eea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.973+0000 7feacdd9b640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 0x7fead01eda30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7feac4011630 tx=0x7feac4011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.973+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400f920 con 0x7fead014c3c0 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.973+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac400fa80 con 0x7fead014c3c0 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.973+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400fdf0 con 0x7fead014c3c0 2023-08-25T10:02:28.978 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.973+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead01f1ca0 con 0x7fead014c3c0 2023-08-25T10:02:28.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.973+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead01f2260 con 0x7fead014c3c0 2023-08-25T10:02:28.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.975+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead00cd9e0 con 0x7fead014c3c0 2023-08-25T10:02:28.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.976+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7feac4015460 con 0x7fead014c3c0 2023-08-25T10:02:28.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.977+0000 7feabeffd640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feab006a050 0x7feab006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:28.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.977+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7feac4089010 con 0x7fead014c3c0 2023-08-25T10:02:28.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.977+0000 7feace59c640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feab006a050 0x7feab006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:28.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.978+0000 7feace59c640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feab006a050 0x7feab006c510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feac0001eb0 tx=0x7feac0001150 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:28.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:28.982+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7feac40541b0 con 0x7fead014c3c0 2023-08-25T10:02:29.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.245+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fead0069160 con 0x7fead014c3c0 2023-08-25T10:02:29.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.246+0000 7feabeffd640 1 -- 172.21.15.71:0/568862485 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7feac4022090 con 0x7fead014c3c0 2023-08-25T10:02:29.248 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:29.248 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:29.248 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:29.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feab006a050 msgr2=0x7feab006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:29.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feab006a050 0x7feab006c510 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feac0001eb0 tx=0x7feac0001150 comp rx=0 tx=0).stop 2023-08-25T10:02:29.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 msgr2=0x7fead01eda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 0x7fead01eda30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7feac4011630 tx=0x7feac4011af0 comp rx=0 tx=0).stop 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 shutdown_connections 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7feab006a050 0x7feab006c510 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead01eef80 0x7fead00c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fead014c3c0 0x7fead01eda30 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 --2- 172.21.15.71:0/568862485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab8004d00 0x7fead01eea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:29.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 >> 172.21.15.71:0/568862485 conn(0x7fead0065910 msgr2=0x7fead00ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:29.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.248+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 shutdown_connections 2023-08-25T10:02:29.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:29.249+0000 7fead4d01640 1 -- 172.21.15.71:0/568862485 wait complete. 2023-08-25T10:02:29.268 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:31.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:31.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.503+0000 7f0f0ad60640 1 Processor -- start 2023-08-25T10:02:31.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.503+0000 7f0f0ad60640 1 -- start start 2023-08-25T10:02:31.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.505+0000 7f0f0ad60640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0f040c56c0 0x7f0f041576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.505+0000 7f0f0ad60640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f04157bf0 con 0x7f0f041542d0 2023-08-25T10:02:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.505+0000 7f0f0ad60640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0406a130 con 0x7f0f04069850 2023-08-25T10:02:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.505+0000 7f0f0ad60640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f040c4b60 con 0x7f0f040c56c0 2023-08-25T10:02:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.506+0000 7f0f0a55f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0f040c56c0 0x7f0f041576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.506+0000 7f0f0a55f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0f040c56c0 0x7f0f041576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48412/0 (socket says 172.21.15.71:48412) 2023-08-25T10:02:31.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.506+0000 7f0f0a55f640 1 -- 172.21.15.71:0/583283175 learned_addr learned my addr 172.21.15.71:0/583283175 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:31.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.506+0000 7f0f0a55f640 1 -- 172.21.15.71:0/583283175 >> v1:172.21.15.164:6789/0 conn(0x7f0f04069850 legacy=0x7f0f040c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:31.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.506+0000 7f0f0a55f640 1 -- 172.21.15.71:0/583283175 >> v1:172.21.15.71:6789/0 conn(0x7f0f041542d0 legacy=0x7f0f040c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:31.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.506+0000 7f0f0a55f640 1 -- 172.21.15.71:0/583283175 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f041546b0 con 0x7f0f040c56c0 2023-08-25T10:02:31.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.507+0000 7f0f0a55f640 1 --2- 172.21.15.71:0/583283175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f040c56c0 0x7f0f041576b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009e50 tx=0x7f0efc02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be02170a7e67d8a0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:31.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.508+0000 7f0f08d5c640 1 -- 172.21.15.71:0/583283175 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc02cb10 con 0x7f0f040c56c0 2023-08-25T10:02:31.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.508+0000 7f0f08d5c640 1 -- 172.21.15.71:0/583283175 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0efc02cc70 con 0x7f0f040c56c0 2023-08-25T10:02:31.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.508+0000 7f0f08d5c640 1 -- 172.21.15.71:0/583283175 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc0343f0 con 0x7f0f040c56c0 2023-08-25T10:02:31.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.508+0000 7f0f0ad60640 1 -- 172.21.15.71:0/583283175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f040c56c0 msgr2=0x7f0f041576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:31.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.509+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/583283175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f040c56c0 0x7f0f041576b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009e50 tx=0x7f0efc02bf70 comp rx=0 tx=0).stop 2023-08-25T10:02:31.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.509+0000 7f0f0ad60640 1 -- 172.21.15.71:0/583283175 shutdown_connections 2023-08-25T10:02:31.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.509+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/583283175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f040c56c0 0x7f0f041576b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.509+0000 7f0f0ad60640 1 -- 172.21.15.71:0/583283175 >> 172.21.15.71:0/583283175 conn(0x7f0f0406c280 msgr2=0x7f0f0406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:31.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.509+0000 7f0f0ad60640 1 -- 172.21.15.71:0/583283175 shutdown_connections 2023-08-25T10:02:31.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.509+0000 7f0f0ad60640 1 -- 172.21.15.71:0/583283175 wait complete. 2023-08-25T10:02:31.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.511+0000 7f0f0ad60640 1 Processor -- start 2023-08-25T10:02:31.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.511+0000 7f0f0ad60640 1 -- start start 2023-08-25T10:02:31.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.512+0000 7f0f0ad60640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 0x7f0f0414ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:31.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.512+0000 7f0f0ad60640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f040c56c0 0x7f0f0406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.512+0000 7f0f0ad60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f041542d0 0x7f0f0406f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.512+0000 7f0f0ad60640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0406e2e0 con 0x7f0f04069850 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.512+0000 7f0f0ad60640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0406ee60 con 0x7f0f040c56c0 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.512+0000 7f0f0ad60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f041ea820 con 0x7f0f041542d0 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 0x7f0f0414ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 0x7f0f0414ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48416/0 (socket says 172.21.15.71:48416) 2023-08-25T10:02:31.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 -- 172.21.15.71:0/1237743120 learned_addr learned my addr 172.21.15.71:0/1237743120 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:31.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f0955d640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f040c56c0 0x7f0f0406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:31.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f0a55f640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f041542d0 0x7f0f0406f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:31.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 -- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f041542d0 msgr2=0x7f0f0406f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:31.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f041542d0 0x7f0f0406f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 -- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f040c56c0 msgr2=0x7f0f0406f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f040c56c0 0x7f0f0406f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.513+0000 7f0f09d5e640 1 -- 172.21.15.71:0/1237743120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc009b30 con 0x7f0f04069850 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.514+0000 7f0f09d5e640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 0x7f0f0414ed10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000c400 tx=0x7f0f0000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.514+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000052d0 con 0x7f0f04069850 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.514+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f00005430 con 0x7f0f04069850 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.514+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0406e530 con 0x7f0f04069850 2023-08-25T10:02:31.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.514+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00003830 con 0x7f0f04069850 2023-08-25T10:02:31.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.514+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0406eaf0 con 0x7f0f04069850 2023-08-25T10:02:31.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.516+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0404e420 con 0x7f0f04069850 2023-08-25T10:02:31.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.517+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0f00003990 con 0x7f0f04069850 2023-08-25T10:02:31.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.518+0000 7f0efa7fc640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ee806a050 0x7f0ee806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:31.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.518+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f0f00087e40 con 0x7f0f04069850 2023-08-25T10:02:31.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.518+0000 7f0f0955d640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ee806a050 0x7f0ee806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:31.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.519+0000 7f0f0955d640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ee806a050 0x7f0ee806c510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0f040ae760 tx=0x7f0ef4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:31.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.523+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0f00053070 con 0x7f0f04069850 2023-08-25T10:02:31.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.790+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f04157bf0 con 0x7f0f04069850 2023-08-25T10:02:31.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.791+0000 7f0efa7fc640 1 -- 172.21.15.71:0/1237743120 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f0f04157bf0 con 0x7f0f04069850 2023-08-25T10:02:31.793 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:31.793 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:31.794 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ee806a050 msgr2=0x7f0ee806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ee806a050 0x7f0ee806c510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0f040ae760 tx=0x7f0ef4009040 comp rx=0 tx=0).stop 2023-08-25T10:02:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 msgr2=0x7f0f0414ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 0x7f0f0414ed10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000c400 tx=0x7f0f0000c8c0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 shutdown_connections 2023-08-25T10:02:31.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f040c56c0 0x7f0f0406f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ee806a050 0x7f0ee806c510 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f041542d0 0x7f0f0406f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 --2- 172.21.15.71:0/1237743120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0f04069850 0x7f0f0414ed10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 >> 172.21.15.71:0/1237743120 conn(0x7f0f0406c280 msgr2=0x7f0f040c4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 shutdown_connections 2023-08-25T10:02:31.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:31.794+0000 7f0f0ad60640 1 -- 172.21.15.71:0/1237743120 wait complete. 2023-08-25T10:02:31.817 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:02:31.818 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:02:33.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:34.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.075+0000 7f6bf17c4640 1 Processor -- start 2023-08-25T10:02:34.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.075+0000 7f6bf17c4640 1 -- start start 2023-08-25T10:02:34.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.076+0000 7f6bf17c4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 0x7f6bec1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.076+0000 7f6bf17c4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec157bf0 con 0x7f6bec1542d0 2023-08-25T10:02:34.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.076+0000 7f6bf17c4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec06a130 con 0x7f6bec069850 2023-08-25T10:02:34.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.076+0000 7f6bf17c4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0c4b60 con 0x7f6bec0c56c0 2023-08-25T10:02:34.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.077+0000 7f6bebfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6bec1542d0 0x7f6bec0c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38482/0 (socket says 172.21.15.71:38482) 2023-08-25T10:02:34.081 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.077+0000 7f6bebfff640 1 -- 172.21.15.71:0/226748273 learned_addr learned my addr 172.21.15.71:0/226748273 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.077+0000 7f6bf0fc3640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 0x7f6bec1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.077+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635260448 0 0) 0x7f6bdc001110 con 0x7f6bec1542d0 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.077+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd4003520 con 0x7f6bec1542d0 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179465112 0 0) 0x7f6bd4003520 con 0x7f6bec1542d0 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 msgr2=0x7f6bec1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 0x7f6bec1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 >> v1:172.21.15.164:6789/0 conn(0x7f6bec069850 legacy=0x7f6bec0c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec1546b0 con 0x7f6bec1542d0 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f6bdc001800 con 0x7f6bec1542d0 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 >> v1:172.21.15.71:6789/0 conn(0x7f6bec1542d0 legacy=0x7f6bec0c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bd4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 0x7f6bd40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bd4004d00 0x7f6bd4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0c4b60 con 0x7f6bd4003a10 2023-08-25T10:02:34.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd400bac0 con 0x7f6bd4004d00 2023-08-25T10:02:34.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd400bc80 con 0x7f6bec0c56c0 2023-08-25T10:02:34.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6bebfff640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bd4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6beb7fe640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 0x7f6bd40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.078+0000 7f6bf0fc3640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bd4004d00 0x7f6bd4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.079+0000 7f6bebfff640 1 -- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 msgr2=0x7f6bd40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:34.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.079+0000 7f6bebfff640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c56c0 0x7f6bd40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.079+0000 7f6bebfff640 1 -- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bd4004d00 msgr2=0x7f6bd4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:34.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.079+0000 7f6bebfff640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bd4004d00 0x7f6bd4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.079+0000 7f6bebfff640 1 -- 172.21.15.71:0/226748273 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd4003520 con 0x7f6bd4003a10 2023-08-25T10:02:34.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.080+0000 7f6bebfff640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bd4003e30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00c160 tx=0x7f6bd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bf4fd41929479 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:34.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.080+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc005ad0 con 0x7f6bd4003a10 2023-08-25T10:02:34.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.080+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc005c50 con 0x7f6bd4003a10 2023-08-25T10:02:34.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.080+0000 7f6beaffd640 1 -- 172.21.15.71:0/226748273 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc007070 con 0x7f6bd4003a10 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf17c4640 1 -- 172.21.15.71:0/226748273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 msgr2=0x7f6bd4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/226748273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bd4003e30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00c160 tx=0x7f6bd40057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf0fc3640 1 -- 172.21.15.71:0/226748273 reap_dead start 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf17c4640 1 -- 172.21.15.71:0/226748273 shutdown_connections 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf17c4640 1 -- 172.21.15.71:0/226748273 >> 172.21.15.71:0/226748273 conn(0x7f6bec06c280 msgr2=0x7f6bec06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf17c4640 1 -- 172.21.15.71:0/226748273 shutdown_connections 2023-08-25T10:02:34.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.081+0000 7f6bf17c4640 1 -- 172.21.15.71:0/226748273 wait complete. 2023-08-25T10:02:34.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.083+0000 7f6bf17c4640 1 Processor -- start 2023-08-25T10:02:34.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.083+0000 7f6bf17c4640 1 -- start start 2023-08-25T10:02:34.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.084+0000 7f6bf17c4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bec14f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.084+0000 7f6bf17c4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bec1538d0 0x7f6bec14ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.084+0000 7f6bf17c4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c94c0 0x7f6bec14d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.084+0000 7f6bf17c4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec1ee000 con 0x7f6bd4003a10 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.084+0000 7f6bf17c4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0c05e0 con 0x7f6bec1538d0 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.084+0000 7f6bf17c4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec145c30 con 0x7f6bec0c94c0 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bec14f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bec14f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48438/0 (socket says 172.21.15.71:48438) 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 -- 172.21.15.71:0/786240384 learned_addr learned my addr 172.21.15.71:0/786240384 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6beb7fe640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bec1538d0 0x7f6bec14ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bf0fc3640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c94c0 0x7f6bec14d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 -- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c94c0 msgr2=0x7f6bec14d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c94c0 0x7f6bec14d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 -- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bec1538d0 msgr2=0x7f6bec14ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bec1538d0 0x7f6bec14ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.085+0000 7f6bebfff640 1 -- 172.21.15.71:0/786240384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec14fbd0 con 0x7f6bd4003a10 2023-08-25T10:02:34.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.086+0000 7f6bebfff640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bec14f690 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00c160 tx=0x7f6bec0c1800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:34.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.086+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc03c040 con 0x7f6bd4003a10 2023-08-25T10:02:34.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.086+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc0074a0 con 0x7f6bd4003a10 2023-08-25T10:02:34.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.086+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc001970 con 0x7f6bd4003a10 2023-08-25T10:02:34.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.086+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bec0c0aa0 con 0x7f6bd4003a10 2023-08-25T10:02:34.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.086+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec0c1060 con 0x7f6bd4003a10 2023-08-25T10:02:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.087+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec04e420 con 0x7f6bd4003a10 2023-08-25T10:02:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.088+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6bdc032070 con 0x7f6bd4003a10 2023-08-25T10:02:34.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.089+0000 7f6be8ff9640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6bd806a080 0x7f6bd806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.090+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6bdc0a9b10 con 0x7f6bd4003a10 2023-08-25T10:02:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.090+0000 7f6beb7fe640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6bd806a080 0x7f6bd806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:34.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.091+0000 7f6beb7fe640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6bd806a080 0x7f6bd806c540 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6bec149160 tx=0x7f6be0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:34.097 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.094+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6bdc074c10 con 0x7f6bd4003a10 2023-08-25T10:02:34.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.342+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bec157bf0 con 0x7f6bd4003a10 2023-08-25T10:02:34.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.344+0000 7f6be8ff9640 1 -- 172.21.15.71:0/786240384 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f6bec157bf0 con 0x7f6bd4003a10 2023-08-25T10:02:34.347 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:34.347 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:34.347 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:34.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6bd806a080 msgr2=0x7f6bd806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6bd806a080 0x7f6bd806c540 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6bec149160 tx=0x7f6be0009040 comp rx=0 tx=0).stop 2023-08-25T10:02:34.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 msgr2=0x7f6bec14f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:34.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bec14f690 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00c160 tx=0x7f6bec0c1800 comp rx=0 tx=0).stop 2023-08-25T10:02:34.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 shutdown_connections 2023-08-25T10:02:34.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bec1538d0 0x7f6bec14ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6bd806a080 0x7f6bd806c540 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6bec0c94c0 0x7f6bec14d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 --2- 172.21.15.71:0/786240384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6bd4003a10 0x7f6bec14f690 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:34.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 >> 172.21.15.71:0/786240384 conn(0x7f6bec06c280 msgr2=0x7f6bec06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:34.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 shutdown_connections 2023-08-25T10:02:34.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:34.347+0000 7f6bf17c4640 1 -- 172.21.15.71:0/786240384 wait complete. 2023-08-25T10:02:34.373 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:36.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:36.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.591+0000 7f4127396640 1 Processor -- start 2023-08-25T10:02:36.594 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.591+0000 7f4127396640 1 -- start start 2023-08-25T10:02:36.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.593+0000 7f4127396640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f412006a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:36.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.593+0000 7f4127396640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f412006aea0 0x7f41201592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:36.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.593+0000 7f4127396640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4120158e40 con 0x7f41201530c0 2023-08-25T10:02:36.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.593+0000 7f4127396640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412006bc80 con 0x7f412006aea0 2023-08-25T10:02:36.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.593+0000 7f4127396640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41201534a0 con 0x7f41200c7740 2023-08-25T10:02:36.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.594+0000 7f4124931640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f412006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:36.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.594+0000 7f4125933640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f412006aea0 0x7f41201592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:36.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.594+0000 7f4124931640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f412006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:38036/0 (socket says 172.21.15.71:38036) 2023-08-25T10:02:36.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.594+0000 7f4124931640 1 -- 172.21.15.71:0/2450877487 learned_addr learned my addr 172.21.15.71:0/2450877487 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:36.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.595+0000 7f4124931640 1 -- 172.21.15.71:0/2450877487 >> v2:172.21.15.164:3300/0 conn(0x7f412006aea0 msgr2=0x7f41201592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:36.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.595+0000 7f4124931640 1 --2- 172.21.15.71:0/2450877487 >> v2:172.21.15.164:3300/0 conn(0x7f412006aea0 0x7f41201592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.595+0000 7f4124931640 1 -- 172.21.15.71:0/2450877487 >> v1:172.21.15.164:6789/0 conn(0x7f41201530c0 legacy=0x7f41200c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:36.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.595+0000 7f4124931640 1 -- 172.21.15.71:0/2450877487 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412015c330 con 0x7f41200c7740 2023-08-25T10:02:36.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.595+0000 7f4125933640 1 --2- 172.21.15.71:0/2450877487 >> v2:172.21.15.164:3300/0 conn(0x7f412006aea0 0x7f41201592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T10:02:36.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.596+0000 7f4124931640 1 --2- 172.21.15.71:0/2450877487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f412006a960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4110009870 tx=0x7f411002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b57677c0183cf server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:36.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.597+0000 7f4117fff640 1 -- 172.21.15.71:0/2450877487 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411002f830 con 0x7f41200c7740 2023-08-25T10:02:36.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.597+0000 7f4117fff640 1 -- 172.21.15.71:0/2450877487 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411002f990 con 0x7f41200c7740 2023-08-25T10:02:36.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.597+0000 7f4117fff640 1 -- 172.21.15.71:0/2450877487 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411002fd20 con 0x7f41200c7740 2023-08-25T10:02:36.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.597+0000 7f4127396640 1 -- 172.21.15.71:0/2450877487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 msgr2=0x7f412006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.597+0000 7f4127396640 1 --2- 172.21.15.71:0/2450877487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f412006a960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4110009870 tx=0x7f411002ba10 comp rx=0 tx=0).stop 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.598+0000 7f4127396640 1 -- 172.21.15.71:0/2450877487 shutdown_connections 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.598+0000 7f4127396640 1 --2- 172.21.15.71:0/2450877487 >> v2:172.21.15.164:3300/0 conn(0x7f412006aea0 0x7f41201592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.598+0000 7f4127396640 1 --2- 172.21.15.71:0/2450877487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f412006a960 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.598+0000 7f4127396640 1 -- 172.21.15.71:0/2450877487 >> 172.21.15.71:0/2450877487 conn(0x7f41200654f0 msgr2=0x7f4120065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.598+0000 7f4127396640 1 -- 172.21.15.71:0/2450877487 shutdown_connections 2023-08-25T10:02:36.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.598+0000 7f4127396640 1 -- 172.21.15.71:0/2450877487 wait complete. 2023-08-25T10:02:36.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.599+0000 7f4127396640 1 Processor -- start 2023-08-25T10:02:36.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.600+0000 7f4127396640 1 -- start start 2023-08-25T10:02:36.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.600+0000 7f4127396640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412006aea0 0x7f4120156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:36.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4127396640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f41200c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:36.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4127396640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 0x7f41201f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:36.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4127396640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41201ee080 con 0x7f41201530c0 2023-08-25T10:02:36.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4127396640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41201ef050 con 0x7f412006aea0 2023-08-25T10:02:36.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4127396640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412014eef0 con 0x7f41200c7740 2023-08-25T10:02:36.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4125933640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 0x7f41201f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4125933640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 0x7f41201f1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48454/0 (socket says 172.21.15.71:48454) 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4125933640 1 -- 172.21.15.71:0/2678230876 learned_addr learned my addr 172.21.15.71:0/2678230876 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4124931640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f41200c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.601+0000 7f4125132640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412006aea0 0x7f4120156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125933640 1 -- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 msgr2=0x7f41200c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125933640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f41200c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125933640 1 -- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412006aea0 msgr2=0x7f4120156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125933640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412006aea0 0x7f4120156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125933640 1 -- 172.21.15.71:0/2678230876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118009a70 con 0x7f41201530c0 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125132640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412006aea0 0x7f4120156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4125933640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 0x7f41201f1630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4118009f00 tx=0x7f411800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118009d70 con 0x7f41201530c0 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800f040 con 0x7f41201530c0 2023-08-25T10:02:36.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118003bb0 con 0x7f41201530c0 2023-08-25T10:02:36.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41100094d0 con 0x7f41201530c0 2023-08-25T10:02:36.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.602+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41201ee290 con 0x7f41201530c0 2023-08-25T10:02:36.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.604+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4120069160 con 0x7f41201530c0 2023-08-25T10:02:36.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.605+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4118014480 con 0x7f41201530c0 2023-08-25T10:02:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.606+0000 7f4115ffb640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410406a050 0x7f410406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.606+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4118089800 con 0x7f41201530c0 2023-08-25T10:02:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.607+0000 7f4125132640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410406a050 0x7f410406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:36.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.607+0000 7f4125132640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410406a050 0x7f410406c510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f411c001660 tx=0x7f411c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:36.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.611+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4118058390 con 0x7f41201530c0 2023-08-25T10:02:36.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.859+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4120158bf0 con 0x7f41201530c0 2023-08-25T10:02:36.862 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.860+0000 7f4115ffb640 1 -- 172.21.15.71:0/2678230876 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4118057f20 con 0x7f41201530c0 2023-08-25T10:02:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:36.863 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:36.863 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:36.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410406a050 msgr2=0x7f410406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:36.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410406a050 0x7f410406c510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f411c001660 tx=0x7f411c009040 comp rx=0 tx=0).stop 2023-08-25T10:02:36.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 msgr2=0x7f41201f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:36.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 0x7f41201f1630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f4118009f00 tx=0x7f411800c490 comp rx=0 tx=0).stop 2023-08-25T10:02:36.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 shutdown_connections 2023-08-25T10:02:36.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f410406a050 0x7f410406c510 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f41201530c0 0x7f41201f1630 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41200c7740 0x7f41200c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.868 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 --2- 172.21.15.71:0/2678230876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f412006aea0 0x7f4120156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:36.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 >> 172.21.15.71:0/2678230876 conn(0x7f41200654f0 msgr2=0x7f4120152040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:36.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 shutdown_connections 2023-08-25T10:02:36.869 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:36.863+0000 7f4127396640 1 -- 172.21.15.71:0/2678230876 wait complete. 2023-08-25T10:02:36.886 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:38.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:39.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.123+0000 7f436a2e1640 1 Processor -- start 2023-08-25T10:02:39.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.124+0000 7f436a2e1640 1 -- start start 2023-08-25T10:02:39.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f436a2e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 0x7f436415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.128 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f436a2e1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f436406b180 0x7f43640c4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f436a2e1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43640c2980 con 0x7f436406b940 2023-08-25T10:02:39.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f436a2e1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43640c2c60 con 0x7f436406b180 2023-08-25T10:02:39.129 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f436a2e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436415a2d0 con 0x7f436406bd20 2023-08-25T10:02:39.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f4363fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f436406b940 0x7f436406a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:38490/0 (socket says 172.21.15.71:38490) 2023-08-25T10:02:39.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f4363fff640 1 -- 172.21.15.71:0/1092337240 learned_addr learned my addr 172.21.15.71:0/1092337240 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:39.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f43637fe640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 0x7f436415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.126+0000 7f436887e640 1 --2- 172.21.15.71:0/1092337240 >> v2:172.21.15.164:3300/0 conn(0x7f436406b180 0x7f43640c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450303101 0 0) 0x7f43640c2980 con 0x7f436406b940 2023-08-25T10:02:39.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f434c003520 con 0x7f436406b940 2023-08-25T10:02:39.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4284612891 0 0) 0x7f434c003520 con 0x7f436406b940 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 msgr2=0x7f436415adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 0x7f436415adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 >> v2:172.21.15.164:3300/0 conn(0x7f436406b180 msgr2=0x7f43640c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 --2- 172.21.15.71:0/1092337240 >> v2:172.21.15.164:3300/0 conn(0x7f436406b180 0x7f43640c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436415a5b0 con 0x7f436406b940 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4354001670 con 0x7f436406b940 2023-08-25T10:02:39.132 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 >> v1:172.21.15.71:6789/0 conn(0x7f436406b940 legacy=0x7f436406a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c0039e0 0x7f434c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 0x7f434c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f434c004d00 0x7f434c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43640c2c60 con 0x7f434c004d00 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436415a2d0 con 0x7f434c0039e0 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.127+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c00fd90 con 0x7f436406bd20 2023-08-25T10:02:39.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f436887e640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f434c004d00 0x7f434c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f43637fe640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 0x7f434c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f4363fff640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c0039e0 0x7f434c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f436887e640 1 -- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 msgr2=0x7f434c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f436887e640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f436406bd20 0x7f434c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f436887e640 1 -- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c0039e0 msgr2=0x7f434c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f436887e640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c0039e0 0x7f434c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.128+0000 7f436887e640 1 -- 172.21.15.71:0/1092337240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c003520 con 0x7f434c004d00 2023-08-25T10:02:39.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.129+0000 7f436887e640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f434c004d00 0x7f434c005180 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f435c009fc0 tx=0x7f434c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b78ea883102b7a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:39.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c02cc60 con 0x7f434c004d00 2023-08-25T10:02:39.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c00f040 con 0x7f434c004d00 2023-08-25T10:02:39.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f4362ffd640 1 -- 172.21.15.71:0/1092337240 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c034440 con 0x7f434c004d00 2023-08-25T10:02:39.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f436a2e1640 1 -- 172.21.15.71:0/1092337240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f434c004d00 msgr2=0x7f434c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:39.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1092337240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f434c004d00 0x7f434c005180 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f435c009fc0 tx=0x7f434c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f436887e640 1 -- 172.21.15.71:0/1092337240 reap_dead start 2023-08-25T10:02:39.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f436a2e1640 1 -- 172.21.15.71:0/1092337240 shutdown_connections 2023-08-25T10:02:39.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f436a2e1640 1 -- 172.21.15.71:0/1092337240 >> 172.21.15.71:0/1092337240 conn(0x7f43640654f0 msgr2=0x7f4364065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:39.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.130+0000 7f436a2e1640 1 -- 172.21.15.71:0/1092337240 shutdown_connections 2023-08-25T10:02:39.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.131+0000 7f436a2e1640 1 -- 172.21.15.71:0/1092337240 wait complete. 2023-08-25T10:02:39.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.133+0000 7f436a2e1640 1 Processor -- start 2023-08-25T10:02:39.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.133+0000 7f436a2e1640 1 -- start start 2023-08-25T10:02:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f436a2e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c004d00 0x7f436414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f436a2e1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 0x7f4364145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f436a2e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4364070de0 0x7f4364071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f436a2e1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4364159cb0 con 0x7f43641439b0 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f436a2e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4364141f10 con 0x7f434c004d00 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f436a2e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43641421f0 con 0x7f4364070de0 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f43637fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 0x7f4364145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f43637fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 0x7f4364145b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:48474/0 (socket says 172.21.15.71:48474) 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.134+0000 7f43637fe640 1 -- 172.21.15.71:0/1330967673 learned_addr learned my addr 172.21.15.71:0/1330967673 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f436887e640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4364070de0 0x7f4364071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f4363fff640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c004d00 0x7f436414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f43637fe640 1 -- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4364070de0 msgr2=0x7f4364071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f43637fe640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4364070de0 0x7f4364071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f43637fe640 1 -- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c004d00 msgr2=0x7f436414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f43637fe640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c004d00 0x7f436414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f43637fe640 1 -- 172.21.15.71:0/1330967673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43641424d0 con 0x7f43641439b0 2023-08-25T10:02:39.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f43637fe640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 0x7f4364145b00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4358011600 tx=0x7f4358011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.135+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435800f920 con 0x7f43641439b0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.136+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435800fa80 con 0x7f43641439b0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.136+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43641427b0 con 0x7f43641439b0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.136+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435800fdf0 con 0x7f43641439b0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.136+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43640c21b0 con 0x7f43641439b0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.137+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4364069160 con 0x7f43641439b0 2023-08-25T10:02:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.138+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4358015460 con 0x7f43641439b0 2023-08-25T10:02:39.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.139+0000 7f4360ff9640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f435006a080 0x7f435006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:39.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.139+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4358089390 con 0x7f43641439b0 2023-08-25T10:02:39.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.140+0000 7f4363fff640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f435006a080 0x7f435006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:39.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.140+0000 7f4363fff640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f435006a080 0x7f435006c540 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4354001050 tx=0x7f4354009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:39.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.144+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4358054520 con 0x7f43641439b0 2023-08-25T10:02:39.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.414+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f436415a860 con 0x7f43641439b0 2023-08-25T10:02:39.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.415+0000 7f4360ff9640 1 -- 172.21.15.71:0/1330967673 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4358057b40 con 0x7f43641439b0 2023-08-25T10:02:39.418 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:39.418 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:39.418 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:39.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f435006a080 msgr2=0x7f435006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:39.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f435006a080 0x7f435006c540 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4354001050 tx=0x7f4354009040 comp rx=0 tx=0).stop 2023-08-25T10:02:39.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 msgr2=0x7f4364145b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 0x7f4364145b00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4358011600 tx=0x7f4358011ac0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 shutdown_connections 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f435006a080 0x7f435006c540 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4364070de0 0x7f4364071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43641439b0 0x7f4364145b00 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 --2- 172.21.15.71:0/1330967673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f434c004d00 0x7f436414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:39.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 >> 172.21.15.71:0/1330967673 conn(0x7f43640654f0 msgr2=0x7f43640c4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:39.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 shutdown_connections 2023-08-25T10:02:39.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:39.417+0000 7f436a2e1640 1 -- 172.21.15.71:0/1330967673 wait complete. 2023-08-25T10:02:39.438 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:41.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:41.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.683+0000 7f2e45b74640 1 Processor -- start 2023-08-25T10:02:41.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.684+0000 7f2e45b74640 1 -- start start 2023-08-25T10:02:41.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.685+0000 7f2e45b74640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e400ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:41.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.685+0000 7f2e45b74640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2e4015cd00 0x7f2e40158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:41.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.685+0000 7f2e45b74640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e400b1af0 con 0x7f2e400cb190 2023-08-25T10:02:41.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.685+0000 7f2e45b74640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e400c98f0 con 0x7f2e4015cd00 2023-08-25T10:02:41.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.685+0000 7f2e45b74640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4015d190 con 0x7f2e400cb570 2023-08-25T10:02:41.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e400ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:41.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e45373640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2e4015cd00 0x7f2e40158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:41.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e400ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:39988/0 (socket says 172.21.15.71:39988) 2023-08-25T10:02:41.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 -- 172.21.15.71:0/2394416189 learned_addr learned my addr 172.21.15.71:0/2394416189 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:41.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 -- 172.21.15.71:0/2394416189 >> v2:172.21.15.164:3300/0 conn(0x7f2e4015cd00 msgr2=0x7f2e40158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:41.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 --2- 172.21.15.71:0/2394416189 >> v2:172.21.15.164:3300/0 conn(0x7f2e4015cd00 0x7f2e40158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 -- 172.21.15.71:0/2394416189 >> v1:172.21.15.164:6789/0 conn(0x7f2e400cb190 legacy=0x7f2e400c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:41.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.686+0000 7f2e3ffff640 1 -- 172.21.15.71:0/2394416189 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e400b0fb0 con 0x7f2e400cb570 2023-08-25T10:02:41.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.688+0000 7f2e3ffff640 1 --2- 172.21.15.71:0/2394416189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e400ca910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009870 tx=0x7f2e2c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72d0b94f7f2afd7f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:41.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e3f7fe640 1 -- 172.21.15.71:0/2394416189 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c02f830 con 0x7f2e400cb570 2023-08-25T10:02:41.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e3f7fe640 1 -- 172.21.15.71:0/2394416189 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2c02f990 con 0x7f2e400cb570 2023-08-25T10:02:41.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e3f7fe640 1 -- 172.21.15.71:0/2394416189 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c02fd20 con 0x7f2e400cb570 2023-08-25T10:02:41.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e45b74640 1 -- 172.21.15.71:0/2394416189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 msgr2=0x7f2e400ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:41.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2394416189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e400ca910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c009870 tx=0x7f2e2c02ba10 comp rx=0 tx=0).stop 2023-08-25T10:02:41.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e45b74640 1 -- 172.21.15.71:0/2394416189 shutdown_connections 2023-08-25T10:02:41.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2394416189 >> v2:172.21.15.164:3300/0 conn(0x7f2e4015cd00 0x7f2e40158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2394416189 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e400ca910 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.689+0000 7f2e45b74640 1 -- 172.21.15.71:0/2394416189 >> 172.21.15.71:0/2394416189 conn(0x7f2e4006ae20 msgr2=0x7f2e4006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:41.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.690+0000 7f2e45b74640 1 -- 172.21.15.71:0/2394416189 shutdown_connections 2023-08-25T10:02:41.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.690+0000 7f2e45b74640 1 -- 172.21.15.71:0/2394416189 wait complete. 2023-08-25T10:02:41.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.692+0000 7f2e45b74640 1 Processor -- start 2023-08-25T10:02:41.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.692+0000 7f2e45b74640 1 -- start start 2023-08-25T10:02:41.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45b74640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e400cb190 0x7f2e40155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:41.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45b74640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e4015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:41.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45b74640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 0x7f2e400cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:41.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45b74640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e400cd3e0 con 0x7f2e4015cd00 2023-08-25T10:02:41.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45b74640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e401563b0 con 0x7f2e400cb190 2023-08-25T10:02:41.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45b74640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e401f2010 con 0x7f2e400cb570 2023-08-25T10:02:41.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.693+0000 7f2e45373640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 0x7f2e400cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:41.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 0x7f2e400cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38222/0 (socket says 172.21.15.71:38222) 2023-08-25T10:02:41.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 -- 172.21.15.71:0/2963256991 learned_addr learned my addr 172.21.15.71:0/2963256991 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:41.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e44b72640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e400cb190 0x7f2e40155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:41.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e3ffff640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e4015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:41.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 -- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 msgr2=0x7f2e4015b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:41.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e4015b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 -- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e400cb190 msgr2=0x7f2e40155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:41.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e400cb190 0x7f2e40155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 -- 172.21.15.71:0/2963256991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0094d0 con 0x7f2e4015cd00 2023-08-25T10:02:41.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.694+0000 7f2e45373640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 0x7f2e400cdb50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800efc0 tx=0x7f2e3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.695+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38014480 con 0x7f2e4015cd00 2023-08-25T10:02:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.695+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3800f040 con 0x7f2e4015cd00 2023-08-25T10:02:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.695+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e401f22f0 con 0x7f2e4015cd00 2023-08-25T10:02:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.695+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38014820 con 0x7f2e4015cd00 2023-08-25T10:02:41.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.695+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e40155e90 con 0x7f2e4015cd00 2023-08-25T10:02:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.696+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e400cd870 con 0x7f2e4015cd00 2023-08-25T10:02:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.697+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2e38014980 con 0x7f2e4015cd00 2023-08-25T10:02:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.698+0000 7f2e3d7fa640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2e2406a080 0x7f2e2406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.698+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f2e380897f0 con 0x7f2e4015cd00 2023-08-25T10:02:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.699+0000 7f2e44b72640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2e2406a080 0x7f2e2406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:41.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.699+0000 7f2e44b72640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2e2406a080 0x7f2e2406c540 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e34000f60 tx=0x7f2e34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:41.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.703+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2e400cd870 con 0x7f2e4015cd00 2023-08-25T10:02:41.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.976+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e400cd870 con 0x7f2e4015cd00 2023-08-25T10:02:41.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.977+0000 7f2e3d7fa640 1 -- 172.21.15.71:0/2963256991 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f2e400cd870 con 0x7f2e4015cd00 2023-08-25T10:02:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:41.981 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:41.982 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:41.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2e2406a080 msgr2=0x7f2e2406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:41.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2e2406a080 0x7f2e2406c540 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2e34000f60 tx=0x7f2e34009040 comp rx=0 tx=0).stop 2023-08-25T10:02:41.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 msgr2=0x7f2e400cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:41.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 0x7f2e400cdb50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800efc0 tx=0x7f2e3800c490 comp rx=0 tx=0).stop 2023-08-25T10:02:41.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 shutdown_connections 2023-08-25T10:02:41.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2e2406a080 0x7f2e2406c540 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2e4015cd00 0x7f2e400cdb50 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2e400cb570 0x7f2e4015b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 --2- 172.21.15.71:0/2963256991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2e400cb190 0x7f2e40155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:41.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 >> 172.21.15.71:0/2963256991 conn(0x7f2e4006ae20 msgr2=0x7f2e40151e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:41.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 shutdown_connections 2023-08-25T10:02:41.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:41.980+0000 7f2e45b74640 1 -- 172.21.15.71:0/2963256991 wait complete. 2023-08-25T10:02:42.001 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:44.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:44.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.236+0000 7f4dd6458640 1 Processor -- start 2023-08-25T10:02:44.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.236+0000 7f4dd6458640 1 -- start start 2023-08-25T10:02:44.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.238+0000 7f4dd6458640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4dd00cb190 0x7f4dd00ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.238+0000 7f4dd6458640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00b1af0 con 0x7f4dd015cd00 2023-08-25T10:02:44.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.238+0000 7f4dd6458640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00b0fb0 con 0x7f4dd00cb570 2023-08-25T10:02:44.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.238+0000 7f4dd6458640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00c98f0 con 0x7f4dd00cb190 2023-08-25T10:02:44.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.239+0000 7f4dd5c57640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4dd015cd00 0x7f4dd0158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47502/0 (socket says 172.21.15.71:47502) 2023-08-25T10:02:44.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.239+0000 7f4dd5c57640 1 -- 172.21.15.71:0/3642684874 learned_addr learned my addr 172.21.15.71:0/3642684874 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:44.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.239+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3642684874 >> v2:172.21.15.164:3300/0 conn(0x7f4dd00cb190 0x7f4dd00ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.239+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850106195 0 0) 0x7f4dd00b1af0 con 0x7f4dd015cd00 2023-08-25T10:02:44.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.239+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dbc003520 con 0x7f4dd015cd00 2023-08-25T10:02:44.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027774281 0 0) 0x7f4dbc003520 con 0x7f4dd015cd00 2023-08-25T10:02:44.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 >> v2:172.21.15.164:3300/0 conn(0x7f4dd00cb190 msgr2=0x7f4dd00ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:44.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 --2- 172.21.15.71:0/3642684874 >> v2:172.21.15.164:3300/0 conn(0x7f4dd00cb190 0x7f4dd00ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 >> v1:172.21.15.164:6789/0 conn(0x7f4dd00cb570 legacy=0x7f4dd00c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd015d130 con 0x7f4dd015cd00 2023-08-25T10:02:44.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4dc80019f0 con 0x7f4dd015cd00 2023-08-25T10:02:44.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 >> v1:172.21.15.71:6789/0 conn(0x7f4dd015cd00 legacy=0x7f4dd0158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dbc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dbc0047c0 0x7f4dbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dbc005180 0x7f4dbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c98f0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc00c090 con 0x7f4dbc0047c0 2023-08-25T10:02:44.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc00c250 con 0x7f4dbc005180 2023-08-25T10:02:44.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.240+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dbc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5c57640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dbc005180 0x7f4dbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd4c55640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dbc0047c0 0x7f4dbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5456640 1 -- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dbc005180 msgr2=0x7f4dbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:44.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dbc005180 0x7f4dbc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5456640 1 -- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dbc0047c0 msgr2=0x7f4dbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:44.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dbc0047c0 0x7f4dbc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5456640 1 -- 172.21.15.71:0/3642684874 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc003520 con 0x7f4dbc0039e0 2023-08-25T10:02:44.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd4c55640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dbc0047c0 0x7f4dbc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:44.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.241+0000 7f4dd5c57640 1 -- 172.21.15.71:0/3642684874 reap_dead start 2023-08-25T10:02:44.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.242+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dbc003e00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009ac0 tx=0x7f4dbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c8d4175a97bb81 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:44.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.242+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc02c6b0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.242+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc02c810 con 0x7f4dbc0039e0 2023-08-25T10:02:44.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.242+0000 7f4dc7fff640 1 -- 172.21.15.71:0/3642684874 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc02cba0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 -- 172.21.15.71:0/3642684874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 msgr2=0x7f4dbc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dbc003e00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009ac0 tx=0x7f4dbc0057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 -- 172.21.15.71:0/3642684874 shutdown_connections 2023-08-25T10:02:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3642684874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dbc003e00 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 -- 172.21.15.71:0/3642684874 >> 172.21.15.71:0/3642684874 conn(0x7f4dd006ae20 msgr2=0x7f4dd006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 -- 172.21.15.71:0/3642684874 shutdown_connections 2023-08-25T10:02:44.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.243+0000 7f4dd6458640 1 -- 172.21.15.71:0/3642684874 wait complete. 2023-08-25T10:02:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.245+0000 7f4dd6458640 1 Processor -- start 2023-08-25T10:02:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.245+0000 7f4dd6458640 1 -- start start 2023-08-25T10:02:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd6458640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dd0155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd6458640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd00c2600 0x7f4dd00c2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd6458640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00c41d0 0x7f4dd00c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd6458640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c2fa0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd6458640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd015c590 con 0x7f4dd00c41d0 2023-08-25T10:02:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd6458640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c3d40 con 0x7f4dd00c2600 2023-08-25T10:02:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.246+0000 7f4dd5456640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dd0155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dd0155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38236/0 (socket says 172.21.15.71:38236) 2023-08-25T10:02:44.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 -- 172.21.15.71:0/3190675396 learned_addr learned my addr 172.21.15.71:0/3190675396 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5c57640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00c41d0 0x7f4dd00c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd4c55640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd00c2600 0x7f4dd00c2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 -- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd00c2600 msgr2=0x7f4dd00c2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd00c2600 0x7f4dd00c2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 -- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00c41d0 msgr2=0x7f4dd00c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00c41d0 0x7f4dd00c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 -- 172.21.15.71:0/3190675396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00c4b90 con 0x7f4dbc0039e0 2023-08-25T10:02:44.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.247+0000 7f4dd5456640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dd0155460 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009ac0 tx=0x7f4dcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:44.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.248+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc003810 con 0x7f4dbc0039e0 2023-08-25T10:02:44.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.248+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc003970 con 0x7f4dbc0039e0 2023-08-25T10:02:44.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.248+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc003d00 con 0x7f4dbc0039e0 2023-08-25T10:02:44.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.248+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd015c7c0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.248+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd00b27a0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.249+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd004e420 con 0x7f4dbc0039e0 2023-08-25T10:02:44.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.250+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4dcc040450 con 0x7f4dbc0039e0 2023-08-25T10:02:44.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.252+0000 7f4dc5ffb640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:44.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.252+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4dcc0aa900 con 0x7f4dbc0039e0 2023-08-25T10:02:44.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.252+0000 7f4dd4c55640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:44.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.253+0000 7f4dd4c55640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00c1a10 tx=0x7f4dc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:44.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.256+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4dcc075b20 con 0x7f4dbc0039e0 2023-08-25T10:02:44.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.545+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd00c9df0 con 0x7f4dbc0039e0 2023-08-25T10:02:44.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.546+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3190675396 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4dcc003e60 con 0x7f4dbc0039e0 2023-08-25T10:02:44.549 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:44.550 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:44.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 msgr2=0x7f4db406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00c1a10 tx=0x7f4dc0009040 comp rx=0 tx=0).stop 2023-08-25T10:02:44.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 msgr2=0x7f4dd0155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:44.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dd0155460 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009ac0 tx=0x7f4dcc004730 comp rx=0 tx=0).stop 2023-08-25T10:02:44.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 shutdown_connections 2023-08-25T10:02:44.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dd00c41d0 0x7f4dd00c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dd00c2600 0x7f4dd00c2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 --2- 172.21.15.71:0/3190675396 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dbc0039e0 0x7f4dd0155460 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:44.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 >> 172.21.15.71:0/3190675396 conn(0x7f4dd006ae20 msgr2=0x7f4dd00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:44.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 shutdown_connections 2023-08-25T10:02:44.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:44.549+0000 7f4dd6458640 1 -- 172.21.15.71:0/3190675396 wait complete. 2023-08-25T10:02:44.571 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:02:44.572 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:02:46.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:46.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.802+0000 7f2a8bfff640 1 Processor -- start 2023-08-25T10:02:46.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.802+0000 7f2a8bfff640 1 -- start start 2023-08-25T10:02:46.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.803+0000 7f2a8bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2a8c150e20 0x7f2a8c06eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.803+0000 7f2a8bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2a8c0c3fc0 0x7f2a8c0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.803+0000 7f2a8bfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c06a010 con 0x7f2a8c06a770 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.803+0000 7f2a8bfff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c4910 con 0x7f2a8c150e20 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.803+0000 7f2a8bfff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c06f0e0 con 0x7f2a8c0c3fc0 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2a8c150e20 0x7f2a8c06eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2a8c150e20 0x7f2a8c06eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38242/0 (socket says 172.21.15.71:38242) 2023-08-25T10:02:46.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8affd640 1 -- 172.21.15.71:0/1226866515 learned_addr learned my addr 172.21.15.71:0/1226866515 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:46.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8a7fc640 1 --1- 172.21.15.71:0/1226866515 >> v1:172.21.15.71:6789/0 conn(0x7f2a8c06a770 0x7f2a8c156a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47508/0 (socket says 172.21.15.71:47508) 2023-08-25T10:02:46.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8b7fe640 1 --2- 172.21.15.71:0/1226866515 >> v2:172.21.15.164:3300/0 conn(0x7f2a8c0c3fc0 0x7f2a8c0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:46.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8affd640 1 -- 172.21.15.71:0/1226866515 >> v2:172.21.15.164:3300/0 conn(0x7f2a8c0c3fc0 msgr2=0x7f2a8c0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:46.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8affd640 1 --2- 172.21.15.71:0/1226866515 >> v2:172.21.15.164:3300/0 conn(0x7f2a8c0c3fc0 0x7f2a8c0c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:46.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.804+0000 7f2a8affd640 1 -- 172.21.15.71:0/1226866515 >> v1:172.21.15.71:6789/0 conn(0x7f2a8c06a770 legacy=0x7f2a8c156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:46.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.805+0000 7f2a8affd640 1 -- 172.21.15.71:0/1226866515 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c151200 con 0x7f2a8c150e20 2023-08-25T10:02:46.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.806+0000 7f2a8affd640 1 --2- 172.21.15.71:0/1226866515 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c150e20 0x7f2a8c06eba0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c06ab50 tx=0x7f2a7c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d05c279963925d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:46.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.806+0000 7f2a89ffb640 1 -- 172.21.15.71:0/1226866515 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c02f850 con 0x7f2a8c150e20 2023-08-25T10:02:46.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.806+0000 7f2a89ffb640 1 -- 172.21.15.71:0/1226866515 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c02f9b0 con 0x7f2a8c150e20 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.806+0000 7f2a89ffb640 1 -- 172.21.15.71:0/1226866515 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c02fd40 con 0x7f2a8c150e20 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1226866515 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c150e20 msgr2=0x7f2a8c06eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1226866515 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c150e20 0x7f2a8c06eba0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c06ab50 tx=0x7f2a7c02ba30 comp rx=0 tx=0).stop 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1226866515 shutdown_connections 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1226866515 >> v2:172.21.15.164:3300/0 conn(0x7f2a8c0c3fc0 0x7f2a8c0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1226866515 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c150e20 0x7f2a8c06eba0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1226866515 >> 172.21.15.71:0/1226866515 conn(0x7f2a8c0cab90 msgr2=0x7f2a8c0cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:46.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1226866515 shutdown_connections 2023-08-25T10:02:46.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.807+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1226866515 wait complete. 2023-08-25T10:02:46.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.809+0000 7f2a8bfff640 1 Processor -- start 2023-08-25T10:02:46.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.809+0000 7f2a8bfff640 1 -- start start 2023-08-25T10:02:46.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.810+0000 7f2a8bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 0x7f2a8c1e33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:46.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.810+0000 7f2a8bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a8c0c3fc0 0x7f2a8c147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:46.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.810+0000 7f2a8bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a8c150e20 0x7f2a8c144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:46.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8bfff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c5ff0 con 0x7f2a8c06a770 2023-08-25T10:02:46.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8bfff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c5ac0 con 0x7f2a8c0c3fc0 2023-08-25T10:02:46.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8bfff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c146fe0 con 0x7f2a8c150e20 2023-08-25T10:02:46.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 0x7f2a8c1e33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:46.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 0x7f2a8c1e33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38246/0 (socket says 172.21.15.71:38246) 2023-08-25T10:02:46.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 -- 172.21.15.71:0/1671597108 learned_addr learned my addr 172.21.15.71:0/1671597108 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:46.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 -- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a8c150e20 msgr2=0x7f2a8c144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:46.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a8c150e20 0x7f2a8c144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:46.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 -- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a8c0c3fc0 msgr2=0x7f2a8c147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8a7fc640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a8c0c3fc0 0x7f2a8c147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a8c0c3fc0 0x7f2a8c147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8affd640 1 -- 172.21.15.71:0/1671597108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c0094d0 con 0x7f2a8c06a770 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.811+0000 7f2a8a7fc640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a8c0c3fc0 0x7f2a8c147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.812+0000 7f2a8affd640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 0x7f2a8c1e33e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c000c00 tx=0x7f2a7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.812+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c037d10 con 0x7f2a8c06a770 2023-08-25T10:02:46.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.812+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c037e90 con 0x7f2a8c06a770 2023-08-25T10:02:46.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.812+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0426c0 con 0x7f2a8c06a770 2023-08-25T10:02:46.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.812+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c0cb0c0 con 0x7f2a8c06a770 2023-08-25T10:02:46.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.812+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c0c24a0 con 0x7f2a8c06a770 2023-08-25T10:02:46.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.814+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c0c2300 con 0x7f2a8c06a770 2023-08-25T10:02:46.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.815+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2a7c02d3e0 con 0x7f2a8c06a770 2023-08-25T10:02:46.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.815+0000 7f2a737fe640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7806a050 0x7f2a7806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:46.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.816+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f2a7c03c070 con 0x7f2a8c06a770 2023-08-25T10:02:46.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.816+0000 7f2a8a7fc640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7806a050 0x7f2a7806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:46.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.817+0000 7f2a8a7fc640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7806a050 0x7f2a7806c510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0c6570 tx=0x7f2a80009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:46.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:46.822+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2a8c0c2300 con 0x7f2a8c06a770 2023-08-25T10:02:47.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.152+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a8c0c2300 con 0x7f2a8c06a770 2023-08-25T10:02:47.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.153+0000 7f2a737fe640 1 -- 172.21.15.71:0/1671597108 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f2a8c0c2300 con 0x7f2a8c06a770 2023-08-25T10:02:47.156 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:47.156 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:47.157 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:47.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.155+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7806a050 msgr2=0x7f2a7806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:47.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.155+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7806a050 0x7f2a7806c510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0c6570 tx=0x7f2a80009040 comp rx=0 tx=0).stop 2023-08-25T10:02:47.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.155+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 msgr2=0x7f2a8c1e33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:47.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.155+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 0x7f2a8c1e33e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c000c00 tx=0x7f2a7c004730 comp rx=0 tx=0).stop 2023-08-25T10:02:47.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 shutdown_connections 2023-08-25T10:02:47.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2a8c0c3fc0 0x7f2a8c147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:47.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2a7806a050 0x7f2a7806c510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:47.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2a8c150e20 0x7f2a8c144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:47.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 --2- 172.21.15.71:0/1671597108 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2a8c06a770 0x7f2a8c1e33e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:47.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 >> 172.21.15.71:0/1671597108 conn(0x7f2a8c0cab90 msgr2=0x7f2a8c071020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:47.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 shutdown_connections 2023-08-25T10:02:47.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:47.156+0000 7f2a8bfff640 1 -- 172.21.15.71:0/1671597108 wait complete. 2023-08-25T10:02:47.179 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:49.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:49.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.411+0000 7ff99b59e640 1 Processor -- start 2023-08-25T10:02:49.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.411+0000 7ff99b59e640 1 -- start start 2023-08-25T10:02:49.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99b59e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff99c0cb190 0x7ff99c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:49.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99b59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff99c0cb570 0x7ff99c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:49.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99b59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99c0b1af0 con 0x7ff99c0c7f10 2023-08-25T10:02:49.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99b59e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99c0c98f0 con 0x7ff99c0cb190 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99b59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99c0b0fb0 con 0x7ff99c0cb570 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99ad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff99c0c7f10 0x7ff99c158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47522/0 (socket says 172.21.15.71:47522) 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.413+0000 7ff99ad9d640 1 -- 172.21.15.71:0/2577859390 learned_addr learned my addr 172.21.15.71:0/2577859390 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff99a59c640 1 --2- 172.21.15.71:0/2577859390 >> v2:172.21.15.71:3300/0 conn(0x7ff99c0cb190 0x7ff99c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff999d9b640 1 --2- 172.21.15.71:0/2577859390 >> v2:172.21.15.164:3300/0 conn(0x7ff99c0cb570 0x7ff99c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff99a59c640 1 -- 172.21.15.71:0/2577859390 >> v2:172.21.15.164:3300/0 conn(0x7ff99c0cb570 msgr2=0x7ff99c15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff99a59c640 1 --2- 172.21.15.71:0/2577859390 >> v2:172.21.15.164:3300/0 conn(0x7ff99c0cb570 0x7ff99c15cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff99a59c640 1 -- 172.21.15.71:0/2577859390 >> v1:172.21.15.71:6789/0 conn(0x7ff99c0c7f10 legacy=0x7ff99c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:49.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff99a59c640 1 -- 172.21.15.71:0/2577859390 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0b1530 con 0x7ff99c0cb190 2023-08-25T10:02:49.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.414+0000 7ff99959a640 1 -- 172.21.15.71:0/2577859390 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470852020 0 0) 0x7ff99c0b1af0 con 0x7ff99c0c7f10 2023-08-25T10:02:49.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.415+0000 7ff99a59c640 1 --2- 172.21.15.71:0/2577859390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb190 0x7ff99c0ca910 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff99c1536c0 tx=0x7ff98c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c217423201acfce server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:49.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.416+0000 7ff99959a640 1 -- 172.21.15.71:0/2577859390 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c030610 con 0x7ff99c0cb190 2023-08-25T10:02:49.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.416+0000 7ff99959a640 1 -- 172.21.15.71:0/2577859390 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c030770 con 0x7ff99c0cb190 2023-08-25T10:02:49.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.416+0000 7ff99959a640 1 -- 172.21.15.71:0/2577859390 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c030b00 con 0x7ff99c0cb190 2023-08-25T10:02:49.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.416+0000 7ff99b59e640 1 -- 172.21.15.71:0/2577859390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb190 msgr2=0x7ff99c0ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:49.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.416+0000 7ff99b59e640 1 --2- 172.21.15.71:0/2577859390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb190 0x7ff99c0ca910 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff99c1536c0 tx=0x7ff98c02b960 comp rx=0 tx=0).stop 2023-08-25T10:02:49.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.417+0000 7ff99b59e640 1 -- 172.21.15.71:0/2577859390 shutdown_connections 2023-08-25T10:02:49.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.417+0000 7ff99b59e640 1 --2- 172.21.15.71:0/2577859390 >> v2:172.21.15.164:3300/0 conn(0x7ff99c0cb570 0x7ff99c15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.417+0000 7ff99b59e640 1 --2- 172.21.15.71:0/2577859390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb190 0x7ff99c0ca910 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.417+0000 7ff99b59e640 1 -- 172.21.15.71:0/2577859390 >> 172.21.15.71:0/2577859390 conn(0x7ff99c06ae20 msgr2=0x7ff99c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:49.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.417+0000 7ff99b59e640 1 -- 172.21.15.71:0/2577859390 shutdown_connections 2023-08-25T10:02:49.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.417+0000 7ff99b59e640 1 -- 172.21.15.71:0/2577859390 wait complete. 2023-08-25T10:02:49.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.418+0000 7ff99b59e640 1 Processor -- start 2023-08-25T10:02:49.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.419+0000 7ff99b59e640 1 -- start start 2023-08-25T10:02:49.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.419+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99c0c7f10 0x7ff99c0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:49.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.419+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff99c0cb190 0x7ff99c14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:49.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.419+0000 7ff99b59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 0x7ff99c14c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99b59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1ef2a0 con 0x7ff99c0cb570 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99b59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1ee2b0 con 0x7ff99c0cb190 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c14eca0 con 0x7ff99c0c7f10 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 0x7ff99c14c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 0x7ff99c14c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38272/0 (socket says 172.21.15.71:38272) 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 -- 172.21.15.71:0/1428515918 learned_addr learned my addr 172.21.15.71:0/1428515918 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:49.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99a59c640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99c0c7f10 0x7ff99c0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff999d9b640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff99c0cb190 0x7ff99c14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 -- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99c0c7f10 msgr2=0x7ff99c0b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99c0c7f10 0x7ff99c0b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 -- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff99c0cb190 msgr2=0x7ff99c14c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff99c0cb190 0x7ff99c14c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99ad9d640 1 -- 172.21.15.71:0/1428515918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c009400 con 0x7ff99c0cb570 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff99a59c640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99c0c7f10 0x7ff99c0b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:49.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.420+0000 7ff999d9b640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff99c0cb190 0x7ff99c14c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:49.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.421+0000 7ff99ad9d640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 0x7ff99c14c8f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c7a0 tx=0x7ff99400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:49.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.421+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400a640 con 0x7ff99c0cb570 2023-08-25T10:02:49.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.421+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400a7a0 con 0x7ff99c0cb570 2023-08-25T10:02:49.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.421+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400abd0 con 0x7ff99c0cb570 2023-08-25T10:02:49.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.421+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c1f1930 con 0x7ff99c0cb570 2023-08-25T10:02:49.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.421+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c1ee7c0 con 0x7ff99c0cb570 2023-08-25T10:02:49.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.423+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c04e420 con 0x7ff99c0cb570 2023-08-25T10:02:49.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.423+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff99400ad30 con 0x7ff99c0cb570 2023-08-25T10:02:49.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.425+0000 7ff98affd640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff97c06a050 0x7ff97c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:49.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.425+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff9940880b0 con 0x7ff99c0cb570 2023-08-25T10:02:49.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.426+0000 7ff99a59c640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff97c06a050 0x7ff97c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:49.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.426+0000 7ff99a59c640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff97c06a050 0x7ff97c06c510 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036000 tx=0x7ff98c03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:49.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.430+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff994053240 con 0x7ff99c0cb570 2023-08-25T10:02:49.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.725+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff99c0b0d90 con 0x7ff99c0cb570 2023-08-25T10:02:49.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.726+0000 7ff98affd640 1 -- 172.21.15.71:0/1428515918 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7ff994018090 con 0x7ff99c0cb570 2023-08-25T10:02:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:49.729 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:49.729 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:49.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff97c06a050 msgr2=0x7ff97c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:49.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff97c06a050 0x7ff97c06c510 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036000 tx=0x7ff98c03b040 comp rx=0 tx=0).stop 2023-08-25T10:02:49.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 msgr2=0x7ff99c14c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:49.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 0x7ff99c14c8f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c7a0 tx=0x7ff99400cc60 comp rx=0 tx=0).stop 2023-08-25T10:02:49.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 shutdown_connections 2023-08-25T10:02:49.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff99c0cb190 0x7ff99c14c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff97c06a050 0x7ff97c06c510 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff99c0cb570 0x7ff99c14c8f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 --2- 172.21.15.71:0/1428515918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff99c0c7f10 0x7ff99c0b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:49.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 >> 172.21.15.71:0/1428515918 conn(0x7ff99c06ae20 msgr2=0x7ff99c0c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:49.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.729+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 shutdown_connections 2023-08-25T10:02:49.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:49.730+0000 7ff99b59e640 1 -- 172.21.15.71:0/1428515918 wait complete. 2023-08-25T10:02:49.753 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:51.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:52.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.011+0000 7f6ed6339640 1 Processor -- start 2023-08-25T10:02:52.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.011+0000 7f6ed6339640 1 -- start start 2023-08-25T10:02:52.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.013+0000 7f6ed6339640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ed0152d80 0x7f6ed006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.013+0000 7f6ed6339640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed006eb60 con 0x7f6ed0070cf0 2023-08-25T10:02:52.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.013+0000 7f6ed6339640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed006ca30 con 0x7f6ed00c4560 2023-08-25T10:02:52.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.013+0000 7f6ed6339640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed006cf90 con 0x7f6ed0152d80 2023-08-25T10:02:52.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.014+0000 7f6ed48d6640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6ed0070cf0 0x7f6ed01637b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33626/0 (socket says 172.21.15.71:33626) 2023-08-25T10:02:52.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.014+0000 7f6ed48d6640 1 -- 172.21.15.71:0/3379720526 learned_addr learned my addr 172.21.15.71:0/3379720526 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:52.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.014+0000 7f6ecffff640 1 --2- 172.21.15.71:0/3379720526 >> v2:172.21.15.164:3300/0 conn(0x7f6ed0152d80 0x7f6ed006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.014+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155581147 0 0) 0x7f6ed006eb60 con 0x7f6ed0070cf0 2023-08-25T10:02:52.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.014+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb8003520 con 0x7f6ed0070cf0 2023-08-25T10:02:52.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1050759581 0 0) 0x7f6eb8003520 con 0x7f6ed0070cf0 2023-08-25T10:02:52.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 >> v2:172.21.15.164:3300/0 conn(0x7f6ed0152d80 msgr2=0x7f6ed006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:52.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 --2- 172.21.15.71:0/3379720526 >> v2:172.21.15.164:3300/0 conn(0x7f6ed0152d80 0x7f6ed006e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 >> v1:172.21.15.164:6789/0 conn(0x7f6ed00c4560 legacy=0x7f6ed00c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0158fd0 con 0x7f6ed0070cf0 2023-08-25T10:02:52.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f6ec80019f0 con 0x7f6ed0070cf0 2023-08-25T10:02:52.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 >> v1:172.21.15.71:6789/0 conn(0x7f6ed0070cf0 legacy=0x7f6ed01637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80039e0 0x7f6eb8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6eb80047c0 0x7f6eb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6eb80051f0 0x7f6eb800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed006cf90 con 0x7f6eb80047c0 2023-08-25T10:02:52.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb800c150 con 0x7f6eb80039e0 2023-08-25T10:02:52.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb800c310 con 0x7f6eb80051f0 2023-08-25T10:02:52.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.015+0000 7f6ecf7fe640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6eb80047c0 0x7f6eb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ed48d6640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6eb80051f0 0x7f6eb800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ecffff640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80039e0 0x7f6eb8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ecf7fe640 1 -- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6eb80051f0 msgr2=0x7f6eb800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ecf7fe640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6eb80051f0 0x7f6eb800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ecf7fe640 1 -- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80039e0 msgr2=0x7f6eb8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ecf7fe640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80039e0 0x7f6eb8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ecf7fe640 1 -- 172.21.15.71:0/3379720526 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb8003520 con 0x7f6eb80047c0 2023-08-25T10:02:52.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.016+0000 7f6ed48d6640 1 -- 172.21.15.71:0/3379720526 reap_dead start 2023-08-25T10:02:52.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.017+0000 7f6ecf7fe640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6eb80047c0 0x7f6eb8004c40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4001750 tx=0x7f6eb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92703ec0c491488b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:52.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40052d0 con 0x7f6eb80047c0 2023-08-25T10:02:52.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec402e050 con 0x7f6eb80047c0 2023-08-25T10:02:52.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6eceffd640 1 -- 172.21.15.71:0/3379720526 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4003830 con 0x7f6eb80047c0 2023-08-25T10:02:52.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 -- 172.21.15.71:0/3379720526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6eb80047c0 msgr2=0x7f6eb8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6eb80047c0 0x7f6eb8004c40 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4001750 tx=0x7f6eb80057f0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 -- 172.21.15.71:0/3379720526 shutdown_connections 2023-08-25T10:02:52.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 --2- 172.21.15.71:0/3379720526 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6eb80047c0 0x7f6eb8004c40 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 -- 172.21.15.71:0/3379720526 >> 172.21.15.71:0/3379720526 conn(0x7f6ed00654f0 msgr2=0x7f6ed0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:52.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 -- 172.21.15.71:0/3379720526 shutdown_connections 2023-08-25T10:02:52.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.018+0000 7f6ed6339640 1 -- 172.21.15.71:0/3379720526 wait complete. 2023-08-25T10:02:52.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.020+0000 7f6ed6339640 1 Processor -- start 2023-08-25T10:02:52.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.021+0000 7f6ed6339640 1 -- start start 2023-08-25T10:02:52.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed6339640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80047c0 0x7f6ed00c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed6339640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ed01563a0 0x7f6ed014c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed6339640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 0x7f6ed01f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed6339640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed01567e0 con 0x7f6ed014cbd0 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed6339640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0155d70 con 0x7f6eb80047c0 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed6339640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0155180 con 0x7f6ed01563a0 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed48d6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 0x7f6ed01f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed48d6640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 0x7f6ed01f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54358/0 (socket says 172.21.15.71:54358) 2023-08-25T10:02:52.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ed48d6640 1 -- 172.21.15.71:0/4198278757 learned_addr learned my addr 172.21.15.71:0/4198278757 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:52.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ecf7fe640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ed01563a0 0x7f6ed014c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.022+0000 7f6ecffff640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80047c0 0x7f6ed00c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed48d6640 1 -- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ed01563a0 msgr2=0x7f6ed014c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed48d6640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ed01563a0 0x7f6ed014c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed48d6640 1 -- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80047c0 msgr2=0x7f6ed00c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed48d6640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80047c0 0x7f6ed00c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed48d6640 1 -- 172.21.15.71:0/4198278757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed01f27d0 con 0x7f6ed014cbd0 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ecffff640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80047c0 0x7f6ed00c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed48d6640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 0x7f6ed01f1c50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0071cc0 tx=0x7f6ec8007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec800ea20 con 0x7f6ed014cbd0 2023-08-25T10:02:52.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec800eb80 con 0x7f6ed014cbd0 2023-08-25T10:02:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed0155440 con 0x7f6ed014cbd0 2023-08-25T10:02:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.023+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8014480 con 0x7f6ed014cbd0 2023-08-25T10:02:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.024+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed0155a00 con 0x7f6ed014cbd0 2023-08-25T10:02:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.025+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed0155bc0 con 0x7f6ed014cbd0 2023-08-25T10:02:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.026+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6ec80145e0 con 0x7f6ed014cbd0 2023-08-25T10:02:52.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.027+0000 7f6eccff9640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ebc06a080 0x7f6ebc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:52.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.027+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6ec8088840 con 0x7f6ed014cbd0 2023-08-25T10:02:52.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.027+0000 7f6ecffff640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ebc06a080 0x7f6ebc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:52.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.028+0000 7f6ecffff640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ebc06a080 0x7f6ebc06c540 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009a80 tx=0x7f6ec0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:52.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.033+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6ec8053940 con 0x7f6ed014cbd0 2023-08-25T10:02:52.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.306+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ed006faa0 con 0x7f6ed014cbd0 2023-08-25T10:02:52.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.307+0000 7f6eccff9640 1 -- 172.21.15.71:0/4198278757 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f6ec8056f60 con 0x7f6ed014cbd0 2023-08-25T10:02:52.309 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:52.309 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:52.310 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:52.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ebc06a080 msgr2=0x7f6ebc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ebc06a080 0x7f6ebc06c540 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009a80 tx=0x7f6ec0008040 comp rx=0 tx=0).stop 2023-08-25T10:02:52.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 msgr2=0x7f6ed01f1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:52.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 0x7f6ed01f1c50 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0071cc0 tx=0x7f6ec8007bc0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 shutdown_connections 2023-08-25T10:02:52.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ebc06a080 0x7f6ebc06c540 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ed014cbd0 0x7f6ed01f1c50 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ed01563a0 0x7f6ed014c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 --2- 172.21.15.71:0/4198278757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6eb80047c0 0x7f6ed00c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:52.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 >> 172.21.15.71:0/4198278757 conn(0x7f6ed00654f0 msgr2=0x7f6ed006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:52.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 shutdown_connections 2023-08-25T10:02:52.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:52.310+0000 7f6ed6339640 1 -- 172.21.15.71:0/4198278757 wait complete. 2023-08-25T10:02:52.331 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:54.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:54.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.568+0000 7fcfc359e640 1 Processor -- start 2023-08-25T10:02:54.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.568+0000 7fcfc359e640 1 -- start start 2023-08-25T10:02:54.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcfc4158d60 0x7fcfc40cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:54.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc40cad40 0x7fcfc40c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:54.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc359e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc415c6c0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc415d260 con 0x7fcfc4158d60 2023-08-25T10:02:54.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41592c0 con 0x7fcfc40cad40 2023-08-25T10:02:54.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc259c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcfc4152fc0 0x7fcfc415cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33630/0 (socket says 172.21.15.71:33630) 2023-08-25T10:02:54.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.570+0000 7fcfc259c640 1 -- 172.21.15.71:0/2132919183 learned_addr learned my addr 172.21.15.71:0/2132919183 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc1d9b640 1 --2- 172.21.15.71:0/2132919183 >> v2:172.21.15.71:3300/0 conn(0x7fcfc4158d60 0x7fcfc40cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc2d9d640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc40cad40 0x7fcfc40c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc159a640 1 -- 172.21.15.71:0/2132919183 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965987342 0 0) 0x7fcfc415c6c0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc159a640 1 -- 172.21.15.71:0/2132919183 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfac003520 con 0x7fcfc4152fc0 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/2132919183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc40cad40 msgr2=0x7fcfc40c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc1d9b640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc40cad40 0x7fcfc40c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/2132919183 >> v1:172.21.15.71:6789/0 conn(0x7fcfc4152fc0 legacy=0x7fcfc415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:54.574 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/2132919183 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc41595a0 con 0x7fcfc4158d60 2023-08-25T10:02:54.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.571+0000 7fcfc2d9d640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc40cad40 0x7fcfc40c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:54.575 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.573+0000 7fcfc1d9b640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4158d60 0x7fcfc40cbc60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80097a0 tx=0x7fcfb802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759fda244cd77cf3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:54.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.573+0000 7fcfc159a640 1 -- 172.21.15.71:0/2132919183 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8030800 con 0x7fcfc4158d60 2023-08-25T10:02:54.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.573+0000 7fcfc159a640 1 -- 172.21.15.71:0/2132919183 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8030960 con 0x7fcfc4158d60 2023-08-25T10:02:54.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.573+0000 7fcfc159a640 1 -- 172.21.15.71:0/2132919183 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8030cf0 con 0x7fcfc4158d60 2023-08-25T10:02:54.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.573+0000 7fcfc359e640 1 -- 172.21.15.71:0/2132919183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4158d60 msgr2=0x7fcfc40cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:54.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.573+0000 7fcfc359e640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4158d60 0x7fcfc40cbc60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80097a0 tx=0x7fcfb802b940 comp rx=0 tx=0).stop 2023-08-25T10:02:54.576 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.574+0000 7fcfc359e640 1 -- 172.21.15.71:0/2132919183 shutdown_connections 2023-08-25T10:02:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.574+0000 7fcfc359e640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc40cad40 0x7fcfc40c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.574+0000 7fcfc359e640 1 --2- 172.21.15.71:0/2132919183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4158d60 0x7fcfc40cbc60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.574+0000 7fcfc359e640 1 -- 172.21.15.71:0/2132919183 >> 172.21.15.71:0/2132919183 conn(0x7fcfc406a3a0 msgr2=0x7fcfc4070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.574+0000 7fcfc359e640 1 -- 172.21.15.71:0/2132919183 shutdown_connections 2023-08-25T10:02:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.574+0000 7fcfc359e640 1 -- 172.21.15.71:0/2132919183 wait complete. 2023-08-25T10:02:54.577 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.576+0000 7fcfc359e640 1 Processor -- start 2023-08-25T10:02:54.578 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.576+0000 7fcfc359e640 1 -- start start 2023-08-25T10:02:54.580 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfc40cad40 0x7fcfc41f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 0x7fcfc41f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc4158d60 0x7fcfc4146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41f03b0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc40ccfe0 con 0x7fcfc40cad40 2023-08-25T10:02:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc41ef450 con 0x7fcfc4158d60 2023-08-25T10:02:54.581 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc1d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 0x7fcfc41f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:54.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc1d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 0x7fcfc41f0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54376/0 (socket says 172.21.15.71:54376) 2023-08-25T10:02:54.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.577+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/375860099 learned_addr learned my addr 172.21.15.71:0/375860099 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:54.582 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc259c640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfc40cad40 0x7fcfc41f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc2d9d640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc4158d60 0x7fcfc4146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc4158d60 msgr2=0x7fcfc4146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc1d9b640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc4158d60 0x7fcfc4146fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfc40cad40 msgr2=0x7fcfc41f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc1d9b640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfc40cad40 0x7fcfc41f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.583 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc1d9b640 1 -- 172.21.15.71:0/375860099 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb8009400 con 0x7fcfc4152fc0 2023-08-25T10:02:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfc1d9b640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 0x7fcfc41f0d10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80098c0 tx=0x7fcfb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.578+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb803e070 con 0x7fcfc4152fc0 2023-08-25T10:02:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.579+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8039a10 con 0x7fcfc4152fc0 2023-08-25T10:02:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.579+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8039dc0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.579+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc41ef730 con 0x7fcfc4152fc0 2023-08-25T10:02:54.584 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.579+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc41efcf0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.580+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc404e420 con 0x7fcfc4152fc0 2023-08-25T10:02:54.585 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.581+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcfb80423f0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.582+0000 7fcfb2ffd640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcfa406a080 0x7fcfa406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.582+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fcfb80aa5b0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.583+0000 7fcfc259c640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcfa406a080 0x7fcfa406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:54.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.583+0000 7fcfc259c640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcfa406a080 0x7fcfa406c540 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40011e0 tx=0x7fcfb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:54.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.589+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fcfb8075750 con 0x7fcfc4152fc0 2023-08-25T10:02:54.875 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.873+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfc404dbd0 con 0x7fcfc4152fc0 2023-08-25T10:02:54.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.874+0000 7fcfb2ffd640 1 -- 172.21.15.71:0/375860099 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fcfb8034090 con 0x7fcfc4152fc0 2023-08-25T10:02:54.876 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:54.877 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:54.877 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:54.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcfa406a080 msgr2=0x7fcfa406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:54.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcfa406a080 0x7fcfa406c540 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40011e0 tx=0x7fcfb4009040 comp rx=0 tx=0).stop 2023-08-25T10:02:54.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 msgr2=0x7fcfc41f0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:54.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 0x7fcfc41f0d10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcfb80098c0 tx=0x7fcfb80046c0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 shutdown_connections 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcfa406a080 0x7fcfa406c540 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcfc4158d60 0x7fcfc4146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcfc4152fc0 0x7fcfc41f0d10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 --2- 172.21.15.71:0/375860099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcfc40cad40 0x7fcfc41f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 >> 172.21.15.71:0/375860099 conn(0x7fcfc406a3a0 msgr2=0x7fcfc4070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 shutdown_connections 2023-08-25T10:02:54.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:54.877+0000 7fcfc359e640 1 -- 172.21.15.71:0/375860099 wait complete. 2023-08-25T10:02:54.897 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:02:56.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:02:56.347 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:02:56.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:57.130 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.129+0000 7f064d116640 1 Processor -- start 2023-08-25T10:02:57.131 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.129+0000 7f064d116640 1 -- start start 2023-08-25T10:02:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f064d116640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f06481542d0 0x7f06481576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f064d116640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06480c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f064d116640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0648157bf0 con 0x7f0648069850 2023-08-25T10:02:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f064d116640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06480c4b60 con 0x7f06481542d0 2023-08-25T10:02:57.133 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f064d116640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064806a130 con 0x7f06480c4fb0 2023-08-25T10:02:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f0647fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f06481542d0 0x7f06481576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f0647fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f06481542d0 0x7f06481576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54380/0 (socket says 172.21.15.71:54380) 2023-08-25T10:02:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.131+0000 7f0647fff640 1 -- 172.21.15.71:0/1684755110 learned_addr learned my addr 172.21.15.71:0/1684755110 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:57.134 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.132+0000 7f064c915640 1 --2- 172.21.15.71:0/1684755110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06480c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.132+0000 7f0647fff640 1 -- 172.21.15.71:0/1684755110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 msgr2=0x7f06480c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.132+0000 7f0647fff640 1 --2- 172.21.15.71:0/1684755110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06480c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.132+0000 7f0647fff640 1 -- 172.21.15.71:0/1684755110 >> v1:172.21.15.164:6789/0 conn(0x7f0648069850 legacy=0x7f06480c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:57.135 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.132+0000 7f0647fff640 1 -- 172.21.15.71:0/1684755110 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06480c5980 con 0x7f06481542d0 2023-08-25T10:02:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.133+0000 7f0647fff640 1 --2- 172.21.15.71:0/1684755110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481576b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0638009870 tx=0x7f063802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363075c17644c951 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.134+0000 7f0646ffd640 1 -- 172.21.15.71:0/1684755110 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638030840 con 0x7f06481542d0 2023-08-25T10:02:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.134+0000 7f0646ffd640 1 -- 172.21.15.71:0/1684755110 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06380309a0 con 0x7f06481542d0 2023-08-25T10:02:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.134+0000 7f0646ffd640 1 -- 172.21.15.71:0/1684755110 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638030d30 con 0x7f06481542d0 2023-08-25T10:02:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.134+0000 7f064d116640 1 -- 172.21.15.71:0/1684755110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 msgr2=0x7f06481576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.134+0000 7f064d116640 1 --2- 172.21.15.71:0/1684755110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481576b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0638009870 tx=0x7f063802b980 comp rx=0 tx=0).stop 2023-08-25T10:02:57.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.135+0000 7f064d116640 1 -- 172.21.15.71:0/1684755110 shutdown_connections 2023-08-25T10:02:57.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.135+0000 7f064d116640 1 --2- 172.21.15.71:0/1684755110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06480c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.135+0000 7f064d116640 1 --2- 172.21.15.71:0/1684755110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481576b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.135+0000 7f064d116640 1 -- 172.21.15.71:0/1684755110 >> 172.21.15.71:0/1684755110 conn(0x7f064806c280 msgr2=0x7f064806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:57.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.135+0000 7f064d116640 1 -- 172.21.15.71:0/1684755110 shutdown_connections 2023-08-25T10:02:57.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.135+0000 7f064d116640 1 -- 172.21.15.71:0/1684755110 wait complete. 2023-08-25T10:02:57.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.137+0000 7f064d116640 1 Processor -- start 2023-08-25T10:02:57.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.137+0000 7f064d116640 1 -- start start 2023-08-25T10:02:57.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.138+0000 7f064d116640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0648069850 0x7f064815a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:57.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.138+0000 7f064d116640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06481581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:57.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.138+0000 7f064d116640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:57.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.138+0000 7f064d116640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064815a7e0 con 0x7f06481542d0 2023-08-25T10:02:57.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.138+0000 7f064d116640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0648145f10 con 0x7f0648069850 2023-08-25T10:02:57.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.138+0000 7f064d116640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06480c9b30 con 0x7f06480c4fb0 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54394/0 (socket says 172.21.15.71:54394) 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 -- 172.21.15.71:0/381493424 learned_addr learned my addr 172.21.15.71:0/381493424 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f06477fe640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06481581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f0647fff640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0648069850 0x7f064815a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 -- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 msgr2=0x7f06481581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:57.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06481581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 -- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0648069850 msgr2=0x7f064815a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:57.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0648069850 0x7f064815a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 -- 172.21.15.71:0/381493424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380094d0 con 0x7f06481542d0 2023-08-25T10:02:57.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.139+0000 7f064c915640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481586f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0640007e40 tx=0x7f064000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:57.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.140+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640014620 con 0x7f06481542d0 2023-08-25T10:02:57.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.140+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06400147a0 con 0x7f06481542d0 2023-08-25T10:02:57.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.140+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06480c9dd0 con 0x7f06481542d0 2023-08-25T10:02:57.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.140+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640014b50 con 0x7f06481542d0 2023-08-25T10:02:57.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.140+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0648146430 con 0x7f06481542d0 2023-08-25T10:02:57.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.141+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0648157ed0 con 0x7f06481542d0 2023-08-25T10:02:57.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.143+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0640014cb0 con 0x7f06481542d0 2023-08-25T10:02:57.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.144+0000 7f0644ff9640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f063406a050 0x7f063406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:57.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.144+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f06400893e0 con 0x7f06481542d0 2023-08-25T10:02:57.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.144+0000 7f0647fff640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f063406a050 0x7f063406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:57.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.145+0000 7f0647fff640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f063406a050 0x7f063406c510 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0638036000 tx=0x7f063803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:57.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.149+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0648157ed0 con 0x7f06481542d0 2023-08-25T10:02:57.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.419+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0648157ed0 con 0x7f06481542d0 2023-08-25T10:02:57.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.420+0000 7f0644ff9640 1 -- 172.21.15.71:0/381493424 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f0648157ed0 con 0x7f06481542d0 2023-08-25T10:02:57.422 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:57.422 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:57.422 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:57.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f063406a050 msgr2=0x7f063406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:57.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f063406a050 0x7f063406c510 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0638036000 tx=0x7f063803b040 comp rx=0 tx=0).stop 2023-08-25T10:02:57.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 msgr2=0x7f06481586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:57.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481586f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0640007e40 tx=0x7f064000e4b0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 shutdown_connections 2023-08-25T10:02:57.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f063406a050 0x7f063406c510 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f06481542d0 0x7f06481586f0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06480c4fb0 0x7f06481581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 --2- 172.21.15.71:0/381493424 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0648069850 0x7f064815a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:57.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 >> 172.21.15.71:0/381493424 conn(0x7f064806c280 msgr2=0x7f0648151480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:57.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 shutdown_connections 2023-08-25T10:02:57.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:57.423+0000 7f064d116640 1 -- 172.21.15.71:0/381493424 wait complete. 2023-08-25T10:02:57.444 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:02:57.445 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:02:59.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:02:59.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.689+0000 7f2d1980e640 1 Processor -- start 2023-08-25T10:02:59.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.689+0000 7f2d1980e640 1 -- start start 2023-08-25T10:02:59.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1980e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140cb190 0x7f2d140ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1980e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2d140cb570 0x7f2d1415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1980e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2d140c7f10 0x7f2d140c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1980e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d140c98f0 con 0x7f2d140cb570 2023-08-25T10:02:59.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1980e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d140b1af0 con 0x7f2d140c7f10 2023-08-25T10:02:59.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1980e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140b0fb0 con 0x7f2d140cb190 2023-08-25T10:02:59.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d13fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2d140cb570 0x7f2d1415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d13fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2d140cb570 0x7f2d1415cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54410/0 (socket says 172.21.15.71:54410) 2023-08-25T10:02:59.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d13fff640 1 -- 172.21.15.71:0/454568615 learned_addr learned my addr 172.21.15.71:0/454568615 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:59.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1880c640 1 --2- 172.21.15.71:0/454568615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140cb190 0x7f2d140ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.691+0000 7f2d1900d640 1 --2- 172.21.15.71:0/454568615 >> v2:172.21.15.164:3300/0 conn(0x7f2d140c7f10 0x7f2d140c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.692+0000 7f2d13fff640 1 -- 172.21.15.71:0/454568615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140cb190 msgr2=0x7f2d140ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:59.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.692+0000 7f2d13fff640 1 --2- 172.21.15.71:0/454568615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140cb190 0x7f2d140ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.692+0000 7f2d13fff640 1 -- 172.21.15.71:0/454568615 >> v2:172.21.15.164:3300/0 conn(0x7f2d140c7f10 msgr2=0x7f2d140c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:02:59.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.692+0000 7f2d13fff640 1 --2- 172.21.15.71:0/454568615 >> v2:172.21.15.164:3300/0 conn(0x7f2d140c7f10 0x7f2d140c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.692+0000 7f2d13fff640 1 -- 172.21.15.71:0/454568615 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d140b1530 con 0x7f2d140cb570 2023-08-25T10:02:59.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.693+0000 7f2d13fff640 1 --2- 172.21.15.71:0/454568615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d1415cd00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2d140cae50 tx=0x7f2d0002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c8aaf4d37d1db7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:59.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.693+0000 7f2d137fe640 1 -- 172.21.15.71:0/454568615 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00030860 con 0x7f2d140cb570 2023-08-25T10:02:59.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.693+0000 7f2d137fe640 1 -- 172.21.15.71:0/454568615 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d000309c0 con 0x7f2d140cb570 2023-08-25T10:02:59.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.693+0000 7f2d137fe640 1 -- 172.21.15.71:0/454568615 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00030d50 con 0x7f2d140cb570 2023-08-25T10:02:59.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 -- 172.21.15.71:0/454568615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 msgr2=0x7f2d1415cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:59.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 --2- 172.21.15.71:0/454568615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d1415cd00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f2d140cae50 tx=0x7f2d0002b9a0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 -- 172.21.15.71:0/454568615 shutdown_connections 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 --2- 172.21.15.71:0/454568615 >> v2:172.21.15.164:3300/0 conn(0x7f2d140c7f10 0x7f2d140c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 --2- 172.21.15.71:0/454568615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d1415cd00 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 --2- 172.21.15.71:0/454568615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140cb190 0x7f2d140ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 -- 172.21.15.71:0/454568615 >> 172.21.15.71:0/454568615 conn(0x7f2d1406ae20 msgr2=0x7f2d1406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 -- 172.21.15.71:0/454568615 shutdown_connections 2023-08-25T10:02:59.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.694+0000 7f2d1980e640 1 -- 172.21.15.71:0/454568615 wait complete. 2023-08-25T10:02:59.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.696+0000 7f2d1980e640 1 Processor -- start 2023-08-25T10:02:59.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.696+0000 7f2d1980e640 1 -- start start 2023-08-25T10:02:59.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1980e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140c7f10 0x7f2d1414bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1980e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d140cb190 0x7f2d140cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1980e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d14159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1980e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d141ee4a0 con 0x7f2d140cb570 2023-08-25T10:02:59.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1980e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140ce7d0 con 0x7f2d140cb190 2023-08-25T10:02:59.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1980e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d140ceab0 con 0x7f2d140c7f10 2023-08-25T10:02:59.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1900d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d14159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1900d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d14159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:54412/0 (socket says 172.21.15.71:54412) 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1900d640 1 -- 172.21.15.71:0/2760835845 learned_addr learned my addr 172.21.15.71:0/2760835845 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.697+0000 7f2d1880c640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140c7f10 0x7f2d1414bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d13fff640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d140cb190 0x7f2d140cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d1900d640 1 -- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140c7f10 msgr2=0x7f2d1414bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d1900d640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140c7f10 0x7f2d1414bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d1900d640 1 -- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d140cb190 msgr2=0x7f2d140cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:59.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d1900d640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d140cb190 0x7f2d140cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d1900d640 1 -- 172.21.15.71:0/2760835845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d000094d0 con 0x7f2d140cb570 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d13fff640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d140cb190 0x7f2d140cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d1900d640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d14159140 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c007e60 tx=0x7f2d0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.698+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c014420 con 0x7f2d140cb570 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.699+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0c0145a0 con 0x7f2d140cb570 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.699+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c014950 con 0x7f2d140cb570 2023-08-25T10:02:59.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.699+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d140ced90 con 0x7f2d140cb570 2023-08-25T10:02:59.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.699+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1414c1d0 con 0x7f2d140cb570 2023-08-25T10:02:59.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.700+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d140cf2b0 con 0x7f2d140cb570 2023-08-25T10:02:59.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.701+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f2d0c014ab0 con 0x7f2d140cb570 2023-08-25T10:02:59.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.702+0000 7f2d117fa640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2cf806a050 0x7f2cf806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:02:59.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.702+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f2d0c089390 con 0x7f2d140cb570 2023-08-25T10:02:59.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.703+0000 7f2d1880c640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2cf806a050 0x7f2cf806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:02:59.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.703+0000 7f2d1880c640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2cf806a050 0x7f2cf806c510 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2d08006fd0 tx=0x7f2d08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:02:59.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.707+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f2d140cf2b0 con 0x7f2d140cb570 2023-08-25T10:02:59.979 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.977+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d140cf2b0 con 0x7f2d140cb570 2023-08-25T10:02:59.980 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.978+0000 7f2d117fa640 1 -- 172.21.15.71:0/2760835845 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f2d140cf2b0 con 0x7f2d140cb570 2023-08-25T10:02:59.980 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:02:59.980 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:02:59.981 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:02:59.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2cf806a050 msgr2=0x7f2cf806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:59.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2cf806a050 0x7f2cf806c510 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2d08006fd0 tx=0x7f2d08008040 comp rx=0 tx=0).stop 2023-08-25T10:02:59.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 msgr2=0x7f2d14159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:02:59.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d14159140 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c007e60 tx=0x7f2d0c00e4b0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 shutdown_connections 2023-08-25T10:02:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d140cb190 0x7f2d140cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f2cf806a050 0x7f2cf806c510 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d140cb570 0x7f2d14159140 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 --2- 172.21.15.71:0/2760835845 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d140c7f10 0x7f2d1414bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:02:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 >> 172.21.15.71:0/2760835845 conn(0x7f2d1406ae20 msgr2=0x7f2d140ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:02:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 shutdown_connections 2023-08-25T10:02:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:02:59.981+0000 7f2d1980e640 1 -- 172.21.15.71:0/2760835845 wait complete. 2023-08-25T10:03:00.002 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:02.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.238+0000 7f0251671640 1 Processor -- start 2023-08-25T10:03:02.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.238+0000 7f0251671640 1 -- start start 2023-08-25T10:03:02.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0251671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0cb190 0x7f024c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:02.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0251671640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f024c0cb570 0x7f024c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:02.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0251671640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0b1af0 con 0x7f024c0c7f10 2023-08-25T10:03:02.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0251671640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024c0c98f0 con 0x7f024c0cb570 2023-08-25T10:03:02.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0251671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0b0fb0 con 0x7f024c0cb190 2023-08-25T10:03:02.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0250e70640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f024c0c7f10 0x7f024c158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42258/0 (socket says 172.21.15.71:42258) 2023-08-25T10:03:02.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f0250e70640 1 -- 172.21.15.71:0/2610664155 learned_addr learned my addr 172.21.15.71:0/2610664155 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:02.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.240+0000 7f024b7fe640 1 --2- 172.21.15.71:0/2610664155 >> v2:172.21.15.71:3300/0 conn(0x7f024c0cb570 0x7f024c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:02.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024bfff640 1 --2- 172.21.15.71:0/2610664155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0cb190 0x7f024c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:02.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024affd640 1 -- 172.21.15.71:0/2610664155 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793970159 0 0) 0x7f024c0b1af0 con 0x7f024c0c7f10 2023-08-25T10:03:02.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024affd640 1 -- 172.21.15.71:0/2610664155 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0234003520 con 0x7f024c0c7f10 2023-08-25T10:03:02.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024b7fe640 1 -- 172.21.15.71:0/2610664155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0cb190 msgr2=0x7f024c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:02.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024b7fe640 1 --2- 172.21.15.71:0/2610664155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0cb190 0x7f024c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024b7fe640 1 -- 172.21.15.71:0/2610664155 >> v1:172.21.15.71:6789/0 conn(0x7f024c0c7f10 legacy=0x7f024c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:02.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.241+0000 7f024b7fe640 1 -- 172.21.15.71:0/2610664155 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0b1530 con 0x7f024c0cb570 2023-08-25T10:03:02.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.242+0000 7f024b7fe640 1 --2- 172.21.15.71:0/2610664155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb570 0x7f024c15cd00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f02400097a0 tx=0x7f024002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77d603bf27ff2df server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:02.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.243+0000 7f024affd640 1 -- 172.21.15.71:0/2610664155 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240030800 con 0x7f024c0cb570 2023-08-25T10:03:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.243+0000 7f024affd640 1 -- 172.21.15.71:0/2610664155 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240030960 con 0x7f024c0cb570 2023-08-25T10:03:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.243+0000 7f024affd640 1 -- 172.21.15.71:0/2610664155 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240030cf0 con 0x7f024c0cb570 2023-08-25T10:03:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.243+0000 7f0251671640 1 -- 172.21.15.71:0/2610664155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb570 msgr2=0x7f024c15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.243+0000 7f0251671640 1 --2- 172.21.15.71:0/2610664155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb570 0x7f024c15cd00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f02400097a0 tx=0x7f024002b940 comp rx=0 tx=0).stop 2023-08-25T10:03:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.244+0000 7f0251671640 1 -- 172.21.15.71:0/2610664155 shutdown_connections 2023-08-25T10:03:02.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.244+0000 7f0251671640 1 --2- 172.21.15.71:0/2610664155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb570 0x7f024c15cd00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.244+0000 7f0251671640 1 --2- 172.21.15.71:0/2610664155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0cb190 0x7f024c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.244+0000 7f0251671640 1 -- 172.21.15.71:0/2610664155 >> 172.21.15.71:0/2610664155 conn(0x7f024c06ae20 msgr2=0x7f024c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.244+0000 7f0251671640 1 -- 172.21.15.71:0/2610664155 shutdown_connections 2023-08-25T10:03:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.244+0000 7f0251671640 1 -- 172.21.15.71:0/2610664155 wait complete. 2023-08-25T10:03:02.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.245+0000 7f0251671640 1 Processor -- start 2023-08-25T10:03:02.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.246+0000 7f0251671640 1 -- start start 2023-08-25T10:03:02.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.246+0000 7f0251671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0c7f10 0x7f024c153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f0251671640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 0x7f024c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f0251671640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f024c0cb570 0x7f024c1561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f0251671640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c154850 con 0x7f024c0cb190 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f0251671640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c151eb0 con 0x7f024c0cb570 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f0251671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c155780 con 0x7f024c0c7f10 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 0x7f024c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:02.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 0x7f024c155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51046/0 (socket says 172.21.15.71:51046) 2023-08-25T10:03:02.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 -- 172.21.15.71:0/3538450332 learned_addr learned my addr 172.21.15.71:0/3538450332 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024bfff640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0c7f10 0x7f024c153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f0250e70640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f024c0cb570 0x7f024c1561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 -- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0c7f10 msgr2=0x7f024c153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0c7f10 0x7f024c153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 -- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f024c0cb570 msgr2=0x7f024c1561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f024c0cb570 0x7f024c1561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.247+0000 7f024b7fe640 1 -- 172.21.15.71:0/3538450332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240009400 con 0x7f024c0cb190 2023-08-25T10:03:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.248+0000 7f024b7fe640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 0x7f024c155c60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f024002bf50 tx=0x7f02400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.248+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024003e070 con 0x7f024c0cb190 2023-08-25T10:03:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.248+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240006e10 con 0x7f024c0cb190 2023-08-25T10:03:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.248+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240039c30 con 0x7f024c0cb190 2023-08-25T10:03:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.248+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c152100 con 0x7f024c0cb190 2023-08-25T10:03:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.249+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c1ee970 con 0x7f024c0cb190 2023-08-25T10:03:02.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.250+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c1ee7d0 con 0x7f024c0cb190 2023-08-25T10:03:02.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.251+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f024004b400 con 0x7f024c0cb190 2023-08-25T10:03:02.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.252+0000 7f0248ff9640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:02.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.252+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f02400aae90 con 0x7f024c0cb190 2023-08-25T10:03:02.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.252+0000 7f024bfff640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:02.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.253+0000 7f024bfff640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f023c009ad0 tx=0x7f023c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:02.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.257+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f024c1ee7d0 con 0x7f024c0cb190 2023-08-25T10:03:02.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.527+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f024c1ee7d0 con 0x7f024c0cb190 2023-08-25T10:03:02.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.528+0000 7f0248ff9640 1 -- 172.21.15.71:0/3538450332 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f024c1ee7d0 con 0x7f024c0cb190 2023-08-25T10:03:02.530 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:02.530 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:02.531 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:02.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 msgr2=0x7f023806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:02.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f023c009ad0 tx=0x7f023c008040 comp rx=0 tx=0).stop 2023-08-25T10:03:02.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 msgr2=0x7f024c155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:02.532 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 0x7f024c155c60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f024002bf50 tx=0x7f02400046c0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.533 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 shutdown_connections 2023-08-25T10:03:02.533 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f024c0cb570 0x7f024c1561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.533 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f023806a050 0x7f023806c510 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.533 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f024c0cb190 0x7f024c155c60 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.533 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 --2- 172.21.15.71:0/3538450332 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f024c0c7f10 0x7f024c153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:02.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 >> 172.21.15.71:0/3538450332 conn(0x7f024c06ae20 msgr2=0x7f024c15b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:02.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 shutdown_connections 2023-08-25T10:03:02.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:02.530+0000 7f0251671640 1 -- 172.21.15.71:0/3538450332 wait complete. 2023-08-25T10:03:02.550 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:04.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:04.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.785+0000 7f6cbffff640 1 Processor -- start 2023-08-25T10:03:04.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.785+0000 7f6cbffff640 1 -- start start 2023-08-25T10:03:04.788 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc00ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:04.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cc015cd00 0x7f6cc0158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:04.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbffff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc00b1af0 con 0x7f6cc00cb570 2023-08-25T10:03:04.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbffff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc00c98f0 con 0x7f6cc015cd00 2023-08-25T10:03:04.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc015d190 con 0x7f6cc00cb190 2023-08-25T10:03:04.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbf7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cc015cd00 0x7f6cc0158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:04.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbeffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc00ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:04.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbf7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cc015cd00 0x7f6cc0158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:36414/0 (socket says 172.21.15.71:36414) 2023-08-25T10:03:04.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.787+0000 7f6cbf7fe640 1 -- 172.21.15.71:0/2440046984 learned_addr learned my addr 172.21.15.71:0/2440046984 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:04.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.788+0000 7f6cbf7fe640 1 -- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 msgr2=0x7f6cc00ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:04.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.788+0000 7f6cbf7fe640 1 --2- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc00ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:04.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.788+0000 7f6cbf7fe640 1 -- 172.21.15.71:0/2440046984 >> v1:172.21.15.164:6789/0 conn(0x7f6cc00cb570 legacy=0x7f6cc00c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:04.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.788+0000 7f6cbf7fe640 1 -- 172.21.15.71:0/2440046984 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc00b0fb0 con 0x7f6cc015cd00 2023-08-25T10:03:04.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.790+0000 7f6cbf7fe640 1 --2- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0158d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009e50 tx=0x7f6cb802bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cde7e5a72f99765 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:04.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.790+0000 7f6cbe7fc640 1 -- 172.21.15.71:0/2440046984 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb802cb90 con 0x7f6cc015cd00 2023-08-25T10:03:04.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.790+0000 7f6cbe7fc640 1 -- 172.21.15.71:0/2440046984 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb802ccf0 con 0x7f6cc015cd00 2023-08-25T10:03:04.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.790+0000 7f6cbe7fc640 1 -- 172.21.15.71:0/2440046984 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8034610 con 0x7f6cc015cd00 2023-08-25T10:03:04.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 -- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 msgr2=0x7f6cc0158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:04.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 --2- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0158d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009e50 tx=0x7f6cb802bff0 comp rx=0 tx=0).stop 2023-08-25T10:03:04.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 -- 172.21.15.71:0/2440046984 shutdown_connections 2023-08-25T10:03:04.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 --2- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0158d80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:04.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 --2- 172.21.15.71:0/2440046984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc00ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:04.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 -- 172.21.15.71:0/2440046984 >> 172.21.15.71:0/2440046984 conn(0x7f6cc006ae20 msgr2=0x7f6cc006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:04.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 -- 172.21.15.71:0/2440046984 shutdown_connections 2023-08-25T10:03:04.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.791+0000 7f6cbffff640 1 -- 172.21.15.71:0/2440046984 wait complete. 2023-08-25T10:03:04.794 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.793+0000 7f6cbffff640 1 Processor -- start 2023-08-25T10:03:04.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.793+0000 7f6cbffff640 1 -- start start 2023-08-25T10:03:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cbffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cbffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 0x7f6cc01ef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cbffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cbffff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc014c360 con 0x7f6cc00cb570 2023-08-25T10:03:04.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cbffff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc014a6c0 con 0x7f6cc015cd00 2023-08-25T10:03:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cbffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00cf2d0 con 0x7f6cc00cb190 2023-08-25T10:03:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cb7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 0x7f6cc01ef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cb7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 0x7f6cc01ef3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51054/0 (socket says 172.21.15.71:51054) 2023-08-25T10:03:04.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.794+0000 7f6cb7fff640 1 -- 172.21.15.71:0/3160014703 learned_addr learned my addr 172.21.15.71:0/3160014703 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:04.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cbf7fe640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:04.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cbeffd640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cb7fff640 1 -- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 msgr2=0x7f6cc014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cb7fff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cb7fff640 1 -- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 msgr2=0x7f6cc0148650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cb7fff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0148650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cb7fff640 1 -- 172.21.15.71:0/3160014703 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb8009b30 con 0x7f6cc00cb570 2023-08-25T10:03:04.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cb7fff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 0x7f6cc01ef3d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c450 tx=0x7f6ca800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.795+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80052d0 con 0x7f6cc00cb570 2023-08-25T10:03:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.796+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca8005430 con 0x7f6cc00cb570 2023-08-25T10:03:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.796+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc014bda0 con 0x7f6cc00cb570 2023-08-25T10:03:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.796+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8012870 con 0x7f6cc00cb570 2023-08-25T10:03:04.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.796+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc0149ec0 con 0x7f6cc00cb570 2023-08-25T10:03:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.797+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc004e420 con 0x7f6cc00cb570 2023-08-25T10:03:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.798+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6ca80129d0 con 0x7f6cc00cb570 2023-08-25T10:03:04.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.799+0000 7f6cbcff9640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ca006a050 0x7f6ca006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.799+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6ca80885b0 con 0x7f6cc00cb570 2023-08-25T10:03:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.799+0000 7f6cbeffd640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ca006a050 0x7f6ca006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:04.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.800+0000 7f6cbeffd640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ca006a050 0x7f6ca006c510 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009a30 tx=0x7f6cb0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:04.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:04.804+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6ca80537d0 con 0x7f6cc00cb570 2023-08-25T10:03:05.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.083+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cc00c7bf0 con 0x7f6cc00cb570 2023-08-25T10:03:05.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.084+0000 7f6cbcff9640 1 -- 172.21.15.71:0/3160014703 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f6ca8056df0 con 0x7f6cc00cb570 2023-08-25T10:03:05.086 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:05.086 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:05.087 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ca006a050 msgr2=0x7f6ca006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ca006a050 0x7f6ca006c510 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009a30 tx=0x7f6cb0008040 comp rx=0 tx=0).stop 2023-08-25T10:03:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 msgr2=0x7f6cc01ef3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 0x7f6cc01ef3d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c450 tx=0x7f6ca800c910 comp rx=0 tx=0).stop 2023-08-25T10:03:05.089 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 shutdown_connections 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc015cd00 0x7f6cc0148650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6ca006a050 0x7f6ca006c510 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6cc00cb570 0x7f6cc01ef3d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 --2- 172.21.15.71:0/3160014703 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc00cb190 0x7f6cc014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 >> 172.21.15.71:0/3160014703 conn(0x7f6cc006ae20 msgr2=0x7f6cc00c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 shutdown_connections 2023-08-25T10:03:05.090 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:05.087+0000 7f6cbffff640 1 -- 172.21.15.71:0/3160014703 wait complete. 2023-08-25T10:03:05.108 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:07.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:07.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.344+0000 7f1a00d6d640 1 Processor -- start 2023-08-25T10:03:07.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.344+0000 7f1a00d6d640 1 -- start start 2023-08-25T10:03:07.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f1a00d6d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f19fc0c76d0 0x7f19fc0bf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:07.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f1a00d6d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f19fc0adbb0 0x7f19fc0ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:07.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f1a00d6d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19fc0c6aa0 con 0x7f19fc0c72f0 2023-08-25T10:03:07.437 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f1a00d6d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc069f60 con 0x7f19fc0c76d0 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f1a00d6d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc0ae580 con 0x7f19fc0adbb0 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f19faffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f19fc0c76d0 0x7f19fc0bf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f19faffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f19fc0c76d0 0x7f19fc0bf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51056/0 (socket says 172.21.15.71:51056) 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.346+0000 7f19faffd640 1 -- 172.21.15.71:0/1150530162 learned_addr learned my addr 172.21.15.71:0/1150530162 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.347+0000 7f19fbfff640 1 --2- 172.21.15.71:0/1150530162 >> v2:172.21.15.164:3300/0 conn(0x7f19fc0adbb0 0x7f19fc0ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.348+0000 7f19fa7fc640 1 -- 172.21.15.71:0/1150530162 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898010642 0 0) 0x7f19fc0c6aa0 con 0x7f19fc0c72f0 2023-08-25T10:03:07.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.435+0000 7f19fa7fc640 1 -- 172.21.15.71:0/1150530162 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e4003500 con 0x7f19fc0c72f0 2023-08-25T10:03:07.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.435+0000 7f19faffd640 1 -- 172.21.15.71:0/1150530162 >> v2:172.21.15.164:3300/0 conn(0x7f19fc0adbb0 msgr2=0x7f19fc0ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:07.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.435+0000 7f19faffd640 1 --2- 172.21.15.71:0/1150530162 >> v2:172.21.15.164:3300/0 conn(0x7f19fc0adbb0 0x7f19fc0ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.435+0000 7f19faffd640 1 -- 172.21.15.71:0/1150530162 >> v1:172.21.15.164:6789/0 conn(0x7f19fc0c72f0 legacy=0x7f19fc069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:07.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.435+0000 7f19faffd640 1 -- 172.21.15.71:0/1150530162 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc06a9d0 con 0x7f19fc0c76d0 2023-08-25T10:03:07.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.435+0000 7f19fbfff640 1 --2- 172.21.15.71:0/1150530162 >> v2:172.21.15.164:3300/0 conn(0x7f19fc0adbb0 0x7f19fc0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T10:03:07.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.437+0000 7f19faffd640 1 --2- 172.21.15.71:0/1150530162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c76d0 0x7f19fc0bf140 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f19f00083f0 tx=0x7f19f000abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7e583183aa55f server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.438+0000 7f19fa7fc640 1 -- 172.21.15.71:0/1150530162 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0008df0 con 0x7f19fc0c76d0 2023-08-25T10:03:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.438+0000 7f19fa7fc640 1 -- 172.21.15.71:0/1150530162 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f000a0d0 con 0x7f19fc0c76d0 2023-08-25T10:03:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.438+0000 7f19fa7fc640 1 -- 172.21.15.71:0/1150530162 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f000a440 con 0x7f19fc0c76d0 2023-08-25T10:03:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1150530162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c76d0 msgr2=0x7f19fc0bf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1150530162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c76d0 0x7f19fc0bf140 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f19f00083f0 tx=0x7f19f000abe0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1150530162 shutdown_connections 2023-08-25T10:03:07.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1150530162 >> v2:172.21.15.164:3300/0 conn(0x7f19fc0adbb0 0x7f19fc0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1150530162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c76d0 0x7f19fc0bf140 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1150530162 >> 172.21.15.71:0/1150530162 conn(0x7f19fc06c280 msgr2=0x7f19fc06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:07.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1150530162 shutdown_connections 2023-08-25T10:03:07.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.439+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1150530162 wait complete. 2023-08-25T10:03:07.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.442+0000 7f1a00d6d640 1 Processor -- start 2023-08-25T10:03:07.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.442+0000 7f1a00d6d640 1 -- start start 2023-08-25T10:03:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.443+0000 7f1a00d6d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19fc0adbb0 0x7f19fc1ec790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.443+0000 7f1a00d6d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 0x7f19fc1ec000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.443+0000 7f1a00d6d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19fc0c76d0 0x7f19fc1fb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:07.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.443+0000 7f1a00d6d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1faf70 con 0x7f19fc0c72f0 2023-08-25T10:03:07.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.443+0000 7f1a00d6d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc0c93e0 con 0x7f19fc0c76d0 2023-08-25T10:03:07.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.443+0000 7f1a00d6d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1ede10 con 0x7f19fc0adbb0 2023-08-25T10:03:07.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.444+0000 7f19faffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 0x7f19fc1ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:07.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.444+0000 7f19faffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 0x7f19fc1ec000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:51070/0 (socket says 172.21.15.71:51070) 2023-08-25T10:03:07.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.444+0000 7f19faffd640 1 -- 172.21.15.71:0/1093872750 learned_addr learned my addr 172.21.15.71:0/1093872750 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:07.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.444+0000 7f19fb7fe640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19fc0adbb0 0x7f19fc1ec790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:07.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.444+0000 7f19fbfff640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19fc0c76d0 0x7f19fc1fb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:07.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19faffd640 1 -- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19fc0adbb0 msgr2=0x7f19fc1ec790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:07.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19faffd640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19fc0adbb0 0x7f19fc1ec790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19faffd640 1 -- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19fc0c76d0 msgr2=0x7f19fc1fb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:07.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19faffd640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19fc0c76d0 0x7f19fc1fb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19faffd640 1 -- 172.21.15.71:0/1093872750 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f400ec70 con 0x7f19fc0c72f0 2023-08-25T10:03:07.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19fb7fe640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19fc0adbb0 0x7f19fc1ec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:03:07.448 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.445+0000 7f19faffd640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 0x7f19fc1ec000 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f19f0008510 tx=0x7f19f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:07.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.446+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f003e070 con 0x7f19fc0c72f0 2023-08-25T10:03:07.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.446+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f00399a0 con 0x7f19fc0c72f0 2023-08-25T10:03:07.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.446+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0039d50 con 0x7f19fc0c72f0 2023-08-25T10:03:07.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.446+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f0008050 con 0x7f19fc0c72f0 2023-08-25T10:03:07.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.446+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc1ec540 con 0x7f19fc0c72f0 2023-08-25T10:03:07.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.448+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc0af580 con 0x7f19fc0c72f0 2023-08-25T10:03:07.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.449+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f19f002f450 con 0x7f19fc0c72f0 2023-08-25T10:03:07.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.451+0000 7f19e3fff640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19e806a050 0x7f19e806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:07.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.451+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f19f00aae30 con 0x7f19fc0c72f0 2023-08-25T10:03:07.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.451+0000 7f19fb7fe640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19e806a050 0x7f19e806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:07.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.452+0000 7f19fb7fe640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19e806a050 0x7f19e806c510 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f19ec001710 tx=0x7f19ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:07.458 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.456+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f19f0076050 con 0x7f19fc0c72f0 2023-08-25T10:03:07.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.759+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19fc0c9920 con 0x7f19fc0c72f0 2023-08-25T10:03:07.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.760+0000 7f19e3fff640 1 -- 172.21.15.71:0/1093872750 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f19f0079670 con 0x7f19fc0c72f0 2023-08-25T10:03:07.762 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:07.762 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:07.762 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:07.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19e806a050 msgr2=0x7f19e806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:07.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19e806a050 0x7f19e806c510 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f19ec001710 tx=0x7f19ec009040 comp rx=0 tx=0).stop 2023-08-25T10:03:07.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 msgr2=0x7f19fc1ec000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:07.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 0x7f19fc1ec000 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f19f0008510 tx=0x7f19f00046c0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 shutdown_connections 2023-08-25T10:03:07.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19fc0c76d0 0x7f19fc1fb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19e806a050 0x7f19e806c510 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19fc0c72f0 0x7f19fc1ec000 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 --2- 172.21.15.71:0/1093872750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19fc0adbb0 0x7f19fc1ec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:07.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 >> 172.21.15.71:0/1093872750 conn(0x7f19fc06c280 msgr2=0x7f19fc06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:07.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 shutdown_connections 2023-08-25T10:03:07.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:07.763+0000 7f1a00d6d640 1 -- 172.21.15.71:0/1093872750 wait complete. 2023-08-25T10:03:07.785 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:09.786 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:10.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.016+0000 7f2809ae0640 1 Processor -- start 2023-08-25T10:03:10.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.016+0000 7f2809ae0640 1 -- start start 2023-08-25T10:03:10.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2809ae0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2804069850 0x7f28041576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:10.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2809ae0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2804157bf0 con 0x7f28041542d0 2023-08-25T10:03:10.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2809ae0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280406a130 con 0x7f28040c4fb0 2023-08-25T10:03:10.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2809ae0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28040c4b60 con 0x7f2804069850 2023-08-25T10:03:10.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2808ade640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f28041542d0 0x7f28040c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41270/0 (socket says 172.21.15.71:41270) 2023-08-25T10:03:10.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2803fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f2804069850 0x7f28041576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.018+0000 7f2808ade640 1 -- 172.21.15.71:0/1105756069 learned_addr learned my addr 172.21.15.71:0/1105756069 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.019+0000 7f28037fe640 1 -- 172.21.15.71:0/1105756069 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699982203 0 0) 0x7f27f8001110 con 0x7f28041542d0 2023-08-25T10:03:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.019+0000 7f28037fe640 1 -- 172.21.15.71:0/1105756069 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f4003520 con 0x7f28041542d0 2023-08-25T10:03:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.019+0000 7f2803fff640 1 -- 172.21.15.71:0/1105756069 >> v1:172.21.15.164:6789/0 conn(0x7f28040c4fb0 legacy=0x7f28040c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.019+0000 7f2803fff640 1 -- 172.21.15.71:0/1105756069 >> v1:172.21.15.71:6789/0 conn(0x7f28041542d0 legacy=0x7f28040c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.019+0000 7f2803fff640 1 -- 172.21.15.71:0/1105756069 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28041546b0 con 0x7f2804069850 2023-08-25T10:03:10.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.020+0000 7f2803fff640 1 --2- 172.21.15.71:0/1105756069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2804069850 0x7f28041576b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f27f00097a0 tx=0x7f27f002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7943b572b4c908 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:10.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.020+0000 7f28037fe640 1 -- 172.21.15.71:0/1105756069 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0030610 con 0x7f2804069850 2023-08-25T10:03:10.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.020+0000 7f28037fe640 1 -- 172.21.15.71:0/1105756069 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0030770 con 0x7f2804069850 2023-08-25T10:03:10.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.020+0000 7f28037fe640 1 -- 172.21.15.71:0/1105756069 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0030b00 con 0x7f2804069850 2023-08-25T10:03:10.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 -- 172.21.15.71:0/1105756069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2804069850 msgr2=0x7f28041576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 --2- 172.21.15.71:0/1105756069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2804069850 0x7f28041576b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f27f00097a0 tx=0x7f27f002b960 comp rx=0 tx=0).stop 2023-08-25T10:03:10.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 -- 172.21.15.71:0/1105756069 shutdown_connections 2023-08-25T10:03:10.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 --2- 172.21.15.71:0/1105756069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2804069850 0x7f28041576b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 -- 172.21.15.71:0/1105756069 >> 172.21.15.71:0/1105756069 conn(0x7f280406c280 msgr2=0x7f280406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:10.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 -- 172.21.15.71:0/1105756069 shutdown_connections 2023-08-25T10:03:10.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.021+0000 7f2809ae0640 1 -- 172.21.15.71:0/1105756069 wait complete. 2023-08-25T10:03:10.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.023+0000 7f2809ae0640 1 Processor -- start 2023-08-25T10:03:10.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.023+0000 7f2809ae0640 1 -- start start 2023-08-25T10:03:10.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f2809ae0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2804069850 0x7f28040c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:10.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f2809ae0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28040c4fb0 0x7f2804148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:10.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f2809ae0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 0x7f28041490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:10.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f2809ae0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28041ebde0 con 0x7f28041542d0 2023-08-25T10:03:10.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f2809ae0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280414ae00 con 0x7f28040c4fb0 2023-08-25T10:03:10.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f2809ae0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280414b0e0 con 0x7f2804069850 2023-08-25T10:03:10.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f28092df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 0x7f28041490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f28092df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 0x7f28041490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:45986/0 (socket says 172.21.15.71:45986) 2023-08-25T10:03:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.024+0000 7f28092df640 1 -- 172.21.15.71:0/2116152930 learned_addr learned my addr 172.21.15.71:0/2116152930 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f2808ade640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2804069850 0x7f28040c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f2803fff640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28040c4fb0 0x7f2804148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f28092df640 1 -- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2804069850 msgr2=0x7f28040c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f28092df640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2804069850 0x7f28040c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f28092df640 1 -- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28040c4fb0 msgr2=0x7f2804148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f28092df640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28040c4fb0 0x7f2804148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f28092df640 1 -- 172.21.15.71:0/2116152930 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0009400 con 0x7f28041542d0 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.025+0000 7f28092df640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 0x7f28041490a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c7a0 tx=0x7f27fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.026+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc003b70 con 0x7f28041542d0 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.026+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc003cf0 con 0x7f28041542d0 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.026+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc003ea0 con 0x7f28041542d0 2023-08-25T10:03:10.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.026+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280414b3c0 con 0x7f28041542d0 2023-08-25T10:03:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.026+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280414b980 con 0x7f28041542d0 2023-08-25T10:03:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.027+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28041ebf30 con 0x7f28041542d0 2023-08-25T10:03:10.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.028+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f27fc00aa90 con 0x7f28041542d0 2023-08-25T10:03:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.029+0000 7f28017fa640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f27e806a080 0x7f27e806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.029+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f27fc088960 con 0x7f28041542d0 2023-08-25T10:03:10.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.029+0000 7f2808ade640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f27e806a080 0x7f27e806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:10.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.030+0000 7f2808ade640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f27e806a080 0x7f27e806c540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f27f8007fd0 tx=0x7f27f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:10.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.034+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f28041ebf30 con 0x7f28041542d0 2023-08-25T10:03:10.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.310+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28041ebf30 con 0x7f28041542d0 2023-08-25T10:03:10.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.311+0000 7f28017fa640 1 -- 172.21.15.71:0/2116152930 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f28041ebf30 con 0x7f28041542d0 2023-08-25T10:03:10.317 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:10.317 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:10.318 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:10.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f27e806a080 msgr2=0x7f27e806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f27e806a080 0x7f27e806c540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f27f8007fd0 tx=0x7f27f8009040 comp rx=0 tx=0).stop 2023-08-25T10:03:10.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 msgr2=0x7f28041490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:10.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 0x7f28041490a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c7a0 tx=0x7f27fc00cc60 comp rx=0 tx=0).stop 2023-08-25T10:03:10.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 shutdown_connections 2023-08-25T10:03:10.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28040c4fb0 0x7f2804148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f27e806a080 0x7f27e806c540 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f28041542d0 0x7f28041490a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 --2- 172.21.15.71:0/2116152930 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2804069850 0x7f28040c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:10.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 >> 172.21.15.71:0/2116152930 conn(0x7f280406c280 msgr2=0x7f28040c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:10.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 shutdown_connections 2023-08-25T10:03:10.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:10.314+0000 7f2809ae0640 1 -- 172.21.15.71:0/2116152930 wait complete. 2023-08-25T10:03:10.336 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:03:10.338 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:03:12.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:12.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.603+0000 7f7b30fab640 1 Processor -- start 2023-08-25T10:03:12.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.603+0000 7f7b30fab640 1 -- start start 2023-08-25T10:03:12.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b30fab640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7b2c0cb190 0x7f7b2c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:12.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b30fab640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7b2c0cb570 0x7f7b2c15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:12.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b30fab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0b1af0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b30fab640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c0c98f0 con 0x7f7b2c0cb570 2023-08-25T10:03:12.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b30fab640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c0b0fb0 con 0x7f7b2c0cb190 2023-08-25T10:03:12.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b2affd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7b2c0cb570 0x7f7b2c15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:12.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b2bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7b2c0c7f10 0x7f7b2c158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41280/0 (socket says 172.21.15.71:41280) 2023-08-25T10:03:12.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b2bfff640 1 -- 172.21.15.71:0/3988915756 learned_addr learned my addr 172.21.15.71:0/3988915756 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:12.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.605+0000 7f7b2b7fe640 1 --2- 172.21.15.71:0/3988915756 >> v2:172.21.15.164:3300/0 conn(0x7f7b2c0cb190 0x7f7b2c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:12.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.606+0000 7f7b2affd640 1 -- 172.21.15.71:0/3988915756 >> v2:172.21.15.164:3300/0 conn(0x7f7b2c0cb190 msgr2=0x7f7b2c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:12.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.606+0000 7f7b2affd640 1 --2- 172.21.15.71:0/3988915756 >> v2:172.21.15.164:3300/0 conn(0x7f7b2c0cb190 0x7f7b2c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.606+0000 7f7b2affd640 1 -- 172.21.15.71:0/3988915756 >> v1:172.21.15.71:6789/0 conn(0x7f7b2c0c7f10 legacy=0x7f7b2c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:12.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.606+0000 7f7b2affd640 1 -- 172.21.15.71:0/3988915756 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0b1530 con 0x7f7b2c0cb570 2023-08-25T10:03:12.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.607+0000 7f7b2affd640 1 --2- 172.21.15.71:0/3988915756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c15cd00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0cc000 tx=0x7f7b2002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2190c56648dce736 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:12.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.607+0000 7f7b2a7fc640 1 -- 172.21.15.71:0/3988915756 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20030610 con 0x7f7b2c0cb570 2023-08-25T10:03:12.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.607+0000 7f7b2a7fc640 1 -- 172.21.15.71:0/3988915756 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20030770 con 0x7f7b2c0cb570 2023-08-25T10:03:12.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b2a7fc640 1 -- 172.21.15.71:0/3988915756 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20030b00 con 0x7f7b2c0cb570 2023-08-25T10:03:12.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 -- 172.21.15.71:0/3988915756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0cb570 msgr2=0x7f7b2c15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:12.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 --2- 172.21.15.71:0/3988915756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c15cd00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0cc000 tx=0x7f7b2002b960 comp rx=0 tx=0).stop 2023-08-25T10:03:12.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 -- 172.21.15.71:0/3988915756 shutdown_connections 2023-08-25T10:03:12.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 --2- 172.21.15.71:0/3988915756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c15cd00 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 --2- 172.21.15.71:0/3988915756 >> v2:172.21.15.164:3300/0 conn(0x7f7b2c0cb190 0x7f7b2c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 -- 172.21.15.71:0/3988915756 >> 172.21.15.71:0/3988915756 conn(0x7f7b2c06ae20 msgr2=0x7f7b2c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:12.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 -- 172.21.15.71:0/3988915756 shutdown_connections 2023-08-25T10:03:12.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.608+0000 7f7b30fab640 1 -- 172.21.15.71:0/3988915756 wait complete. 2023-08-25T10:03:12.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.610+0000 7f7b30fab640 1 Processor -- start 2023-08-25T10:03:12.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.611+0000 7f7b30fab640 1 -- start start 2023-08-25T10:03:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b30fab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 0x7f7b2c0cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b30fab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b2c0cb190 0x7f7b2c14e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b30fab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c14be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b30fab640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0ce4d0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b30fab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0cf260 con 0x7f7b2c0cb570 2023-08-25T10:03:12.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b30fab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c14c360 con 0x7f7b2c0cb190 2023-08-25T10:03:12.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b2b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 0x7f7b2c0cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:12.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b2b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 0x7f7b2c0cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46000/0 (socket says 172.21.15.71:46000) 2023-08-25T10:03:12.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.612+0000 7f7b2b7fe640 1 -- 172.21.15.71:0/1523781390 learned_addr learned my addr 172.21.15.71:0/1523781390 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:12.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2bfff640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c14be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:12.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2b7fe640 1 -- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b2c0cb190 msgr2=0x7f7b2c14e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:12.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2b7fe640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b2c0cb190 0x7f7b2c14e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2b7fe640 1 -- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b2c0cb570 msgr2=0x7f7b2c14be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2b7fe640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c14be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2b7fe640 1 -- 172.21.15.71:0/1523781390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20009400 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2bfff640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c14be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b2b7fe640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 0x7f7b2c0cd720 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00d830 tx=0x7f7b1c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00b6c0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00b820 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c0cdf30 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.613+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00bbb0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.614+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c0ce8c0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.615+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c0ce610 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.616+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7b1c01e460 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.617+0000 7f7b13fff640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7b1806a050 0x7f7b1806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:12.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.618+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f7b1c0891c0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.618+0000 7f7b2affd640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7b1806a050 0x7f7b1806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:12.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.619+0000 7f7b2affd640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7b1806a050 0x7f7b1806c510 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b20036000 tx=0x7f7b20030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:12.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.623+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7b1c0543f0 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.902+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b2c04e420 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.903+0000 7f7b13fff640 1 -- 172.21.15.71:0/1523781390 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f7b1c057a10 con 0x7f7b2c0c7f10 2023-08-25T10:03:12.906 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:12.906 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:12.907 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:12.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7b1806a050 msgr2=0x7f7b1806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:12.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7b1806a050 0x7f7b1806c510 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b20036000 tx=0x7f7b20030000 comp rx=0 tx=0).stop 2023-08-25T10:03:12.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 msgr2=0x7f7b2c0cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:12.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 0x7f7b2c0cd720 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00d830 tx=0x7f7b1c00dcf0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 shutdown_connections 2023-08-25T10:03:12.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b2c0cb570 0x7f7b2c14be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7b1806a050 0x7f7b1806c510 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b2c0cb190 0x7f7b2c14e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 --2- 172.21.15.71:0/1523781390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b2c0c7f10 0x7f7b2c0cd720 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:12.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 >> 172.21.15.71:0/1523781390 conn(0x7f7b2c06ae20 msgr2=0x7f7b2c0b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:12.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 shutdown_connections 2023-08-25T10:03:12.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:12.906+0000 7f7b30fab640 1 -- 172.21.15.71:0/1523781390 wait complete. 2023-08-25T10:03:12.927 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:14.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:15.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.161+0000 7f84d5d96640 1 Processor -- start 2023-08-25T10:03:15.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.161+0000 7f84d5d96640 1 -- start start 2023-08-25T10:03:15.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84d5d96640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84d0152d80 0x7f84d00c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:15.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84d5d96640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d00cacb0 0x7f84d00c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:15.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84d5d96640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d01591c0 con 0x7f84d015c740 2023-08-25T10:03:15.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84d5d96640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d01594a0 con 0x7f84d0152d80 2023-08-25T10:03:15.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84d5d96640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0159780 con 0x7f84d00cacb0 2023-08-25T10:03:15.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84cf7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84d0152d80 0x7f84d00c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:15.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84cf7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f84d0152d80 0x7f84d00c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46014/0 (socket says 172.21.15.71:46014) 2023-08-25T10:03:15.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.162+0000 7f84cf7fe640 1 -- 172.21.15.71:0/1450014514 learned_addr learned my addr 172.21.15.71:0/1450014514 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:15.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.163+0000 7f84ceffd640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d00cacb0 0x7f84d00c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:15.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.163+0000 7f84cf7fe640 1 -- 172.21.15.71:0/1450014514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d00cacb0 msgr2=0x7f84d00c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:15.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.163+0000 7f84cf7fe640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d00cacb0 0x7f84d00c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.163+0000 7f84cf7fe640 1 -- 172.21.15.71:0/1450014514 >> v1:172.21.15.164:6789/0 conn(0x7f84d015c740 legacy=0x7f84d015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:15.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.163+0000 7f84cf7fe640 1 -- 172.21.15.71:0/1450014514 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0159a60 con 0x7f84d0152d80 2023-08-25T10:03:15.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.163+0000 7f84ceffd640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d00cacb0 0x7f84d00c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:15.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.164+0000 7f84cf7fe640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d0152d80 0x7f84d00c9360 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009870 tx=0x7f84c002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56327a7ebf47c4ba server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:15.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.164+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1450014514 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0030840 con 0x7f84d0152d80 2023-08-25T10:03:15.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1450014514 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c00309a0 con 0x7f84d0152d80 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84ce7fc640 1 -- 172.21.15.71:0/1450014514 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0030d30 con 0x7f84d0152d80 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84d5d96640 1 -- 172.21.15.71:0/1450014514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d0152d80 msgr2=0x7f84d00c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84d5d96640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d0152d80 0x7f84d00c9360 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009870 tx=0x7f84c002b980 comp rx=0 tx=0).stop 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84d5d96640 1 -- 172.21.15.71:0/1450014514 shutdown_connections 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84d5d96640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d00cacb0 0x7f84d00c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84d5d96640 1 --2- 172.21.15.71:0/1450014514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d0152d80 0x7f84d00c9360 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.165+0000 7f84d5d96640 1 -- 172.21.15.71:0/1450014514 >> 172.21.15.71:0/1450014514 conn(0x7f84d006b8f0 msgr2=0x7f84d00654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:15.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.166+0000 7f84d5d96640 1 -- 172.21.15.71:0/1450014514 shutdown_connections 2023-08-25T10:03:15.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.166+0000 7f84d5d96640 1 -- 172.21.15.71:0/1450014514 wait complete. 2023-08-25T10:03:15.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.168+0000 7f84d5d96640 1 Processor -- start 2023-08-25T10:03:15.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.168+0000 7f84d5d96640 1 -- start start 2023-08-25T10:03:15.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84d5d96640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 0x7f84d015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:15.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84d5d96640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d0152d80 0x7f84d0158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:15.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84d5d96640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84d015c740 0x7f84d015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:15.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84d5d96640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d01f2660 con 0x7f84d00cacb0 2023-08-25T10:03:15.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84d5d96640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d01f20e0 con 0x7f84d015c740 2023-08-25T10:03:15.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84d5d96640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d01f2380 con 0x7f84d0152d80 2023-08-25T10:03:15.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.169+0000 7f84cf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 0x7f84d015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:15.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 0x7f84d015bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46022/0 (socket says 172.21.15.71:46022) 2023-08-25T10:03:15.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 -- 172.21.15.71:0/78220026 learned_addr learned my addr 172.21.15.71:0/78220026 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:15.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cffff640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84d015c740 0x7f84d015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:15.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84ceffd640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d0152d80 0x7f84d0158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:15.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 -- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d0152d80 msgr2=0x7f84d0158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:15.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d0152d80 0x7f84d0158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 -- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84d015c740 msgr2=0x7f84d015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:15.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84d015c740 0x7f84d015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 -- 172.21.15.71:0/78220026 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c00094d0 con 0x7f84d00cacb0 2023-08-25T10:03:15.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cffff640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84d015c740 0x7f84d015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:15.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84ceffd640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d0152d80 0x7f84d0158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:15.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.170+0000 7f84cf7fe640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 0x7f84d015bb80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009870 tx=0x7f84c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:15.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.171+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00307f0 con 0x7f84d00cacb0 2023-08-25T10:03:15.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.171+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c0043400 con 0x7f84d00cacb0 2023-08-25T10:03:15.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.171+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00437b0 con 0x7f84d00cacb0 2023-08-25T10:03:15.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.171+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d015a1b0 con 0x7f84d00cacb0 2023-08-25T10:03:15.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.171+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d015a770 con 0x7f84d00cacb0 2023-08-25T10:03:15.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.172+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d00b1a60 con 0x7f84d00cacb0 2023-08-25T10:03:15.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.173+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f84c0043910 con 0x7f84d00cacb0 2023-08-25T10:03:15.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.174+0000 7f84b7fff640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84bc06a050 0x7f84bc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:15.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.174+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f84c003e070 con 0x7f84d00cacb0 2023-08-25T10:03:15.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.175+0000 7f84ceffd640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84bc06a050 0x7f84bc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:15.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.176+0000 7f84ceffd640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84bc06a050 0x7f84bc06c510 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f84c4006fd0 tx=0x7f84c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:15.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.179+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f84c0076f20 con 0x7f84d00cacb0 2023-08-25T10:03:15.434 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.433+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84d0069160 con 0x7f84d00cacb0 2023-08-25T10:03:15.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.434+0000 7f84b7fff640 1 -- 172.21.15.71:0/78220026 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f84c007a540 con 0x7f84d00cacb0 2023-08-25T10:03:15.436 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:15.436 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:15.436 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:15.438 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84bc06a050 msgr2=0x7f84bc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84bc06a050 0x7f84bc06c510 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f84c4006fd0 tx=0x7f84c4008040 comp rx=0 tx=0).stop 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 msgr2=0x7f84d015bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 0x7f84d015bb80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009870 tx=0x7f84c0004730 comp rx=0 tx=0).stop 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 shutdown_connections 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f84d015c740 0x7f84d015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f84bc06a050 0x7f84bc06c510 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f84d0152d80 0x7f84d0158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 --2- 172.21.15.71:0/78220026 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84d00cacb0 0x7f84d015bb80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:15.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.436+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 >> 172.21.15.71:0/78220026 conn(0x7f84d006b8f0 msgr2=0x7f84d00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:15.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.437+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 shutdown_connections 2023-08-25T10:03:15.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:15.437+0000 7f84d5d96640 1 -- 172.21.15.71:0/78220026 wait complete. 2023-08-25T10:03:15.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:17.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:17.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.698+0000 7f05815b2640 1 Processor -- start 2023-08-25T10:03:17.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.698+0000 7f05815b2640 1 -- start start 2023-08-25T10:03:17.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.700+0000 7f05815b2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f057c06ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.700+0000 7f05815b2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c06ef70 con 0x7f057c0c6df0 2023-08-25T10:03:17.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.700+0000 7f05815b2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057c06c620 con 0x7f057c0c7520 2023-08-25T10:03:17.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.700+0000 7f05815b2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c06f250 con 0x7f057c1532a0 2023-08-25T10:03:17.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.700+0000 7f057b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f057c0c6df0 0x7f057c06e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41286/0 (socket says 172.21.15.71:41286) 2023-08-25T10:03:17.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.700+0000 7f057b7fe640 1 -- 172.21.15.71:0/803939313 learned_addr learned my addr 172.21.15.71:0/803939313 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:17.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057bfff640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f057c06ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230033631 0 0) 0x7f057c06ef70 con 0x7f057c0c6df0 2023-08-25T10:03:17.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0564003520 con 0x7f057c0c6df0 2023-08-25T10:03:17.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192892322 0 0) 0x7f0564003520 con 0x7f057c0c6df0 2023-08-25T10:03:17.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 msgr2=0x7f057c06ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f057c06ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 >> v1:172.21.15.164:6789/0 conn(0x7f057c0c7520 legacy=0x7f057c1637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c15cf70 con 0x7f057c0c6df0 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0570001670 con 0x7f057c0c6df0 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 >> v1:172.21.15.71:6789/0 conn(0x7f057c0c6df0 legacy=0x7f057c06e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f0564003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0564003f90 0x7f05640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.701+0000 7f057affd640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564004d00 0x7f0564005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c06f250 con 0x7f0564003f90 2023-08-25T10:03:17.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056400bb70 con 0x7f0564004d00 2023-08-25T10:03:17.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056400bd30 con 0x7f057c1532a0 2023-08-25T10:03:17.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057b7fe640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0564003f90 0x7f05640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057bfff640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f0564003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f0580db1640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564004d00 0x7f0564005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057b7fe640 1 -- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 msgr2=0x7f0564003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:17.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057b7fe640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f0564003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057b7fe640 1 -- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564004d00 msgr2=0x7f0564005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:17.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057b7fe640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564004d00 0x7f0564005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057b7fe640 1 -- 172.21.15.71:0/803939313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0564003520 con 0x7f0564003f90 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.702+0000 7f057bfff640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c1532a0 0x7f0564003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.704+0000 7f057b7fe640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0564003f90 0x7f05640047c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f057000c210 tx=0x7f05640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8176c755348968d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.704+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0570005ad0 con 0x7f0564003f90 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.704+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0570005c30 con 0x7f0564003f90 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.704+0000 7f057affd640 1 -- 172.21.15.71:0/803939313 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057002ca00 con 0x7f0564003f90 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f05815b2640 1 -- 172.21.15.71:0/803939313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0564003f90 msgr2=0x7f05640047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f05815b2640 1 --2- 172.21.15.71:0/803939313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0564003f90 0x7f05640047c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f057000c210 tx=0x7f05640057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f0580db1640 1 -- 172.21.15.71:0/803939313 reap_dead start 2023-08-25T10:03:17.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f05815b2640 1 -- 172.21.15.71:0/803939313 shutdown_connections 2023-08-25T10:03:17.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f05815b2640 1 -- 172.21.15.71:0/803939313 >> 172.21.15.71:0/803939313 conn(0x7f057c0654f0 msgr2=0x7f057c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:17.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f05815b2640 1 -- 172.21.15.71:0/803939313 shutdown_connections 2023-08-25T10:03:17.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.705+0000 7f05815b2640 1 -- 172.21.15.71:0/803939313 wait complete. 2023-08-25T10:03:17.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.707+0000 7f05815b2640 1 Processor -- start 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.707+0000 7f05815b2640 1 -- start start 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.708+0000 7f05815b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564003f90 0x7f057c1ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.708+0000 7f05815b2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c154080 0x7f057c0becd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.708+0000 7f05815b2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 0x7f057c1f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.708+0000 7f05815b2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c15d350 con 0x7f057c155630 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.708+0000 7f05815b2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c153930 con 0x7f0564003f90 2023-08-25T10:03:17.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.708+0000 7f05815b2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c153e60 con 0x7f057c154080 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 0x7f057c1f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 0x7f057c1f1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46032/0 (socket says 172.21.15.71:46032) 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 -- 172.21.15.71:0/1818042606 learned_addr learned my addr 172.21.15.71:0/1818042606 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f057bfff640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564003f90 0x7f057c1ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f057b7fe640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c154080 0x7f057c0becd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 -- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c154080 msgr2=0x7f057c0becd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c154080 0x7f057c0becd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 -- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564003f90 msgr2=0x7f057c1ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564003f90 0x7f057c1ef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 -- 172.21.15.71:0/1818042606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057c155ab0 con 0x7f057c155630 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f057b7fe640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c154080 0x7f057c0becd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.709+0000 7f0580db1640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 0x7f057c1f1900 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0574000c00 tx=0x7f057400e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.710+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05740052e0 con 0x7f057c155630 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.710+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0574005440 con 0x7f057c155630 2023-08-25T10:03:17.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.710+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574003830 con 0x7f057c155630 2023-08-25T10:03:17.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.710+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c1f22d0 con 0x7f057c155630 2023-08-25T10:03:17.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.710+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c153b80 con 0x7f057c155630 2023-08-25T10:03:17.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.711+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c04e420 con 0x7f057c155630 2023-08-25T10:03:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.713+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0574003990 con 0x7f057c155630 2023-08-25T10:03:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.714+0000 7f0578ff9640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f056806a050 0x7f056806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.714+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f0574088140 con 0x7f057c155630 2023-08-25T10:03:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.714+0000 7f057bfff640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f056806a050 0x7f056806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:17.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.715+0000 7f057bfff640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f056806a050 0x7f056806c510 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f056c007520 tx=0x7f056c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:17.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.719+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f05740532d0 con 0x7f057c155630 2023-08-25T10:03:17.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.983+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f057c06abf0 con 0x7f057c155630 2023-08-25T10:03:17.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.984+0000 7f0578ff9640 1 -- 172.21.15.71:0/1818042606 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f057400eea0 con 0x7f057c155630 2023-08-25T10:03:17.986 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:17.986 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:17.986 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:17.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.986+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f056806a050 msgr2=0x7f056806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.986+0000 7f05815b2640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f056806a050 0x7f056806c510 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f056c007520 tx=0x7f056c00b040 comp rx=0 tx=0).stop 2023-08-25T10:03:17.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 msgr2=0x7f057c1f1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:17.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 0x7f057c1f1900 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0574000c00 tx=0x7f057400e820 comp rx=0 tx=0).stop 2023-08-25T10:03:17.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 shutdown_connections 2023-08-25T10:03:17.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f056806a050 0x7f056806c510 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.989 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f057c155630 0x7f057c1f1900 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f057c154080 0x7f057c0becd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 --2- 172.21.15.71:0/1818042606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0564003f90 0x7f057c1ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 >> 172.21.15.71:0/1818042606 conn(0x7f057c0654f0 msgr2=0x7f057c158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 shutdown_connections 2023-08-25T10:03:17.990 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:17.987+0000 7f05815b2640 1 -- 172.21.15.71:0/1818042606 wait complete. 2023-08-25T10:03:18.010 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:20.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:20.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.253+0000 7ff46c0bf640 1 Processor -- start 2023-08-25T10:03:20.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.254+0000 7ff46c0bf640 1 -- start start 2023-08-25T10:03:20.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.256+0000 7ff46c0bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 0x7ff4640c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.256+0000 7ff46c0bf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640c1dc0 con 0x7ff4640c3a40 2023-08-25T10:03:20.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.256+0000 7ff46c0bf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff464156e00 con 0x7ff4641509d0 2023-08-25T10:03:20.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.256+0000 7ff46c0bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff464157a80 con 0x7ff4640c3660 2023-08-25T10:03:20.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.256+0000 7ff46965a640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff4640c3a40 0x7ff46406ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43734/0 (socket says 172.21.15.71:43734) 2023-08-25T10:03:20.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.256+0000 7ff46965a640 1 -- 172.21.15.71:0/4170068312 learned_addr learned my addr 172.21.15.71:0/4170068312 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff469e5b640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 0x7ff4640c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596913595 0 0) 0x7ff4640c1dc0 con 0x7ff4640c3a40 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff450003520 con 0x7ff4640c3a40 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464880927 0 0) 0x7ff464156e00 con 0x7ff4641509d0 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4640c1dc0 con 0x7ff4641509d0 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298521757 0 0) 0x7ff450003520 con 0x7ff4640c3a40 2023-08-25T10:03:20.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 msgr2=0x7ff4640c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 0x7ff4640c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 >> v1:172.21.15.164:6789/0 conn(0x7ff4641509d0 legacy=0x7ff4640c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4641570e0 con 0x7ff4640c3a40 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.257+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff454001670 con 0x7ff4640c3a40 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 >> v1:172.21.15.71:6789/0 conn(0x7ff4640c3a40 legacy=0x7ff46406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff450003ba0 0x7ff450003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4500047c0 0x7ff450004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 0x7ff450005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff464157a80 con 0x7ff450003ba0 2023-08-25T10:03:20.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45000ba60 con 0x7ff4500047c0 2023-08-25T10:03:20.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45000bc20 con 0x7ff4640c3660 2023-08-25T10:03:20.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff469e5b640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff450003ba0 0x7ff450003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff46965a640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4500047c0 0x7ff450004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.258+0000 7ff46a65c640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 0x7ff450005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.259+0000 7ff469e5b640 1 -- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 msgr2=0x7ff450005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.259+0000 7ff469e5b640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4640c3660 0x7ff450005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.259+0000 7ff469e5b640 1 -- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4500047c0 msgr2=0x7ff450004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.259+0000 7ff469e5b640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4500047c0 0x7ff450004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.259+0000 7ff469e5b640 1 -- 172.21.15.71:0/4170068312 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450003520 con 0x7ff450003ba0 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.259+0000 7ff46965a640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4500047c0 0x7ff450004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.260+0000 7ff469e5b640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff450003ba0 0x7ff450003fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff460009a90 tx=0x7ff4500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa509ac19aa80aa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.260+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46002c680 con 0x7ff450003ba0 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.260+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46002c7e0 con 0x7ff450003ba0 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.260+0000 7ff468e59640 1 -- 172.21.15.71:0/4170068312 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46002cb70 con 0x7ff450003ba0 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46c0bf640 1 -- 172.21.15.71:0/4170068312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff450003ba0 msgr2=0x7ff450003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/4170068312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff450003ba0 0x7ff450003fa0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff460009a90 tx=0x7ff4500057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46a65c640 1 -- 172.21.15.71:0/4170068312 reap_dead start 2023-08-25T10:03:20.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46c0bf640 1 -- 172.21.15.71:0/4170068312 shutdown_connections 2023-08-25T10:03:20.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46c0bf640 1 -- 172.21.15.71:0/4170068312 >> 172.21.15.71:0/4170068312 conn(0x7ff4640654f0 msgr2=0x7ff464065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:20.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46c0bf640 1 -- 172.21.15.71:0/4170068312 shutdown_connections 2023-08-25T10:03:20.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.261+0000 7ff46c0bf640 1 -- 172.21.15.71:0/4170068312 wait complete. 2023-08-25T10:03:20.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.263+0000 7ff46c0bf640 1 Processor -- start 2023-08-25T10:03:20.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.264+0000 7ff46c0bf640 1 -- start start 2023-08-25T10:03:20.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46c0bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff450003ba0 0x7ff464152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46c0bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff464157e20 0x7ff46414b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46c0bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 0x7ff4641efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46c0bf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff464151c30 con 0x7ff46415ad00 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46c0bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4641f0660 con 0x7ff464157e20 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46c0bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff464155f30 con 0x7ff450003ba0 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46a65c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 0x7ff4641efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46a65c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 0x7ff4641efb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38550/0 (socket says 172.21.15.71:38550) 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46a65c640 1 -- 172.21.15.71:0/327276082 learned_addr learned my addr 172.21.15.71:0/327276082 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff46965a640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff464157e20 0x7ff46414b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.265+0000 7ff469e5b640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff450003ba0 0x7ff464152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46a65c640 1 -- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff450003ba0 msgr2=0x7ff464152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46a65c640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff450003ba0 0x7ff464152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46a65c640 1 -- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff464157e20 msgr2=0x7ff46414b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46a65c640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff464157e20 0x7ff46414b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46a65c640 1 -- 172.21.15.71:0/327276082 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46414aca0 con 0x7ff46415ad00 2023-08-25T10:03:20.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46a65c640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 0x7ff4641efb60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff464150db0 tx=0x7ff45c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:20.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c003970 con 0x7ff46415ad00 2023-08-25T10:03:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45c003ad0 con 0x7ff46415ad00 2023-08-25T10:03:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46414af80 con 0x7ff46415ad00 2023-08-25T10:03:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.266+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45c003e80 con 0x7ff46415ad00 2023-08-25T10:03:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.267+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46406a8a0 con 0x7ff46415ad00 2023-08-25T10:03:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.268+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46414b4a0 con 0x7ff46415ad00 2023-08-25T10:03:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.269+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff45c0050d0 con 0x7ff46415ad00 2023-08-25T10:03:20.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.270+0000 7ff45a7fc640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff44806a050 0x7ff44806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:20.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.270+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff45c0888f0 con 0x7ff46415ad00 2023-08-25T10:03:20.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.270+0000 7ff469e5b640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff44806a050 0x7ff44806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:20.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.271+0000 7ff469e5b640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff44806a050 0x7ff44806c510 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff460036000 tx=0x7ff460030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:20.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.276+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff46414b4a0 con 0x7ff46415ad00 2023-08-25T10:03:20.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.564+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff46414b4a0 con 0x7ff46415ad00 2023-08-25T10:03:20.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.565+0000 7ff45a7fc640 1 -- 172.21.15.71:0/327276082 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7ff46414b4a0 con 0x7ff46415ad00 2023-08-25T10:03:20.567 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:20.567 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:20.568 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:20.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.567+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff44806a050 msgr2=0x7ff44806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.567+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff44806a050 0x7ff44806c510 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff460036000 tx=0x7ff460030000 comp rx=0 tx=0).stop 2023-08-25T10:03:20.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.567+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 msgr2=0x7ff4641efb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:20.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.567+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 0x7ff4641efb60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff464150db0 tx=0x7ff45c00eb20 comp rx=0 tx=0).stop 2023-08-25T10:03:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 shutdown_connections 2023-08-25T10:03:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff464157e20 0x7ff46414b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff44806a050 0x7ff44806c510 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff46415ad00 0x7ff4641efb60 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 --2- 172.21.15.71:0/327276082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff450003ba0 0x7ff464152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 >> 172.21.15.71:0/327276082 conn(0x7ff4640654f0 msgr2=0x7ff4640c5350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 shutdown_connections 2023-08-25T10:03:20.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:20.568+0000 7ff46c0bf640 1 -- 172.21.15.71:0/327276082 wait complete. 2023-08-25T10:03:20.589 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:22.590 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:22.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.826+0000 7f78dea83640 1 Processor -- start 2023-08-25T10:03:22.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.827+0000 7f78dea83640 1 -- start start 2023-08-25T10:03:22.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.829+0000 7f78dea83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d814fe10 0x7f78d8156550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.829+0000 7f78dea83640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f78d80c47d0 0x7f78d80c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.829+0000 7f78dea83640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f78d8069850 0x7f78d8069c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.829+0000 7f78dea83640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80c3290 con 0x7f78d8069850 2023-08-25T10:03:22.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.829+0000 7f78dea83640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80c43f0 con 0x7f78d80c47d0 2023-08-25T10:03:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.829+0000 7f78dea83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80c21f0 con 0x7f78d814fe10 2023-08-25T10:03:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f78d8069850 0x7f78d8069c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f78d8069850 0x7f78d8069c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38566/0 (socket says 172.21.15.71:38566) 2023-08-25T10:03:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 -- 172.21.15.71:0/2713177735 learned_addr learned my addr 172.21.15.71:0/2713177735 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78d7fff640 1 --2- 172.21.15.71:0/2713177735 >> v2:172.21.15.164:3300/0 conn(0x7f78d80c47d0 0x7f78d80c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dc81f640 1 --2- 172.21.15.71:0/2713177735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d814fe10 0x7f78d8156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:22.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 -- 172.21.15.71:0/2713177735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d814fe10 msgr2=0x7f78d8156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:22.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 --2- 172.21.15.71:0/2713177735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d814fe10 0x7f78d8156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 -- 172.21.15.71:0/2713177735 >> v2:172.21.15.164:3300/0 conn(0x7f78d80c47d0 msgr2=0x7f78d80c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:22.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 --2- 172.21.15.71:0/2713177735 >> v2:172.21.15.164:3300/0 conn(0x7f78d80c47d0 0x7f78d80c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.830+0000 7f78dd020640 1 -- 172.21.15.71:0/2713177735 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80c3870 con 0x7f78d8069850 2023-08-25T10:03:22.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.832+0000 7f78dd020640 1 --2- 172.21.15.71:0/2713177735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d8069c60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f78d806aec0 tx=0x7f78d002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790bb1b879548d3d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.832+0000 7f78d77fe640 1 -- 172.21.15.71:0/2713177735 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d002ed10 con 0x7f78d8069850 2023-08-25T10:03:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.832+0000 7f78d77fe640 1 -- 172.21.15.71:0/2713177735 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d002ee70 con 0x7f78d8069850 2023-08-25T10:03:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.832+0000 7f78d77fe640 1 -- 172.21.15.71:0/2713177735 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d00365f0 con 0x7f78d8069850 2023-08-25T10:03:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 -- 172.21.15.71:0/2713177735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 msgr2=0x7f78d8069c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 --2- 172.21.15.71:0/2713177735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d8069c60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f78d806aec0 tx=0x7f78d002e170 comp rx=0 tx=0).stop 2023-08-25T10:03:22.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 -- 172.21.15.71:0/2713177735 shutdown_connections 2023-08-25T10:03:22.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 --2- 172.21.15.71:0/2713177735 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d8069c60 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 --2- 172.21.15.71:0/2713177735 >> v2:172.21.15.164:3300/0 conn(0x7f78d80c47d0 0x7f78d80c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 --2- 172.21.15.71:0/2713177735 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d814fe10 0x7f78d8156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 -- 172.21.15.71:0/2713177735 >> 172.21.15.71:0/2713177735 conn(0x7f78d806c8b0 msgr2=0x7f78d806ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:22.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 -- 172.21.15.71:0/2713177735 shutdown_connections 2023-08-25T10:03:22.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.833+0000 7f78dea83640 1 -- 172.21.15.71:0/2713177735 wait complete. 2023-08-25T10:03:22.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.835+0000 7f78dea83640 1 Processor -- start 2023-08-25T10:03:22.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.835+0000 7f78dea83640 1 -- start start 2023-08-25T10:03:22.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.836+0000 7f78dea83640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d80cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.836+0000 7f78dea83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d80c47d0 0x7f78d81ea200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.836+0000 7f78dea83640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78d814fe10 0x7f78d8147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.836+0000 7f78dea83640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80c1100 con 0x7f78d8069850 2023-08-25T10:03:22.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.836+0000 7f78dea83640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80c1350 con 0x7f78d814fe10 2023-08-25T10:03:22.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.836+0000 7f78dea83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80c1630 con 0x7f78d80c47d0 2023-08-25T10:03:22.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d80cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:22.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d80cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38580/0 (socket says 172.21.15.71:38580) 2023-08-25T10:03:22.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 -- 172.21.15.71:0/2299981315 learned_addr learned my addr 172.21.15.71:0/2299981315 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:22.847 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78d7fff640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d80c47d0 0x7f78d81ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:22.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 -- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d80c47d0 msgr2=0x7f78d81ea200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:22.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d80c47d0 0x7f78d81ea200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 -- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78d814fe10 msgr2=0x7f78d8147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:22.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78d814fe10 0x7f78d8147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:22.848 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 -- 172.21.15.71:0/2299981315 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0009b30 con 0x7f78d8069850 2023-08-25T10:03:22.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.837+0000 7f78dc81f640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d80cae20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00d5f0 tx=0x7f78cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:22.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.838+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc0125f0 con 0x7f78d8069850 2023-08-25T10:03:22.849 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.838+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78cc012750 con 0x7f78d8069850 2023-08-25T10:03:22.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.838+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d80c1910 con 0x7f78d8069850 2023-08-25T10:03:22.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.838+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc012b00 con 0x7f78d8069850 2023-08-25T10:03:22.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.838+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d81e9ce0 con 0x7f78d8069850 2023-08-25T10:03:22.850 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.839+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d81e9ea0 con 0x7f78d8069850 2023-08-25T10:03:22.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.840+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f78cc012c60 con 0x7f78d8069850 2023-08-25T10:03:22.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.841+0000 7f78d57fa640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f78bc06a080 0x7f78bc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:22.851 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.841+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f78cc088de0 con 0x7f78d8069850 2023-08-25T10:03:22.852 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.842+0000 7f78d7fff640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f78bc06a080 0x7f78bc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:22.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.843+0000 7f78d7fff640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f78bc06a080 0x7f78bc06c540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f78d8159740 tx=0x7f78c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:22.853 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:22.847+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f78cc0540a0 con 0x7f78d8069850 2023-08-25T10:03:23.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.118+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78d8069160 con 0x7f78d8069850 2023-08-25T10:03:23.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.118+0000 7f78d57fa640 1 -- 172.21.15.71:0/2299981315 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f78cc0576c0 con 0x7f78d8069850 2023-08-25T10:03:23.121 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:23.121 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:23.121 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:23.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.121+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f78bc06a080 msgr2=0x7f78bc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:23.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.121+0000 7f78dea83640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f78bc06a080 0x7f78bc06c540 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f78d8159740 tx=0x7f78c4008040 comp rx=0 tx=0).stop 2023-08-25T10:03:23.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.121+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 msgr2=0x7f78d80cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:23.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.121+0000 7f78dea83640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d80cae20 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00d5f0 tx=0x7f78cc00dab0 comp rx=0 tx=0).stop 2023-08-25T10:03:23.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 shutdown_connections 2023-08-25T10:03:23.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f78d814fe10 0x7f78d8147ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:23.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f78bc06a080 0x7f78bc06c540 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:23.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f78d80c47d0 0x7f78d81ea200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:23.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 --2- 172.21.15.71:0/2299981315 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f78d8069850 0x7f78d80cae20 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:23.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 >> 172.21.15.71:0/2299981315 conn(0x7f78d806c8b0 msgr2=0x7f78d80c27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:23.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 shutdown_connections 2023-08-25T10:03:23.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:23.122+0000 7f78dea83640 1 -- 172.21.15.71:0/2299981315 wait complete. 2023-08-25T10:03:23.147 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:03:23.148 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:03:25.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:25.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.374+0000 7f4e4c071640 1 Processor -- start 2023-08-25T10:03:25.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.374+0000 7f4e4c071640 1 -- start start 2023-08-25T10:03:25.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4c071640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e440c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4c071640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e440c3a40 0x7f4e4406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4c071640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4e441509d0 0x7f4e44150e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4c071640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e44157a80 con 0x7f4e441509d0 2023-08-25T10:03:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4c071640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e440c1dc0 con 0x7f4e440c3a40 2023-08-25T10:03:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4c071640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44151380 con 0x7f4e440c3660 2023-08-25T10:03:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.376+0000 7f4e4a60e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4e441509d0 0x7f4e44150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4e441509d0 0x7f4e44150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38592/0 (socket says 172.21.15.71:38592) 2023-08-25T10:03:25.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 -- 172.21.15.71:0/3943721195 learned_addr learned my addr 172.21.15.71:0/3943721195 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4960c640 1 --2- 172.21.15.71:0/3943721195 >> v2:172.21.15.164:3300/0 conn(0x7f4e440c3a40 0x7f4e4406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e49e0d640 1 --2- 172.21.15.71:0/3943721195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e440c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 -- 172.21.15.71:0/3943721195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 msgr2=0x7f4e440c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 --2- 172.21.15.71:0/3943721195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e440c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 -- 172.21.15.71:0/3943721195 >> v2:172.21.15.164:3300/0 conn(0x7f4e440c3a40 msgr2=0x7f4e4406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 --2- 172.21.15.71:0/3943721195 >> v2:172.21.15.164:3300/0 conn(0x7f4e440c3a40 0x7f4e4406ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.377+0000 7f4e4a60e640 1 -- 172.21.15.71:0/3943721195 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4406cf70 con 0x7f4e441509d0 2023-08-25T10:03:25.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.378+0000 7f4e4a60e640 1 --2- 172.21.15.71:0/3943721195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e44150e40 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4e44071d70 tx=0x7f4e3c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5849489f055d534 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e48e0b640 1 -- 172.21.15.71:0/3943721195 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c02cb00 con 0x7f4e441509d0 2023-08-25T10:03:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e48e0b640 1 -- 172.21.15.71:0/3943721195 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c02cc60 con 0x7f4e441509d0 2023-08-25T10:03:25.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e48e0b640 1 -- 172.21.15.71:0/3943721195 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0343f0 con 0x7f4e441509d0 2023-08-25T10:03:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 -- 172.21.15.71:0/3943721195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 msgr2=0x7f4e44150e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:25.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 --2- 172.21.15.71:0/3943721195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e44150e40 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f4e44071d70 tx=0x7f4e3c02bf60 comp rx=0 tx=0).stop 2023-08-25T10:03:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 -- 172.21.15.71:0/3943721195 shutdown_connections 2023-08-25T10:03:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 --2- 172.21.15.71:0/3943721195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e44150e40 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 --2- 172.21.15.71:0/3943721195 >> v2:172.21.15.164:3300/0 conn(0x7f4e440c3a40 0x7f4e4406ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 --2- 172.21.15.71:0/3943721195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e440c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.379+0000 7f4e4c071640 1 -- 172.21.15.71:0/3943721195 >> 172.21.15.71:0/3943721195 conn(0x7f4e440654f0 msgr2=0x7f4e44065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:25.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.380+0000 7f4e4c071640 1 -- 172.21.15.71:0/3943721195 shutdown_connections 2023-08-25T10:03:25.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.380+0000 7f4e4c071640 1 -- 172.21.15.71:0/3943721195 wait complete. 2023-08-25T10:03:25.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.381+0000 7f4e4c071640 1 Processor -- start 2023-08-25T10:03:25.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.381+0000 7f4e4c071640 1 -- start start 2023-08-25T10:03:25.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.382+0000 7f4e4c071640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e441f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.382+0000 7f4e4c071640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e440c3a40 0x7f4e44145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.382+0000 7f4e4c071640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e440c0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.382+0000 7f4e4c071640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44151b60 con 0x7f4e441509d0 2023-08-25T10:03:25.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.382+0000 7f4e4c071640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44146320 con 0x7f4e440c3a40 2023-08-25T10:03:25.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.382+0000 7f4e4c071640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e44151550 con 0x7f4e440c3660 2023-08-25T10:03:25.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e440c0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e440c0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38600/0 (socket says 172.21.15.71:38600) 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 -- 172.21.15.71:0/226927416 learned_addr learned my addr 172.21.15.71:0/226927416 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4960c640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e440c3a40 0x7f4e44145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e49e0d640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e441f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 -- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 msgr2=0x7f4e441f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e441f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 -- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e440c3a40 msgr2=0x7f4e44145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:25.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e440c3a40 0x7f4e44145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e4a60e640 1 -- 172.21.15.71:0/226927416 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c009b30 con 0x7f4e441509d0 2023-08-25T10:03:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.383+0000 7f4e49e0d640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e441f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.384+0000 7f4e4a60e640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e440c0710 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c036040 tx=0x7f4e3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.384+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c009510 con 0x7f4e441509d0 2023-08-25T10:03:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.384+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c03e450 con 0x7f4e441509d0 2023-08-25T10:03:25.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.384+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c03e800 con 0x7f4e441509d0 2023-08-25T10:03:25.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.384+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4406e5c0 con 0x7f4e441509d0 2023-08-25T10:03:25.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.384+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4406eb80 con 0x7f4e441509d0 2023-08-25T10:03:25.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.386+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e440c29c0 con 0x7f4e441509d0 2023-08-25T10:03:25.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.386+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4e3c03e960 con 0x7f4e441509d0 2023-08-25T10:03:25.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.387+0000 7f4e3a7fc640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e2806a050 0x7f4e2806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:25.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.387+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4e3c0ab8d0 con 0x7f4e441509d0 2023-08-25T10:03:25.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.388+0000 7f4e49e0d640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e2806a050 0x7f4e2806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:25.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.388+0000 7f4e49e0d640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e2806a050 0x7f4e2806c510 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4e40006fd0 tx=0x7f4e40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:25.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.392+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4e3c076b80 con 0x7f4e441509d0 2023-08-25T10:03:25.677 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.675+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e44157390 con 0x7f4e441509d0 2023-08-25T10:03:25.678 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.676+0000 7f4e3a7fc640 1 -- 172.21.15.71:0/226927416 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4e3c07a1a0 con 0x7f4e441509d0 2023-08-25T10:03:25.678 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:25.678 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:25.679 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:25.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e2806a050 msgr2=0x7f4e2806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:25.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e2806a050 0x7f4e2806c510 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4e40006fd0 tx=0x7f4e40008040 comp rx=0 tx=0).stop 2023-08-25T10:03:25.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 msgr2=0x7f4e440c0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:25.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e440c0710 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c036040 tx=0x7f4e3c004a40 comp rx=0 tx=0).stop 2023-08-25T10:03:25.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 shutdown_connections 2023-08-25T10:03:25.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e440c3a40 0x7f4e44145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4e2806a050 0x7f4e2806c510 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e441509d0 0x7f4e440c0710 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 --2- 172.21.15.71:0/226927416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e440c3660 0x7f4e441f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:25.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 >> 172.21.15.71:0/226927416 conn(0x7f4e440654f0 msgr2=0x7f4e44150250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:25.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 shutdown_connections 2023-08-25T10:03:25.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:25.679+0000 7f4e4c071640 1 -- 172.21.15.71:0/226927416 wait complete. 2023-08-25T10:03:25.698 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:26.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:03:26.500 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:03:27.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:27.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.944+0000 7fcf225dd640 1 Processor -- start 2023-08-25T10:03:27.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.945+0000 7fcf225dd640 1 -- start start 2023-08-25T10:03:27.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.946+0000 7fcf225dd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf1c0c76d0 0x7fcf1c152da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:27.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.946+0000 7fcf225dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c06aed0 0x7fcf1c1532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.946+0000 7fcf225dd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0c1060 con 0x7fcf1c0c72f0 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.946+0000 7fcf225dd640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c0adf70 con 0x7fcf1c0c76d0 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.946+0000 7fcf225dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c0af730 con 0x7fcf1c06aed0 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf1c0c76d0 0x7fcf1c152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fcf1c0c76d0 0x7fcf1c152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38602/0 (socket says 172.21.15.71:38602) 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 -- 172.21.15.71:0/47715465 learned_addr learned my addr 172.21.15.71:0/47715465 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:27.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf21ddc640 1 --2- 172.21.15.71:0/47715465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c06aed0 0x7fcf1c1532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 -- 172.21.15.71:0/47715465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c06aed0 msgr2=0x7fcf1c1532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 --2- 172.21.15.71:0/47715465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c06aed0 0x7fcf1c1532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 -- 172.21.15.71:0/47715465 >> v1:172.21.15.164:6789/0 conn(0x7fcf1c0c72f0 legacy=0x7fcf1c0acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:27.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.947+0000 7fcf20dda640 1 -- 172.21.15.71:0/47715465 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c06b4a0 con 0x7fcf1c0c76d0 2023-08-25T10:03:27.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.949+0000 7fcf20dda640 1 --2- 172.21.15.71:0/47715465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c0c76d0 0x7fcf1c152da0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00bb10 tx=0x7fcf0c02e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fad4cdb5684c57 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:27.952 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.949+0000 7fcf13fff640 1 -- 172.21.15.71:0/47715465 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c036440 con 0x7fcf1c0c76d0 2023-08-25T10:03:27.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.949+0000 7fcf13fff640 1 -- 172.21.15.71:0/47715465 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0365a0 con 0x7fcf1c0c76d0 2023-08-25T10:03:27.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.949+0000 7fcf13fff640 1 -- 172.21.15.71:0/47715465 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c036930 con 0x7fcf1c0c76d0 2023-08-25T10:03:27.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 -- 172.21.15.71:0/47715465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c0c76d0 msgr2=0x7fcf1c152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:27.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 --2- 172.21.15.71:0/47715465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c0c76d0 0x7fcf1c152da0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00bb10 tx=0x7fcf0c02e1f0 comp rx=0 tx=0).stop 2023-08-25T10:03:27.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 -- 172.21.15.71:0/47715465 shutdown_connections 2023-08-25T10:03:27.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 --2- 172.21.15.71:0/47715465 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c06aed0 0x7fcf1c1532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:27.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 --2- 172.21.15.71:0/47715465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c0c76d0 0x7fcf1c152da0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:27.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 -- 172.21.15.71:0/47715465 >> 172.21.15.71:0/47715465 conn(0x7fcf1c06c280 msgr2=0x7fcf1c06c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:27.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 -- 172.21.15.71:0/47715465 shutdown_connections 2023-08-25T10:03:27.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.950+0000 7fcf225dd640 1 -- 172.21.15.71:0/47715465 wait complete. 2023-08-25T10:03:27.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.952+0000 7fcf225dd640 1 Processor -- start 2023-08-25T10:03:27.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.952+0000 7fcf225dd640 1 -- start start 2023-08-25T10:03:27.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.953+0000 7fcf225dd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 0x7fcf1c0ab590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:27.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.953+0000 7fcf225dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf1c0c72f0 0x7fcf1c154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:27.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.953+0000 7fcf225dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c0c76d0 0x7fcf1c1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:27.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.953+0000 7fcf225dd640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1f16c0 con 0x7fcf1c06aed0 2023-08-25T10:03:27.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.953+0000 7fcf225dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1f1980 con 0x7fcf1c0c72f0 2023-08-25T10:03:27.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.953+0000 7fcf225dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1f1c60 con 0x7fcf1c0c76d0 2023-08-25T10:03:27.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 0x7fcf1c0ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:27.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 0x7fcf1c0ab590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38612/0 (socket says 172.21.15.71:38612) 2023-08-25T10:03:27.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 -- 172.21.15.71:0/1704672927 learned_addr learned my addr 172.21.15.71:0/1704672927 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:27.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf21ddc640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c0c76d0 0x7fcf1c1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:27.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 -- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c0c76d0 msgr2=0x7fcf1c1553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:27.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c0c76d0 0x7fcf1c1553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf20dda640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf1c0c72f0 0x7fcf1c154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 -- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf1c0c72f0 msgr2=0x7fcf1c154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf1c0c72f0 0x7fcf1c154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.954+0000 7fcf215db640 1 -- 172.21.15.71:0/1704672927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c00b770 con 0x7fcf1c06aed0 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.955+0000 7fcf215db640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 0x7fcf1c0ab590 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c450 tx=0x7fcf1800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.955+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180052d0 con 0x7fcf1c06aed0 2023-08-25T10:03:27.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.955+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf18005430 con 0x7fcf1c06aed0 2023-08-25T10:03:27.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.955+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18012870 con 0x7fcf1c06aed0 2023-08-25T10:03:27.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.955+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c1f1f40 con 0x7fcf1c06aed0 2023-08-25T10:03:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.955+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c0abad0 con 0x7fcf1c06aed0 2023-08-25T10:03:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.956+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c0c0ca0 con 0x7fcf1c06aed0 2023-08-25T10:03:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.957+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fcf180129d0 con 0x7fcf1c06aed0 2023-08-25T10:03:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.959+0000 7fcf11ffb640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf0006a080 0x7fcf0006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.959+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fcf18088600 con 0x7fcf1c06aed0 2023-08-25T10:03:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.959+0000 7fcf20dda640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf0006a080 0x7fcf0006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:27.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.960+0000 7fcf20dda640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf0006a080 0x7fcf0006c540 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c038000 tx=0x7fcf0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:27.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:27.964+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fcf18053820 con 0x7fcf1c06aed0 2023-08-25T10:03:28.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.222+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf1c04e420 con 0x7fcf1c06aed0 2023-08-25T10:03:28.226 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.223+0000 7fcf11ffb640 1 -- 172.21.15.71:0/1704672927 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fcf18056e40 con 0x7fcf1c06aed0 2023-08-25T10:03:28.227 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:28.227 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:28.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.225+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf0006a080 msgr2=0x7fcf0006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:28.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.225+0000 7fcf225dd640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf0006a080 0x7fcf0006c540 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c038000 tx=0x7fcf0c008040 comp rx=0 tx=0).stop 2023-08-25T10:03:28.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.225+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 msgr2=0x7fcf1c0ab590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:28.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.225+0000 7fcf225dd640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 0x7fcf1c0ab590 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c450 tx=0x7fcf1800c910 comp rx=0 tx=0).stop 2023-08-25T10:03:28.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 shutdown_connections 2023-08-25T10:03:28.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcf1c0c72f0 0x7fcf1c154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:28.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fcf0006a080 0x7fcf0006c540 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:28.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcf1c0c76d0 0x7fcf1c1553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:28.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 --2- 172.21.15.71:0/1704672927 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcf1c06aed0 0x7fcf1c0ab590 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:28.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 >> 172.21.15.71:0/1704672927 conn(0x7fcf1c06c280 msgr2=0x7fcf1c06a550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:28.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 shutdown_connections 2023-08-25T10:03:28.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:28.226+0000 7fcf225dd640 1 -- 172.21.15.71:0/1704672927 wait complete. 2023-08-25T10:03:28.247 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:30.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:30.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.504+0000 7fdb5f59e640 1 Processor -- start 2023-08-25T10:03:30.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.504+0000 7fdb5f59e640 1 -- start start 2023-08-25T10:03:30.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb601562f0 0x7fdb600c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5f59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdb600c3720 0x7fdb60065780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5f59e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdb600c4620 0x7fdb60065cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5f59e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600bfd10 con 0x7fdb600c4620 2023-08-25T10:03:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5f59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600c7260 con 0x7fdb600c3720 2023-08-25T10:03:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600c6dd0 con 0x7fdb601562f0 2023-08-25T10:03:30.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdb600c4620 0x7fdb60065cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fdb600c4620 0x7fdb60065cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38308/0 (socket says 172.21.15.71:38308) 2023-08-25T10:03:30.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5ed9d640 1 -- 172.21.15.71:0/1181432369 learned_addr learned my addr 172.21.15.71:0/1181432369 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:30.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.506+0000 7fdb5e59c640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb601562f0 0x7fdb600c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5dd9b640 1 --2- 172.21.15.71:0/1181432369 >> v2:172.21.15.164:3300/0 conn(0x7fdb600c3720 0x7fdb60065780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5ed9d640 1 -- 172.21.15.71:0/1181432369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb601562f0 msgr2=0x7fdb600c40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:30.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5ed9d640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb601562f0 0x7fdb600c40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5ed9d640 1 -- 172.21.15.71:0/1181432369 >> v2:172.21.15.164:3300/0 conn(0x7fdb600c3720 msgr2=0x7fdb60065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:30.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5ed9d640 1 --2- 172.21.15.71:0/1181432369 >> v2:172.21.15.164:3300/0 conn(0x7fdb600c3720 0x7fdb60065780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5ed9d640 1 -- 172.21.15.71:0/1181432369 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb601566d0 con 0x7fdb600c4620 2023-08-25T10:03:30.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.507+0000 7fdb5e59c640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb601562f0 0x7fdb600c40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:03:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.508+0000 7fdb5ed9d640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c4620 0x7fdb60065cc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdb600cd390 tx=0x7fdb5802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad368463f7842fc5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5d59a640 1 -- 172.21.15.71:0/1181432369 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5802cb00 con 0x7fdb600c4620 2023-08-25T10:03:30.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5d59a640 1 -- 172.21.15.71:0/1181432369 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5802cc60 con 0x7fdb600c4620 2023-08-25T10:03:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5d59a640 1 -- 172.21.15.71:0/1181432369 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb580343f0 con 0x7fdb600c4620 2023-08-25T10:03:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1181432369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c4620 msgr2=0x7fdb60065cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c4620 0x7fdb60065cc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdb600cd390 tx=0x7fdb5802bf60 comp rx=0 tx=0).stop 2023-08-25T10:03:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1181432369 shutdown_connections 2023-08-25T10:03:30.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c4620 0x7fdb60065cc0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1181432369 >> v2:172.21.15.164:3300/0 conn(0x7fdb600c3720 0x7fdb60065780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1181432369 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb601562f0 0x7fdb600c40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1181432369 >> 172.21.15.71:0/1181432369 conn(0x7fdb60070530 msgr2=0x7fdb60070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1181432369 shutdown_connections 2023-08-25T10:03:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.509+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1181432369 wait complete. 2023-08-25T10:03:30.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.511+0000 7fdb5f59e640 1 Processor -- start 2023-08-25T10:03:30.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.511+0000 7fdb5f59e640 1 -- start start 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.512+0000 7fdb5f59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 0x7fdb600cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.512+0000 7fdb5f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb600c4620 0x7fdb600c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.512+0000 7fdb5f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb601562f0 0x7fdb600cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.512+0000 7fdb5f59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600ca2c0 con 0x7fdb600c3720 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.512+0000 7fdb5f59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600ccae0 con 0x7fdb601562f0 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.512+0000 7fdb5f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600ccdc0 con 0x7fdb600c4620 2023-08-25T10:03:30.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 0x7fdb600cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 0x7fdb600cbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38314/0 (socket says 172.21.15.71:38314) 2023-08-25T10:03:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 -- 172.21.15.71:0/1204748860 learned_addr learned my addr 172.21.15.71:0/1204748860 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5dd9b640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb600c4620 0x7fdb600c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5ed9d640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb601562f0 0x7fdb600cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 -- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb600c4620 msgr2=0x7fdb600c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:30.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb600c4620 0x7fdb600c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 -- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb601562f0 msgr2=0x7fdb600cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb601562f0 0x7fdb600cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5e59c640 1 -- 172.21.15.71:0/1204748860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58009b30 con 0x7fdb600c3720 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.513+0000 7fdb5dd9b640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb600c4620 0x7fdb600c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.514+0000 7fdb5e59c640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 0x7fdb600cbfa0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e7e0 tx=0x7fdb5000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.514+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb50004030 con 0x7fdb600c3720 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.514+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb50004190 con 0x7fdb600c3720 2023-08-25T10:03:30.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.514+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb600ca560 con 0x7fdb600c3720 2023-08-25T10:03:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.514+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb50004520 con 0x7fdb600c3720 2023-08-25T10:03:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.514+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb600cc4e0 con 0x7fdb600c3720 2023-08-25T10:03:30.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.515+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6004e420 con 0x7fdb600c3720 2023-08-25T10:03:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.517+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fdb5000c6d0 con 0x7fdb600c3720 2023-08-25T10:03:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.518+0000 7fdb4effd640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdb4006a050 0x7fdb4006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.518+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fdb50088730 con 0x7fdb600c3720 2023-08-25T10:03:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.519+0000 7fdb5dd9b640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdb4006a050 0x7fdb4006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:30.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.520+0000 7fdb5dd9b640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdb4006a050 0x7fdb4006c510 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdb54006fd0 tx=0x7fdb54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:30.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.522+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fdb50053950 con 0x7fdb600c3720 2023-08-25T10:03:30.777 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.774+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb600c7410 con 0x7fdb600c3720 2023-08-25T10:03:30.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.775+0000 7fdb4effd640 1 -- 172.21.15.71:0/1204748860 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fdb50056f70 con 0x7fdb600c3720 2023-08-25T10:03:30.780 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:30.780 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:30.782 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:30.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdb4006a050 msgr2=0x7fdb4006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:30.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdb4006a050 0x7fdb4006c510 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdb54006fd0 tx=0x7fdb54008040 comp rx=0 tx=0).stop 2023-08-25T10:03:30.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 msgr2=0x7fdb600cbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:30.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 0x7fdb600cbfa0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e7e0 tx=0x7fdb5000eca0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 shutdown_connections 2023-08-25T10:03:30.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb601562f0 0x7fdb600cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fdb4006a050 0x7fdb4006c510 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb600c4620 0x7fdb600c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 --2- 172.21.15.71:0/1204748860 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdb600c3720 0x7fdb600cbfa0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:30.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 >> 172.21.15.71:0/1204748860 conn(0x7fdb60070530 msgr2=0x7fdb601577a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:30.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 shutdown_connections 2023-08-25T10:03:30.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:30.778+0000 7fdb5f59e640 1 -- 172.21.15.71:0/1204748860 wait complete. 2023-08-25T10:03:30.804 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:32.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:33.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.033+0000 7f7c62263640 1 Processor -- start 2023-08-25T10:03:33.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.033+0000 7f7c62263640 1 -- start start 2023-08-25T10:03:33.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.035+0000 7f7c62263640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7c5c152d80 0x7f7c5c0c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:33.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.035+0000 7f7c62263640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c5c0cacb0 0x7f7c5c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:33.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c62263640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c1591c0 con 0x7f7c5c15c740 2023-08-25T10:03:33.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c62263640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c1594a0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c62263640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c159780 con 0x7f7c5c152d80 2023-08-25T10:03:33.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c5b7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c5c0cacb0 0x7f7c5c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:33.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c5b7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c5c0cacb0 0x7f7c5c0c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38328/0 (socket says 172.21.15.71:38328) 2023-08-25T10:03:33.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c5b7fe640 1 -- 172.21.15.71:0/590609242 learned_addr learned my addr 172.21.15.71:0/590609242 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:33.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c5bfff640 1 --2- 172.21.15.71:0/590609242 >> v2:172.21.15.164:3300/0 conn(0x7f7c5c152d80 0x7f7c5c0c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:33.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.036+0000 7f7c5b7fe640 1 -- 172.21.15.71:0/590609242 >> v2:172.21.15.164:3300/0 conn(0x7f7c5c152d80 msgr2=0x7f7c5c0c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:33.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.037+0000 7f7c5b7fe640 1 --2- 172.21.15.71:0/590609242 >> v2:172.21.15.164:3300/0 conn(0x7f7c5c152d80 0x7f7c5c0c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.037+0000 7f7c5b7fe640 1 -- 172.21.15.71:0/590609242 >> v1:172.21.15.71:6789/0 conn(0x7f7c5c15c740 legacy=0x7f7c5c15cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:33.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.037+0000 7f7c5b7fe640 1 -- 172.21.15.71:0/590609242 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c159a60 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.037+0000 7f7c5affd640 1 -- 172.21.15.71:0/590609242 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807094256 0 0) 0x7f7c5c1591c0 con 0x7f7c5c15c740 2023-08-25T10:03:33.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.038+0000 7f7c5b7fe640 1 --2- 172.21.15.71:0/590609242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c0c98a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c06bfa0 tx=0x7f7c5002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2526adc6881b7849 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:33.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.038+0000 7f7c5affd640 1 -- 172.21.15.71:0/590609242 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5002f850 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.038+0000 7f7c5affd640 1 -- 172.21.15.71:0/590609242 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5002f9b0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.038+0000 7f7c5affd640 1 -- 172.21.15.71:0/590609242 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5002fd40 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 -- 172.21.15.71:0/590609242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 msgr2=0x7f7c5c0c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:33.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 --2- 172.21.15.71:0/590609242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c0c98a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c06bfa0 tx=0x7f7c5002ba30 comp rx=0 tx=0).stop 2023-08-25T10:03:33.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 -- 172.21.15.71:0/590609242 shutdown_connections 2023-08-25T10:03:33.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 --2- 172.21.15.71:0/590609242 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c0c98a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 --2- 172.21.15.71:0/590609242 >> v2:172.21.15.164:3300/0 conn(0x7f7c5c152d80 0x7f7c5c0c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 -- 172.21.15.71:0/590609242 >> 172.21.15.71:0/590609242 conn(0x7f7c5c06b8f0 msgr2=0x7f7c5c0654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:33.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 -- 172.21.15.71:0/590609242 shutdown_connections 2023-08-25T10:03:33.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.039+0000 7f7c62263640 1 -- 172.21.15.71:0/590609242 wait complete. 2023-08-25T10:03:33.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.041+0000 7f7c62263640 1 Processor -- start 2023-08-25T10:03:33.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.041+0000 7f7c62263640 1 -- start start 2023-08-25T10:03:33.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c62263640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c1f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c62263640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c5c152d80 0x7f7c5c1483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c62263640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c5c15c740 0x7f7c5c15b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c62263640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0cd660 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c62263640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c0cde70 con 0x7f7c5c152d80 2023-08-25T10:03:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c62263640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c15b180 con 0x7f7c5c15c740 2023-08-25T10:03:33.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c1f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c1f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38338/0 (socket says 172.21.15.71:38338) 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 -- 172.21.15.71:0/3952123395 learned_addr learned my addr 172.21.15.71:0/3952123395 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 -- 172.21.15.71:0/3952123395 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c5c15c740 msgr2=0x7f7c5c15b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c5c15c740 0x7f7c5c15b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 -- 172.21.15.71:0/3952123395 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c5c152d80 msgr2=0x7f7c5c1483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c5c152d80 0x7f7c5c1483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.043+0000 7f7c5bfff640 1 -- 172.21.15.71:0/3952123395 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c500094d0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.044+0000 7f7c5bfff640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c1f22c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00d800 tx=0x7f7c4c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.044+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00b6c0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.044+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4c00b820 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.044+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c00bbb0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.044+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c1f2800 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.044+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c0cd0a0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.046+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0b1a60 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.046+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7c4c00bd10 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.047+0000 7f7c58ff9640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c4806a080 0x7f7c4806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:33.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.047+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f7c4c088fe0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.048+0000 7f7c5b7fe640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c4806a080 0x7f7c4806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:33.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.049+0000 7f7c5b7fe640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c4806a080 0x7f7c4806c540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7c50036000 tx=0x7f7c50039040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:33.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.053+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7c4c0541b0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.346+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c5c069160 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.347+0000 7f7c58ff9640 1 -- 172.21.15.71:0/3952123395 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f7c4c0577d0 con 0x7f7c5c0cacb0 2023-08-25T10:03:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:33.350 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:33.351 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:33.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c4806a080 msgr2=0x7f7c4806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c4806a080 0x7f7c4806c540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7c50036000 tx=0x7f7c50039040 comp rx=0 tx=0).stop 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 msgr2=0x7f7c5c1f22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c1f22c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00d800 tx=0x7f7c4c00dcc0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 shutdown_connections 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c5c152d80 0x7f7c5c1483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c4806a080 0x7f7c4806c540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c5c15c740 0x7f7c5c15b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 --2- 172.21.15.71:0/3952123395 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c5c0cacb0 0x7f7c5c1f22c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 >> 172.21.15.71:0/3952123395 conn(0x7f7c5c06b8f0 msgr2=0x7f7c5c0c35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 shutdown_connections 2023-08-25T10:03:33.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:33.350+0000 7f7c62263640 1 -- 172.21.15.71:0/3952123395 wait complete. 2023-08-25T10:03:33.371 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:35.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:35.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.619+0000 7f4b54657640 1 Processor -- start 2023-08-25T10:03:35.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.620+0000 7f4b54657640 1 -- start start 2023-08-25T10:03:35.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.621+0000 7f4b54657640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4b4c159fc0 0x7f4b4c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.621+0000 7f4b54657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4c06a960 con 0x7f4b4c1594d0 2023-08-25T10:03:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b54657640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4c06ac20 con 0x7f4b4c06cc70 2023-08-25T10:03:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b54657640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4c06af00 con 0x7f4b4c159fc0 2023-08-25T10:03:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b523f3640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4b4c1594d0 0x7f4b4c14f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:40268/0 (socket says 172.21.15.71:40268) 2023-08-25T10:03:35.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b523f3640 1 -- 172.21.15.71:0/2817031323 learned_addr learned my addr 172.21.15.71:0/2817031323 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b52bf4640 1 --2- 172.21.15.71:0/2817031323 >> v2:172.21.15.164:3300/0 conn(0x7f4b4c159fc0 0x7f4b4c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325090303 0 0) 0x7f4b4c06a960 con 0x7f4b4c1594d0 2023-08-25T10:03:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.622+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b38003520 con 0x7f4b4c1594d0 2023-08-25T10:03:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058067115 0 0) 0x7f4b38003520 con 0x7f4b4c1594d0 2023-08-25T10:03:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 >> v2:172.21.15.164:3300/0 conn(0x7f4b4c159fc0 msgr2=0x7f4b4c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:35.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 --2- 172.21.15.71:0/2817031323 >> v2:172.21.15.164:3300/0 conn(0x7f4b4c159fc0 0x7f4b4c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 >> v1:172.21.15.164:6789/0 conn(0x7f4b4c06cc70 legacy=0x7f4b4c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c0c2e60 con 0x7f4b4c1594d0 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4b48001670 con 0x7f4b4c1594d0 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 >> v1:172.21.15.71:6789/0 conn(0x7f4b4c1594d0 legacy=0x7f4b4c14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b380047c0 0x7f4b38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b38005180 0x7f4b3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c06af00 con 0x7f4b380039e0 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3800c090 con 0x7f4b380047c0 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3800c250 con 0x7f4b38005180 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.623+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b52bf4640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b38005180 0x7f4b3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b523f3640 1 -- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b38005180 msgr2=0x7f4b3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:35.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b51bf2640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b380047c0 0x7f4b38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b38005180 0x7f4b3800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b523f3640 1 -- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b380047c0 msgr2=0x7f4b38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b380047c0 0x7f4b38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b523f3640 1 -- 172.21.15.71:0/2817031323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b38003520 con 0x7f4b380039e0 2023-08-25T10:03:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b51bf2640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b380047c0 0x7f4b38004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:35.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.624+0000 7f4b52bf4640 1 -- 172.21.15.71:0/2817031323 reap_dead start 2023-08-25T10:03:35.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.625+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b38003e00 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800c190 tx=0x7f4b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4670d02a1722f8fe server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.625+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48005ad0 con 0x7f4b380039e0 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.625+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b48005c50 con 0x7f4b380039e0 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.625+0000 7f4b513f1640 1 -- 172.21.15.71:0/2817031323 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4802c7e0 con 0x7f4b380039e0 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 -- 172.21.15.71:0/2817031323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 msgr2=0x7f4b38003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b38003e00 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800c190 tx=0x7f4b380057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 -- 172.21.15.71:0/2817031323 shutdown_connections 2023-08-25T10:03:35.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 --2- 172.21.15.71:0/2817031323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b38003e00 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 -- 172.21.15.71:0/2817031323 >> 172.21.15.71:0/2817031323 conn(0x7f4b4c0654f0 msgr2=0x7f4b4c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 -- 172.21.15.71:0/2817031323 shutdown_connections 2023-08-25T10:03:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.626+0000 7f4b54657640 1 -- 172.21.15.71:0/2817031323 wait complete. 2023-08-25T10:03:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.629+0000 7f4b54657640 1 Processor -- start 2023-08-25T10:03:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.629+0000 7f4b54657640 1 -- start start 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b54657640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b4c1ee8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b54657640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b4c0c62f0 0x7f4b4c14dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b54657640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b4c14b0c0 0x7f4b4c14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b54657640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c1eee10 con 0x7f4b380039e0 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b54657640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0c91d0 con 0x7f4b4c14b0c0 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b54657640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0c9b50 con 0x7f4b4c0c62f0 2023-08-25T10:03:35.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b523f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b4c1ee8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b523f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b4c1ee8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38348/0 (socket says 172.21.15.71:38348) 2023-08-25T10:03:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.630+0000 7f4b523f3640 1 -- 172.21.15.71:0/2967074767 learned_addr learned my addr 172.21.15.71:0/2967074767 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b51bf2640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b4c0c62f0 0x7f4b4c14dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b52bf4640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b4c14b0c0 0x7f4b4c14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b523f3640 1 -- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b4c0c62f0 msgr2=0x7f4b4c14dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:35.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b4c0c62f0 0x7f4b4c14dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b523f3640 1 -- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b4c14b0c0 msgr2=0x7f4b4c14b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b4c14b0c0 0x7f4b4c14b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b523f3640 1 -- 172.21.15.71:0/2967074767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c1588b0 con 0x7f4b380039e0 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b52bf4640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b4c14b0c0 0x7f4b4c14b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b523f3640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b4c1ee8d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800c190 tx=0x7f4b4802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4803c040 con 0x7f4b380039e0 2023-08-25T10:03:35.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b4802cbf0 con 0x7f4b380039e0 2023-08-25T10:03:35.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4c158b90 con 0x7f4b380039e0 2023-08-25T10:03:35.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.631+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b48007070 con 0x7f4b380039e0 2023-08-25T10:03:35.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.632+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c158e70 con 0x7f4b380039e0 2023-08-25T10:03:35.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.633+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4c14bfc0 con 0x7f4b380039e0 2023-08-25T10:03:35.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.634+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4b48032070 con 0x7f4b380039e0 2023-08-25T10:03:35.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.635+0000 7f4b42ffd640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4b3006a050 0x7f4b3006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:35.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.636+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4b480a9fa0 con 0x7f4b380039e0 2023-08-25T10:03:35.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.636+0000 7f4b51bf2640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4b3006a050 0x7f4b3006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:35.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.637+0000 7f4b51bf2640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4b3006a050 0x7f4b3006c510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0ca0e0 tx=0x7f4b3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:35.643 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.640+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4b480751e0 con 0x7f4b380039e0 2023-08-25T10:03:35.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.908+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b4c069160 con 0x7f4b380039e0 2023-08-25T10:03:35.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.908+0000 7f4b42ffd640 1 -- 172.21.15.71:0/2967074767 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4b48046020 con 0x7f4b380039e0 2023-08-25T10:03:35.913 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:35.913 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4b3006a050 msgr2=0x7f4b3006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4b3006a050 0x7f4b3006c510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0ca0e0 tx=0x7f4b3c009040 comp rx=0 tx=0).stop 2023-08-25T10:03:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 msgr2=0x7f4b4c1ee8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b4c1ee8d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800c190 tx=0x7f4b4802e070 comp rx=0 tx=0).stop 2023-08-25T10:03:35.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 shutdown_connections 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b4c14b0c0 0x7f4b4c14b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4b3006a050 0x7f4b3006c510 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b4c0c62f0 0x7f4b4c14dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 --2- 172.21.15.71:0/2967074767 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4b380039e0 0x7f4b4c1ee8d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 >> 172.21.15.71:0/2967074767 conn(0x7f4b4c0654f0 msgr2=0x7f4b4c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 shutdown_connections 2023-08-25T10:03:35.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:35.911+0000 7f4b54657640 1 -- 172.21.15.71:0/2967074767 wait complete. 2023-08-25T10:03:35.933 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:03:35.934 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:03:37.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:38.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.157+0000 7f8482a1f640 1 Processor -- start 2023-08-25T10:03:38.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.158+0000 7f8482a1f640 1 -- start start 2023-08-25T10:03:38.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.159+0000 7f8482a1f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f847c0c3660 0x7f847c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:38.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.159+0000 7f8482a1f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f847c1509d0 0x7f847c150de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:38.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.159+0000 7f8482a1f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0c1dc0 con 0x7f847c0c3a40 2023-08-25T10:03:38.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.159+0000 7f8482a1f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c157a80 con 0x7f847c1509d0 2023-08-25T10:03:38.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.159+0000 7f8482a1f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c151320 con 0x7f847c0c3660 2023-08-25T10:03:38.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f847c1509d0 0x7f847c150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:38.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f847c1509d0 0x7f847c150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38362/0 (socket says 172.21.15.71:38362) 2023-08-25T10:03:38.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 -- 172.21.15.71:0/3488471049 learned_addr learned my addr 172.21.15.71:0/3488471049 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:38.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f847bfff640 1 --2- 172.21.15.71:0/3488471049 >> v2:172.21.15.164:3300/0 conn(0x7f847c0c3660 0x7f847c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:38.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 -- 172.21.15.71:0/3488471049 >> v2:172.21.15.164:3300/0 conn(0x7f847c0c3660 msgr2=0x7f847c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:38.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 --2- 172.21.15.71:0/3488471049 >> v2:172.21.15.164:3300/0 conn(0x7f847c0c3660 0x7f847c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 -- 172.21.15.71:0/3488471049 >> v1:172.21.15.71:6789/0 conn(0x7f847c0c3a40 legacy=0x7f847c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.160+0000 7f8480fbc640 1 -- 172.21.15.71:0/3488471049 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c156e00 con 0x7f847c1509d0 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.162+0000 7f8480fbc640 1 --2- 172.21.15.71:0/3488471049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c1509d0 0x7f847c150de0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f847c066440 tx=0x7f847402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23d6804d1fe1900 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.162+0000 7f847affd640 1 -- 172.21.15.71:0/3488471049 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847402cbb0 con 0x7f847c1509d0 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.162+0000 7f847affd640 1 -- 172.21.15.71:0/3488471049 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847402cd10 con 0x7f847c1509d0 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.162+0000 7f847affd640 1 -- 172.21.15.71:0/3488471049 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474034610 con 0x7f847c1509d0 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.162+0000 7f8482a1f640 1 -- 172.21.15.71:0/3488471049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c1509d0 msgr2=0x7f847c150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:38.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 --2- 172.21.15.71:0/3488471049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c1509d0 0x7f847c150de0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f847c066440 tx=0x7f847402c010 comp rx=0 tx=0).stop 2023-08-25T10:03:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 -- 172.21.15.71:0/3488471049 shutdown_connections 2023-08-25T10:03:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 --2- 172.21.15.71:0/3488471049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c1509d0 0x7f847c150de0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 --2- 172.21.15.71:0/3488471049 >> v2:172.21.15.164:3300/0 conn(0x7f847c0c3660 0x7f847c0c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 -- 172.21.15.71:0/3488471049 >> 172.21.15.71:0/3488471049 conn(0x7f847c0654f0 msgr2=0x7f847c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 -- 172.21.15.71:0/3488471049 shutdown_connections 2023-08-25T10:03:38.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.163+0000 7f8482a1f640 1 -- 172.21.15.71:0/3488471049 wait complete. 2023-08-25T10:03:38.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.164+0000 7f8482a1f640 1 Processor -- start 2023-08-25T10:03:38.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.165+0000 7f8482a1f640 1 -- start start 2023-08-25T10:03:38.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8482a1f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 0x7f847c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:38.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8482a1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f847c0c3a40 0x7f847c06fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8482a1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f847c1509d0 0x7f847c159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8482a1f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c158540 con 0x7f847c0c3660 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8482a1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c157fc0 con 0x7f847c0c3a40 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8482a1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c158260 con 0x7f847c1509d0 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 0x7f847c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 0x7f847c15aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:38374/0 (socket says 172.21.15.71:38374) 2023-08-25T10:03:38.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 -- 172.21.15.71:0/1414017672 learned_addr learned my addr 172.21.15.71:0/1414017672 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 -- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f847c1509d0 msgr2=0x7f847c159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f8480fbc640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f847c1509d0 0x7f847c159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847b7fe640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f847c0c3a40 0x7f847c06fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f847c1509d0 0x7f847c159d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 -- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f847c0c3a40 msgr2=0x7f847c06fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f847c0c3a40 0x7f847c06fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.166+0000 7f847bfff640 1 -- 172.21.15.71:0/1414017672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474009b30 con 0x7f847c0c3660 2023-08-25T10:03:38.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f847b7fe640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f847c0c3a40 0x7f847c06fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:38.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f847bfff640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 0x7f847c15aab0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d5f0 tx=0x7f846c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:38.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0125f0 con 0x7f847c0c3660 2023-08-25T10:03:38.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c012750 con 0x7f847c0c3660 2023-08-25T10:03:38.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c06a9d0 con 0x7f847c0c3660 2023-08-25T10:03:38.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c012b00 con 0x7f847c0c3660 2023-08-25T10:03:38.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.167+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c06af90 con 0x7f847c0c3660 2023-08-25T10:03:38.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.169+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c0c29c0 con 0x7f847c0c3660 2023-08-25T10:03:38.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.169+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f846c012c60 con 0x7f847c0c3660 2023-08-25T10:03:38.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.171+0000 7f8478ff9640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f846806a050 0x7f846806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:38.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.171+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f846c088f20 con 0x7f847c0c3660 2023-08-25T10:03:38.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.171+0000 7f847b7fe640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f846806a050 0x7f846806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:38.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.172+0000 7f847b7fe640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f846806a050 0x7f846806c510 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c9680 tx=0x7f8470009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:38.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.175+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f846c054140 con 0x7f847c0c3660 2023-08-25T10:03:38.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.426+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f847c069160 con 0x7f847c0c3660 2023-08-25T10:03:38.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.428+0000 7f8478ff9640 1 -- 172.21.15.71:0/1414017672 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f846c057760 con 0x7f847c0c3660 2023-08-25T10:03:38.432 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:38.433 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:38.433 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f846806a050 msgr2=0x7f846806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f846806a050 0x7f846806c510 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c9680 tx=0x7f8470009040 comp rx=0 tx=0).stop 2023-08-25T10:03:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 msgr2=0x7f847c15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 0x7f847c15aab0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f846c00d5f0 tx=0x7f846c00dab0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 shutdown_connections 2023-08-25T10:03:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f847c0c3a40 0x7f847c06fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f846806a050 0x7f846806c510 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f847c1509d0 0x7f847c159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 --2- 172.21.15.71:0/1414017672 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f847c0c3660 0x7f847c15aab0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:38.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 >> 172.21.15.71:0/1414017672 conn(0x7f847c0654f0 msgr2=0x7f847c0c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:38.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 shutdown_connections 2023-08-25T10:03:38.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:38.432+0000 7f8482a1f640 1 -- 172.21.15.71:0/1414017672 wait complete. 2023-08-25T10:03:38.453 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:40.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:40.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.697+0000 7f5568263640 1 Processor -- start 2023-08-25T10:03:40.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.698+0000 7f5568263640 1 -- start start 2023-08-25T10:03:40.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.699+0000 7f5568263640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 0x7f55600c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:40.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.699+0000 7f5568263640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f55600c3a40 0x7f556006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:40.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.699+0000 7f5568263640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55600c1dc0 con 0x7f55601509d0 2023-08-25T10:03:40.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.699+0000 7f5568263640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5560157a80 con 0x7f55600c3a40 2023-08-25T10:03:40.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.699+0000 7f5568263640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556006cf70 con 0x7f55600c3660 2023-08-25T10:03:40.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.700+0000 7f5565fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 0x7f55600c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:40.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.700+0000 7f5565fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 0x7f55600c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:37032/0 (socket says 172.21.15.71:37032) 2023-08-25T10:03:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.700+0000 7f5565fff640 1 -- 172.21.15.71:0/1427276463 learned_addr learned my addr 172.21.15.71:0/1427276463 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.700+0000 7f55657fe640 1 --2- 172.21.15.71:0/1427276463 >> v2:172.21.15.164:3300/0 conn(0x7f55600c3a40 0x7f556006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.701+0000 7f5565fff640 1 -- 172.21.15.71:0/1427276463 >> v2:172.21.15.164:3300/0 conn(0x7f55600c3a40 msgr2=0x7f556006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.701+0000 7f5565fff640 1 --2- 172.21.15.71:0/1427276463 >> v2:172.21.15.164:3300/0 conn(0x7f55600c3a40 0x7f556006ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.701+0000 7f5565fff640 1 -- 172.21.15.71:0/1427276463 >> v1:172.21.15.164:6789/0 conn(0x7f55601509d0 legacy=0x7f5560150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:40.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.701+0000 7f5565fff640 1 -- 172.21.15.71:0/1427276463 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560156e00 con 0x7f55600c3660 2023-08-25T10:03:40.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.702+0000 7f5565fff640 1 --2- 172.21.15.71:0/1427276463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 0x7f55600c1710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f555c009870 tx=0x7f555c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d39d9160cd0aa3b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:40.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5564ffd640 1 -- 172.21.15.71:0/1427276463 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c02f840 con 0x7f55600c3660 2023-08-25T10:03:40.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5564ffd640 1 -- 172.21.15.71:0/1427276463 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c02f9a0 con 0x7f55600c3660 2023-08-25T10:03:40.708 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5564ffd640 1 -- 172.21.15.71:0/1427276463 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c02fd30 con 0x7f55600c3660 2023-08-25T10:03:40.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5568263640 1 -- 172.21.15.71:0/1427276463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 msgr2=0x7f55600c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:40.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5568263640 1 --2- 172.21.15.71:0/1427276463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 0x7f55600c1710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f555c009870 tx=0x7f555c02ba20 comp rx=0 tx=0).stop 2023-08-25T10:03:40.709 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5568263640 1 -- 172.21.15.71:0/1427276463 shutdown_connections 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5568263640 1 --2- 172.21.15.71:0/1427276463 >> v2:172.21.15.164:3300/0 conn(0x7f55600c3a40 0x7f556006ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5568263640 1 --2- 172.21.15.71:0/1427276463 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55600c3660 0x7f55600c1710 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.703+0000 7f5568263640 1 -- 172.21.15.71:0/1427276463 >> 172.21.15.71:0/1427276463 conn(0x7f55600654f0 msgr2=0x7f5560065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.704+0000 7f5568263640 1 -- 172.21.15.71:0/1427276463 shutdown_connections 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.704+0000 7f5568263640 1 -- 172.21.15.71:0/1427276463 wait complete. 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.706+0000 7f5568263640 1 Processor -- start 2023-08-25T10:03:40.710 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.706+0000 7f5568263640 1 -- start start 2023-08-25T10:03:40.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5568263640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 0x7f55601539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:40.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5568263640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55600c3a40 0x7f5560153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:40.712 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5568263640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55601509d0 0x7f556014b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:40.713 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5568263640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560152440 con 0x7f55600c3660 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5568263640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560153000 con 0x7f55600c3a40 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5568263640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556015b130 con 0x7f55601509d0 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5565fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 0x7f55601539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5565fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 0x7f55601539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47342/0 (socket says 172.21.15.71:47342) 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.707+0000 7f5565fff640 1 -- 172.21.15.71:0/3051610818 learned_addr learned my addr 172.21.15.71:0/3051610818 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5566800640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55601509d0 0x7f556014b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:40.714 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f55657fe640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55600c3a40 0x7f5560153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5565fff640 1 -- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55601509d0 msgr2=0x7f556014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5565fff640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55601509d0 0x7f556014b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5565fff640 1 -- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55600c3a40 msgr2=0x7f5560153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5565fff640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55600c3a40 0x7f5560153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5565fff640 1 -- 172.21.15.71:0/3051610818 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0094d0 con 0x7f55600c3660 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f5565fff640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 0x7f55601539a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f555c02ba60 tx=0x7f555c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:40.715 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.708+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c037c40 con 0x7f55600c3660 2023-08-25T10:03:40.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.709+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c037da0 con 0x7f55600c3660 2023-08-25T10:03:40.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.709+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55600c9750 con 0x7f55600c3660 2023-08-25T10:03:40.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.709+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c0426c0 con 0x7f55600c3660 2023-08-25T10:03:40.716 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.709+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556006ab60 con 0x7f55600c3660 2023-08-25T10:03:40.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.710+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55600c29c0 con 0x7f55600c3660 2023-08-25T10:03:40.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.711+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f555c02d3d0 con 0x7f55600c3660 2023-08-25T10:03:40.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.712+0000 7f55567fc640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f554406a050 0x7f554406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:40.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.713+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f555c03c070 con 0x7f55600c3660 2023-08-25T10:03:40.717 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.713+0000 7f55657fe640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f554406a050 0x7f554406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:40.718 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.714+0000 7f55657fe640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f554406a050 0x7f554406c510 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f55600c5220 tx=0x7f5550008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:40.720 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:40.718+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f555c0763c0 con 0x7f55600c3660 2023-08-25T10:03:41.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.008+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5560157670 con 0x7f55600c3660 2023-08-25T10:03:41.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.009+0000 7f55567fc640 1 -- 172.21.15.71:0/3051610818 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f555c047090 con 0x7f55600c3660 2023-08-25T10:03:41.014 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:41.014 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:41.015 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:41.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f554406a050 msgr2=0x7f554406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:41.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f554406a050 0x7f554406c510 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f55600c5220 tx=0x7f5550008040 comp rx=0 tx=0).stop 2023-08-25T10:03:41.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 msgr2=0x7f55601539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:41.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 0x7f55601539a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f555c02ba60 tx=0x7f555c004730 comp rx=0 tx=0).stop 2023-08-25T10:03:41.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 shutdown_connections 2023-08-25T10:03:41.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55600c3a40 0x7f5560153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:41.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f554406a050 0x7f554406c510 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:41.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55601509d0 0x7f556014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:41.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 --2- 172.21.15.71:0/3051610818 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55600c3660 0x7f55601539a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:41.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.012+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 >> 172.21.15.71:0/3051610818 conn(0x7f55600654f0 msgr2=0x7f5560065bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:41.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.013+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 shutdown_connections 2023-08-25T10:03:41.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:41.013+0000 7f5568263640 1 -- 172.21.15.71:0/3051610818 wait complete. 2023-08-25T10:03:41.033 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:43.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:43.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.263+0000 7f480fd22640 1 Processor -- start 2023-08-25T10:03:43.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.263+0000 7f480fd22640 1 -- start start 2023-08-25T10:03:43.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.264+0000 7f480fd22640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4808159fc0 0x7f48080c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.264+0000 7f480fd22640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480806a960 con 0x7f48081594d0 2023-08-25T10:03:43.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.264+0000 7f480fd22640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480806ac20 con 0x7f480806cc70 2023-08-25T10:03:43.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.264+0000 7f480fd22640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480806af00 con 0x7f4808159fc0 2023-08-25T10:03:43.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.265+0000 7f480dabe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f48081594d0 0x7f480814f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46204/0 (socket says 172.21.15.71:46204) 2023-08-25T10:03:43.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.265+0000 7f480dabe640 1 -- 172.21.15.71:0/708061909 learned_addr learned my addr 172.21.15.71:0/708061909 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:43.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.265+0000 7f480e2bf640 1 --2- 172.21.15.71:0/708061909 >> v2:172.21.15.164:3300/0 conn(0x7f4808159fc0 0x7f48080c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.265+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880934139 0 0) 0x7f480806a960 con 0x7f48081594d0 2023-08-25T10:03:43.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.265+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f4003520 con 0x7f48081594d0 2023-08-25T10:03:43.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975617396 0 0) 0x7f47f4003520 con 0x7f48081594d0 2023-08-25T10:03:43.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 >> v2:172.21.15.164:3300/0 conn(0x7f4808159fc0 msgr2=0x7f48080c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:43.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 --2- 172.21.15.71:0/708061909 >> v2:172.21.15.164:3300/0 conn(0x7f4808159fc0 0x7f48080c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 >> v1:172.21.15.164:6789/0 conn(0x7f480806cc70 legacy=0x7f48081598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48080c2e60 con 0x7f48081594d0 2023-08-25T10:03:43.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4804001670 con 0x7f48081594d0 2023-08-25T10:03:43.270 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 >> v1:172.21.15.71:6789/0 conn(0x7f48081594d0 legacy=0x7f480814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f47f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47f40047c0 0x7f47f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47f4005180 0x7f47f400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480806af00 con 0x7f47f40039e0 2023-08-25T10:03:43.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f400e2a0 con 0x7f47f4005180 2023-08-25T10:03:43.272 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f400e460 con 0x7f47f40047c0 2023-08-25T10:03:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.266+0000 7f480dabe640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f47f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480d2bd640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47f40047c0 0x7f47f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480e2bf640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47f4005180 0x7f47f400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480dabe640 1 -- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47f40047c0 msgr2=0x7f47f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480dabe640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f47f40047c0 0x7f47f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480dabe640 1 -- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47f4005180 msgr2=0x7f47f400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:43.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480dabe640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f47f4005180 0x7f47f400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480dabe640 1 -- 172.21.15.71:0/708061909 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f4003520 con 0x7f47f40039e0 2023-08-25T10:03:43.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.267+0000 7f480e2bf640 1 -- 172.21.15.71:0/708061909 reap_dead start 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.268+0000 7f480dabe640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f47f4003e00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f480400c190 tx=0x7f47f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84af1eabf784b39 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.268+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804005ad0 con 0x7f47f40039e0 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.268+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4804005c30 con 0x7f47f40039e0 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.268+0000 7f480cabc640 1 -- 172.21.15.71:0/708061909 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480402cab0 con 0x7f47f40039e0 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 -- 172.21.15.71:0/708061909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 msgr2=0x7f47f4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f47f4003e00 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f480400c190 tx=0x7f47f40057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 -- 172.21.15.71:0/708061909 shutdown_connections 2023-08-25T10:03:43.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 --2- 172.21.15.71:0/708061909 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f47f4003e00 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 -- 172.21.15.71:0/708061909 >> 172.21.15.71:0/708061909 conn(0x7f48080654f0 msgr2=0x7f4808065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:43.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 -- 172.21.15.71:0/708061909 shutdown_connections 2023-08-25T10:03:43.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.269+0000 7f480fd22640 1 -- 172.21.15.71:0/708061909 wait complete. 2023-08-25T10:03:43.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.271+0000 7f480fd22640 1 Processor -- start 2023-08-25T10:03:43.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.272+0000 7f480fd22640 1 -- start start 2023-08-25T10:03:43.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480fd22640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f48081ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480fd22640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480814eb70 0x7f48080c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480fd22640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48081ec130 0x7f48081ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480fd22640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48081ee600 con 0x7f47f40039e0 2023-08-25T10:03:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480fd22640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48081ecb20 con 0x7f48081ec130 2023-08-25T10:03:43.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480fd22640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4808144a50 con 0x7f480814eb70 2023-08-25T10:03:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f48081ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f48081ee0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47358/0 (socket says 172.21.15.71:47358) 2023-08-25T10:03:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 -- 172.21.15.71:0/800506002 learned_addr learned my addr 172.21.15.71:0/800506002 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480d2bd640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480814eb70 0x7f48080c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480e2bf640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48081ec130 0x7f48081ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 -- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480814eb70 msgr2=0x7f48080c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480814eb70 0x7f48080c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 -- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48081ec130 msgr2=0x7f48081ec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48081ec130 0x7f48081ec5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.273+0000 7f480dabe640 1 -- 172.21.15.71:0/800506002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480814e740 con 0x7f47f40039e0 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.274+0000 7f480e2bf640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48081ec130 0x7f48081ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.274+0000 7f480dabe640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f48081ee0c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f480400c190 tx=0x7f480402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.274+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480403c040 con 0x7f47f40039e0 2023-08-25T10:03:43.282 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.274+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480402cc80 con 0x7f47f40039e0 2023-08-25T10:03:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.274+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804007070 con 0x7f47f40039e0 2023-08-25T10:03:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.274+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48080c4fb0 con 0x7f47f40039e0 2023-08-25T10:03:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.275+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48080c5570 con 0x7f47f40039e0 2023-08-25T10:03:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.276+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48080c3940 con 0x7f47f40039e0 2023-08-25T10:03:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.277+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4804032070 con 0x7f47f40039e0 2023-08-25T10:03:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.278+0000 7f47fe7fc640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f47ec06a080 0x7f47ec06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.278+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f48040a9be0 con 0x7f47f40039e0 2023-08-25T10:03:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.278+0000 7f480d2bd640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f47ec06a080 0x7f47ec06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:43.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.279+0000 7f480d2bd640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f47ec06a080 0x7f47ec06c540 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f48080c6320 tx=0x7f47f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:43.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.284+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4804074fe0 con 0x7f47f40039e0 2023-08-25T10:03:43.544 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.541+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48080c3110 con 0x7f47f40039e0 2023-08-25T10:03:43.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.542+0000 7f47fe7fc640 1 -- 172.21.15.71:0/800506002 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4804046090 con 0x7f47f40039e0 2023-08-25T10:03:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:43.546 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:43.546 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:43.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.544+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f47ec06a080 msgr2=0x7f47ec06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.544+0000 7f480fd22640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f47ec06a080 0x7f47ec06c540 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f48080c6320 tx=0x7f47f8009040 comp rx=0 tx=0).stop 2023-08-25T10:03:43.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.544+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 msgr2=0x7f48081ee0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:43.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.544+0000 7f480fd22640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f48081ee0c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f480400c190 tx=0x7f480402e070 comp rx=0 tx=0).stop 2023-08-25T10:03:43.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 shutdown_connections 2023-08-25T10:03:43.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48081ec130 0x7f48081ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f47ec06a080 0x7f47ec06c540 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f480814eb70 0x7f48080c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 --2- 172.21.15.71:0/800506002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f47f40039e0 0x7f48081ee0c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 >> 172.21.15.71:0/800506002 conn(0x7f48080654f0 msgr2=0x7f480806c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 shutdown_connections 2023-08-25T10:03:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:43.545+0000 7f480fd22640 1 -- 172.21.15.71:0/800506002 wait complete. 2023-08-25T10:03:43.566 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:45.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:45.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.815+0000 7f544cfcb640 1 Processor -- start 2023-08-25T10:03:45.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.815+0000 7f544cfcb640 1 -- start start 2023-08-25T10:03:45.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.817+0000 7f544cfcb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5448159490 0x7f54480ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.817+0000 7f544cfcb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480b1af0 con 0x7f54480cb570 2023-08-25T10:03:45.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.817+0000 7f544cfcb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54480b0fb0 con 0x7f54480cb190 2023-08-25T10:03:45.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.817+0000 7f544cfcb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54480c98f0 con 0x7f5448159490 2023-08-25T10:03:45.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.817+0000 7f5446ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f54480cb570 0x7f5448158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46216/0 (socket says 172.21.15.71:46216) 2023-08-25T10:03:45.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.817+0000 7f5446ffd640 1 -- 172.21.15.71:0/4204703628 learned_addr learned my addr 172.21.15.71:0/4204703628 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:45.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f5447fff640 1 --2- 172.21.15.71:0/4204703628 >> v2:172.21.15.164:3300/0 conn(0x7f5448159490 0x7f54480ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364343626 0 0) 0x7f54480b1af0 con 0x7f54480cb570 2023-08-25T10:03:45.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5430003520 con 0x7f54480cb570 2023-08-25T10:03:45.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990605511 0 0) 0x7f5430003520 con 0x7f54480cb570 2023-08-25T10:03:45.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 >> v2:172.21.15.164:3300/0 conn(0x7f5448159490 msgr2=0x7f54480ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:45.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 --2- 172.21.15.71:0/4204703628 >> v2:172.21.15.164:3300/0 conn(0x7f5448159490 0x7f54480ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 >> v1:172.21.15.164:6789/0 conn(0x7f54480cb190 legacy=0x7f54480c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:45.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54480b1530 con 0x7f54480cb570 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f543c001670 con 0x7f54480cb570 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 >> v1:172.21.15.71:6789/0 conn(0x7f54480cb570 legacy=0x7f5448158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54300039e0 0x7f5430003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300047c0 0x7f5430004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5430005180 0x7f543000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.818+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54480c98f0 con 0x7f54300039e0 2023-08-25T10:03:45.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f543000c060 con 0x7f54300047c0 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f543000c220 con 0x7f5430005180 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54477fe640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54300039e0 0x7f5430003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f5447fff640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5430005180 0x7f543000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f5446ffd640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300047c0 0x7f5430004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54477fe640 1 -- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5430005180 msgr2=0x7f543000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54477fe640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5430005180 0x7f543000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54477fe640 1 -- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300047c0 msgr2=0x7f5430004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54477fe640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300047c0 0x7f5430004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f54477fe640 1 -- 172.21.15.71:0/4204703628 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54480b1530 con 0x7f54300039e0 2023-08-25T10:03:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f5446ffd640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300047c0 0x7f5430004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:45.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.819+0000 7f5447fff640 1 -- 172.21.15.71:0/4204703628 reap_dead start 2023-08-25T10:03:45.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.821+0000 7f54477fe640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54300039e0 0x7f5430003e20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5438001750 tx=0x7f54300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf1404d47a60648 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:45.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.821+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54380052d0 con 0x7f54300039e0 2023-08-25T10:03:45.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.821+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543802e050 con 0x7f54300039e0 2023-08-25T10:03:45.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.821+0000 7f54467fc640 1 -- 172.21.15.71:0/4204703628 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438003830 con 0x7f54300039e0 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.821+0000 7f544cfcb640 1 -- 172.21.15.71:0/4204703628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54300039e0 msgr2=0x7f5430003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.821+0000 7f544cfcb640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54300039e0 0x7f5430003e20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5438001750 tx=0x7f54300057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.822+0000 7f544cfcb640 1 -- 172.21.15.71:0/4204703628 shutdown_connections 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.822+0000 7f544cfcb640 1 --2- 172.21.15.71:0/4204703628 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54300039e0 0x7f5430003e20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.822+0000 7f544cfcb640 1 -- 172.21.15.71:0/4204703628 >> 172.21.15.71:0/4204703628 conn(0x7f544806ae20 msgr2=0x7f544806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.822+0000 7f544cfcb640 1 -- 172.21.15.71:0/4204703628 shutdown_connections 2023-08-25T10:03:45.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.822+0000 7f544cfcb640 1 -- 172.21.15.71:0/4204703628 wait complete. 2023-08-25T10:03:45.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.824+0000 7f544cfcb640 1 Processor -- start 2023-08-25T10:03:45.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.824+0000 7f544cfcb640 1 -- start start 2023-08-25T10:03:45.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f544cfcb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300039e0 0x7f544814f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f544cfcb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5448150210 0x7f5448069850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f544cfcb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 0x7f5448069d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f544cfcb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54481f2200 con 0x7f54480cd6d0 2023-08-25T10:03:45.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f544cfcb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54481573e0 con 0x7f54300039e0 2023-08-25T10:03:45.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f544cfcb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54481ee080 con 0x7f5448150210 2023-08-25T10:03:45.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f5447fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 0x7f5448069d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f5447fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 0x7f5448069d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47378/0 (socket says 172.21.15.71:47378) 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.825+0000 7f5447fff640 1 -- 172.21.15.71:0/601999549 learned_addr learned my addr 172.21.15.71:0/601999549 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f54477fe640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300039e0 0x7f544814f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5446ffd640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5448150210 0x7f5448069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5447fff640 1 -- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5448150210 msgr2=0x7f5448069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5447fff640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5448150210 0x7f5448069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5447fff640 1 -- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300039e0 msgr2=0x7f544814f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:45.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5447fff640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300039e0 0x7f544814f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:45.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5447fff640 1 -- 172.21.15.71:0/601999549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54481ee360 con 0x7f54480cd6d0 2023-08-25T10:03:45.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.826+0000 7f5447fff640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 0x7f5448069d90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f544806b9d0 tx=0x7f544000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:45.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.827+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440016620 con 0x7f54480cd6d0 2023-08-25T10:03:45.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.827+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54400167a0 con 0x7f54480cd6d0 2023-08-25T10:03:45.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.827+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54481ee640 con 0x7f54480cd6d0 2023-08-25T10:03:45.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.827+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440016b50 con 0x7f54480cd6d0 2023-08-25T10:03:45.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.827+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544814fdc0 con 0x7f54480cd6d0 2023-08-25T10:03:45.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.828+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544804e420 con 0x7f54480cd6d0 2023-08-25T10:03:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.829+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5440016cb0 con 0x7f54480cd6d0 2023-08-25T10:03:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.831+0000 7f542ffff640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a080 0x7f543406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.831+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f5440089590 con 0x7f54480cd6d0 2023-08-25T10:03:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.831+0000 7f54477fe640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a080 0x7f543406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:45.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.832+0000 7f54477fe640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a080 0x7f543406c540 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5438033000 tx=0x7f5438009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:45.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:45.836+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5440054740 con 0x7f54480cd6d0 2023-08-25T10:03:46.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.103+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5448157520 con 0x7f54480cd6d0 2023-08-25T10:03:46.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.104+0000 7f542ffff640 1 -- 172.21.15.71:0/601999549 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f5440057d60 con 0x7f54480cd6d0 2023-08-25T10:03:46.107 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:46.107 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:46.107 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:46.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.106+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a080 msgr2=0x7f543406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:46.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.106+0000 7f544cfcb640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a080 0x7f543406c540 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5438033000 tx=0x7f5438009030 comp rx=0 tx=0).stop 2023-08-25T10:03:46.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.106+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 msgr2=0x7f5448069d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:46.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.106+0000 7f544cfcb640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 0x7f5448069d90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f544806b9d0 tx=0x7f544000f520 comp rx=0 tx=0).stop 2023-08-25T10:03:46.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 shutdown_connections 2023-08-25T10:03:46.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f543406a080 0x7f543406c540 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:46.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f54480cd6d0 0x7f5448069d90 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:46.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5448150210 0x7f5448069850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:46.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 --2- 172.21.15.71:0/601999549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54300039e0 0x7f544814f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:46.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 >> 172.21.15.71:0/601999549 conn(0x7f544806ae20 msgr2=0x7f54480cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:46.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 shutdown_connections 2023-08-25T10:03:46.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:46.107+0000 7f544cfcb640 1 -- 172.21.15.71:0/601999549 wait complete. 2023-08-25T10:03:46.128 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:48.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:48.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.359+0000 7f19f4b4e640 1 Processor -- start 2023-08-25T10:03:48.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.360+0000 7f19f4b4e640 1 -- start start 2023-08-25T10:03:48.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.361+0000 7f19f4b4e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f19f0152d80 0x7f19f00c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:48.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.361+0000 7f19f4b4e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f01591c0 con 0x7f19f00cacb0 2023-08-25T10:03:48.363 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19f4b4e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f01594a0 con 0x7f19f015ce50 2023-08-25T10:03:48.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19f4b4e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f0159780 con 0x7f19f0152d80 2023-08-25T10:03:48.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19ee59c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f19f0152d80 0x7f19f00c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:48.364 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19e5d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f19f00cacb0 0x7f19f015c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46228/0 (socket says 172.21.15.71:46228) 2023-08-25T10:03:48.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19e5d9b640 1 -- 172.21.15.71:0/3289804596 learned_addr learned my addr 172.21.15.71:0/3289804596 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:48.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19ee59c640 1 -- 172.21.15.71:0/3289804596 >> v1:172.21.15.164:6789/0 conn(0x7f19f015ce50 legacy=0x7f19f00cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:48.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19ee59c640 1 -- 172.21.15.71:0/3289804596 >> v1:172.21.15.71:6789/0 conn(0x7f19f00cacb0 legacy=0x7f19f015c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:48.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.362+0000 7f19ee59c640 1 -- 172.21.15.71:0/3289804596 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0159a60 con 0x7f19f0152d80 2023-08-25T10:03:48.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.363+0000 7f19ee59c640 1 --2- 172.21.15.71:0/3289804596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f00c9360 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f19e00097a0 tx=0x7f19e002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bbe8ce2873ff92 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:48.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19edd9b640 1 -- 172.21.15.71:0/3289804596 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0030610 con 0x7f19f0152d80 2023-08-25T10:03:48.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19edd9b640 1 -- 172.21.15.71:0/3289804596 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0030770 con 0x7f19f0152d80 2023-08-25T10:03:48.366 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19edd9b640 1 -- 172.21.15.71:0/3289804596 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0030b00 con 0x7f19f0152d80 2023-08-25T10:03:48.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3289804596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 msgr2=0x7f19f00c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:48.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3289804596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f00c9360 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f19e00097a0 tx=0x7f19e002b960 comp rx=0 tx=0).stop 2023-08-25T10:03:48.367 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3289804596 shutdown_connections 2023-08-25T10:03:48.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.364+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3289804596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f00c9360 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.365+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3289804596 >> 172.21.15.71:0/3289804596 conn(0x7f19f006b8f0 msgr2=0x7f19f00654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:48.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.365+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3289804596 shutdown_connections 2023-08-25T10:03:48.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.365+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3289804596 wait complete. 2023-08-25T10:03:48.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.366+0000 7f19f4b4e640 1 Processor -- start 2023-08-25T10:03:48.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.367+0000 7f19f4b4e640 1 -- start start 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.367+0000 7f19f4b4e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19f00cacb0 0x7f19f015a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.367+0000 7f19f4b4e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f01ee5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19f4b4e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19f015ce50 0x7f19f00cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19f4b4e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015aca0 con 0x7f19f0152d80 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19f4b4e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f01f1b30 con 0x7f19f015ce50 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19f4b4e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f015a6e0 con 0x7f19f00cacb0 2023-08-25T10:03:48.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f01ee5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:48.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f01ee5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:47400/0 (socket says 172.21.15.71:47400) 2023-08-25T10:03:48.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 -- 172.21.15.71:0/3595829393 learned_addr learned my addr 172.21.15.71:0/3595829393 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:48.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19ee59c640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19f00cacb0 0x7f19f015a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:48.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19eed9d640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19f015ce50 0x7f19f00cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 -- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19f00cacb0 msgr2=0x7f19f015a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19f00cacb0 0x7f19f015a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 -- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19f015ce50 msgr2=0x7f19f00cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19f015ce50 0x7f19f00cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19e5d9b640 1 -- 172.21.15.71:0/3595829393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e0009400 con 0x7f19f0152d80 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.368+0000 7f19eed9d640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19f015ce50 0x7f19f00cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.369+0000 7f19e5d9b640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f01ee5c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c4a0 tx=0x7f19d800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:48.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.369+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8005ce0 con 0x7f19f0152d80 2023-08-25T10:03:48.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.369+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8005e40 con 0x7f19f0152d80 2023-08-25T10:03:48.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.369+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f015a9c0 con 0x7f19f0152d80 2023-08-25T10:03:48.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.369+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8007270 con 0x7f19f0152d80 2023-08-25T10:03:48.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.369+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f01f2020 con 0x7f19f0152d80 2023-08-25T10:03:48.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.370+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f00b1a60 con 0x7f19f0152d80 2023-08-25T10:03:48.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.372+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f19d80073d0 con 0x7f19f0152d80 2023-08-25T10:03:48.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.373+0000 7f19e77fe640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19d006a080 0x7f19d006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:48.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.373+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f19d8087db0 con 0x7f19f0152d80 2023-08-25T10:03:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.373+0000 7f19ee59c640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19d006a080 0x7f19d006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:48.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.374+0000 7f19ee59c640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19d006a080 0x7f19d006c540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:48.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.378+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f19d8053120 con 0x7f19f0152d80 2023-08-25T10:03:48.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.648+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19f0069160 con 0x7f19f0152d80 2023-08-25T10:03:48.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.648+0000 7f19e77fe640 1 -- 172.21.15.71:0/3595829393 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f19d8017090 con 0x7f19f0152d80 2023-08-25T10:03:48.650 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:48.650 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:48.651 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:48.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19d006a080 msgr2=0x7f19d006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:48.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19d006a080 0x7f19d006c540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003b040 comp rx=0 tx=0).stop 2023-08-25T10:03:48.653 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 msgr2=0x7f19f01ee5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f01ee5c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c4a0 tx=0x7f19d800c960 comp rx=0 tx=0).stop 2023-08-25T10:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 shutdown_connections 2023-08-25T10:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19f015ce50 0x7f19f00cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f19d006a080 0x7f19d006c540 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f19f0152d80 0x7f19f01ee5c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.654 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 --2- 172.21.15.71:0/3595829393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f19f00cacb0 0x7f19f015a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:48.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 >> 172.21.15.71:0/3595829393 conn(0x7f19f006b8f0 msgr2=0x7f19f00c9730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:48.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 shutdown_connections 2023-08-25T10:03:48.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:48.651+0000 7f19f4b4e640 1 -- 172.21.15.71:0/3595829393 wait complete. 2023-08-25T10:03:48.672 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:03:48.673 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:03:50.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:50.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.890+0000 7f3eb29cb640 1 Processor -- start 2023-08-25T10:03:50.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.890+0000 7f3eb29cb640 1 -- start start 2023-08-25T10:03:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb29cb640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3eac0cb190 0x7f3eac0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb29cb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac15cd00 0x7f3eac158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb29cb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0b1af0 con 0x7f3eac0cb570 2023-08-25T10:03:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb29cb640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eac0c98f0 con 0x7f3eac0cb190 2023-08-25T10:03:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb29cb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac15d170 con 0x7f3eac15cd00 2023-08-25T10:03:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb19c9640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3eac0cb190 0x7f3eac0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb11c8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3eac0cb570 0x7f3eac0c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47602/0 (socket says 172.21.15.71:47602) 2023-08-25T10:03:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.892+0000 7f3eb11c8640 1 -- 172.21.15.71:0/2602152497 learned_addr learned my addr 172.21.15.71:0/2602152497 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.893+0000 7f3eb21ca640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac15cd00 0x7f3eac158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.893+0000 7f3eb19c9640 1 -- 172.21.15.71:0/2602152497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac15cd00 msgr2=0x7f3eac158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.893+0000 7f3eb19c9640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac15cd00 0x7f3eac158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.893+0000 7f3eb19c9640 1 -- 172.21.15.71:0/2602152497 >> v1:172.21.15.71:6789/0 conn(0x7f3eac0cb570 legacy=0x7f3eac0c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.893+0000 7f3eb19c9640 1 -- 172.21.15.71:0/2602152497 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac0b0fb0 con 0x7f3eac0cb190 2023-08-25T10:03:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.893+0000 7f3eb21ca640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac15cd00 0x7f3eac158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:50.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.894+0000 7f3eb19c9640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac0cb190 0x7f3eac0ca910 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80097a0 tx=0x7f3ea802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6c0b02fee078ff server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:50.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.894+0000 7f3eb09c7640 1 -- 172.21.15.71:0/2602152497 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8030800 con 0x7f3eac0cb190 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb09c7640 1 -- 172.21.15.71:0/2602152497 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea8030960 con 0x7f3eac0cb190 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb09c7640 1 -- 172.21.15.71:0/2602152497 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8030cf0 con 0x7f3eac0cb190 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2602152497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac0cb190 msgr2=0x7f3eac0ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac0cb190 0x7f3eac0ca910 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80097a0 tx=0x7f3ea802b940 comp rx=0 tx=0).stop 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2602152497 shutdown_connections 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac15cd00 0x7f3eac158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2602152497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac0cb190 0x7f3eac0ca910 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:50.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.895+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2602152497 >> 172.21.15.71:0/2602152497 conn(0x7f3eac06ae20 msgr2=0x7f3eac06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:50.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.896+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2602152497 shutdown_connections 2023-08-25T10:03:50.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.896+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2602152497 wait complete. 2023-08-25T10:03:50.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.898+0000 7f3eb29cb640 1 Processor -- start 2023-08-25T10:03:50.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.898+0000 7f3eb29cb640 1 -- start start 2023-08-25T10:03:50.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.899+0000 7f3eb29cb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac0cb190 0x7f3eac149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:50.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.899+0000 7f3eb29cb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3eac0cb570 0x7f3eac0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:50.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.899+0000 7f3eb29cb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 0x7f3eac0ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:50.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.899+0000 7f3eb29cb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac14b730 con 0x7f3eac15cd00 2023-08-25T10:03:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.899+0000 7f3eb29cb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac14bac0 con 0x7f3eac0cb570 2023-08-25T10:03:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.899+0000 7f3eb29cb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac0c52a0 con 0x7f3eac0cb190 2023-08-25T10:03:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 0x7f3eac0ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 0x7f3eac0ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55488/0 (socket says 172.21.15.71:55488) 2023-08-25T10:03:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 -- 172.21.15.71:0/2829998458 learned_addr learned my addr 172.21.15.71:0/2829998458 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb19c9640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac0cb190 0x7f3eac149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb11c8640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3eac0cb570 0x7f3eac0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 -- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac0cb190 msgr2=0x7f3eac149380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac0cb190 0x7f3eac149380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 -- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3eac0cb570 msgr2=0x7f3eac0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3eac0cb570 0x7f3eac0cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 -- 172.21.15.71:0/2829998458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8009400 con 0x7f3eac15cd00 2023-08-25T10:03:50.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.900+0000 7f3eb21ca640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 0x7f3eac0ce1b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400cd50 tx=0x7f3ea400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:50.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.901+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400f680 con 0x7f3eac15cd00 2023-08-25T10:03:50.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.901+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea400f800 con 0x7f3eac15cd00 2023-08-25T10:03:50.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.901+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea400fbb0 con 0x7f3eac15cd00 2023-08-25T10:03:50.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.901+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eac0cd6e0 con 0x7f3eac15cd00 2023-08-25T10:03:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.901+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eac0c4fc0 con 0x7f3eac15cd00 2023-08-25T10:03:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.903+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eac0c53e0 con 0x7f3eac15cd00 2023-08-25T10:03:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.904+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3ea400b040 con 0x7f3eac15cd00 2023-08-25T10:03:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.905+0000 7f3ea27fc640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e9006a050 0x7f3e9006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.905+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f3ea40895a0 con 0x7f3eac15cd00 2023-08-25T10:03:50.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.905+0000 7f3eb19c9640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e9006a050 0x7f3e9006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:50.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.906+0000 7f3eb19c9640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e9006a050 0x7f3e9006c510 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8036000 tx=0x7f3ea8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:50.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:50.910+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3ea4054730 con 0x7f3eac15cd00 2023-08-25T10:03:51.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.161+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eac0ceef0 con 0x7f3eac15cd00 2023-08-25T10:03:51.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.162+0000 7f3ea27fc640 1 -- 172.21.15.71:0/2829998458 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f3ea4057d50 con 0x7f3eac15cd00 2023-08-25T10:03:51.165 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:51.165 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:51.166 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:51.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.165+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e9006a050 msgr2=0x7f3e9006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:51.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.165+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e9006a050 0x7f3e9006c510 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8036000 tx=0x7f3ea8008030 comp rx=0 tx=0).stop 2023-08-25T10:03:51.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.165+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 msgr2=0x7f3eac0ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:51.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.165+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 0x7f3eac0ce1b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400cd50 tx=0x7f3ea400a650 comp rx=0 tx=0).stop 2023-08-25T10:03:51.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 shutdown_connections 2023-08-25T10:03:51.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3eac0cb570 0x7f3eac0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:51.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3e9006a050 0x7f3e9006c510 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:51.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eac15cd00 0x7f3eac0ce1b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:51.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 --2- 172.21.15.71:0/2829998458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3eac0cb190 0x7f3eac149380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:51.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 >> 172.21.15.71:0/2829998458 conn(0x7f3eac06ae20 msgr2=0x7f3eac072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:51.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 shutdown_connections 2023-08-25T10:03:51.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:51.166+0000 7f3eb29cb640 1 -- 172.21.15.71:0/2829998458 wait complete. 2023-08-25T10:03:51.187 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:53.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:53.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.416+0000 7f810d245640 1 Processor -- start 2023-08-25T10:03:53.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.416+0000 7f810d245640 1 -- start start 2023-08-25T10:03:53.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810d245640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8108069850 0x7f81081576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:53.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810d245640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f81080c4fb0 0x7f81080c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:53.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810d245640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8108157bf0 con 0x7f81081542d0 2023-08-25T10:03:53.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810d245640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81080c4b60 con 0x7f81080c4fb0 2023-08-25T10:03:53.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810d245640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810806a130 con 0x7f8108069850 2023-08-25T10:03:53.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810ca44640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f81080c4fb0 0x7f81080c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:53.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810ca44640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f81080c4fb0 0x7f81080c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55498/0 (socket says 172.21.15.71:55498) 2023-08-25T10:03:53.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f810ca44640 1 -- 172.21.15.71:0/114152089 learned_addr learned my addr 172.21.15.71:0/114152089 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:53.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.418+0000 7f81077fe640 1 --2- 172.21.15.71:0/114152089 >> v2:172.21.15.164:3300/0 conn(0x7f8108069850 0x7f81081576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:53.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.419+0000 7f810ca44640 1 -- 172.21.15.71:0/114152089 >> v2:172.21.15.164:3300/0 conn(0x7f8108069850 msgr2=0x7f81081576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:53.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.419+0000 7f810ca44640 1 --2- 172.21.15.71:0/114152089 >> v2:172.21.15.164:3300/0 conn(0x7f8108069850 0x7f81081576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.419+0000 7f810ca44640 1 -- 172.21.15.71:0/114152089 >> v1:172.21.15.164:6789/0 conn(0x7f81081542d0 legacy=0x7f81080c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:53.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.419+0000 7f810ca44640 1 -- 172.21.15.71:0/114152089 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81080c5980 con 0x7f81080c4fb0 2023-08-25T10:03:53.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.420+0000 7f810ca44640 1 --2- 172.21.15.71:0/114152089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81080c4fb0 0x7f81080c5440 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f81080c13c0 tx=0x7f810002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360b4f1868bf2151 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:53.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f8106ffd640 1 -- 172.21.15.71:0/114152089 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810002ca80 con 0x7f81080c4fb0 2023-08-25T10:03:53.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f8106ffd640 1 -- 172.21.15.71:0/114152089 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810002cbe0 con 0x7f81080c4fb0 2023-08-25T10:03:53.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f8106ffd640 1 -- 172.21.15.71:0/114152089 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81000343f0 con 0x7f81080c4fb0 2023-08-25T10:03:53.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 -- 172.21.15.71:0/114152089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81080c4fb0 msgr2=0x7f81080c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:53.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 --2- 172.21.15.71:0/114152089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81080c4fb0 0x7f81080c5440 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f81080c13c0 tx=0x7f810002bee0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 -- 172.21.15.71:0/114152089 shutdown_connections 2023-08-25T10:03:53.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 --2- 172.21.15.71:0/114152089 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f81080c4fb0 0x7f81080c5440 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 --2- 172.21.15.71:0/114152089 >> v2:172.21.15.164:3300/0 conn(0x7f8108069850 0x7f81081576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 -- 172.21.15.71:0/114152089 >> 172.21.15.71:0/114152089 conn(0x7f810806c280 msgr2=0x7f810806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:53.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 -- 172.21.15.71:0/114152089 shutdown_connections 2023-08-25T10:03:53.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.421+0000 7f810d245640 1 -- 172.21.15.71:0/114152089 wait complete. 2023-08-25T10:03:53.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.423+0000 7f810d245640 1 Processor -- start 2023-08-25T10:03:53.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.424+0000 7f810d245640 1 -- start start 2023-08-25T10:03:53.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810d245640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 0x7f810815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:53.427 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810d245640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81080c4fb0 0x7f810814ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:53.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810d245640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81081542d0 0x7f81080c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:53.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810d245640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081585c0 con 0x7f8108069850 2023-08-25T10:03:53.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810d245640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81081f1be0 con 0x7f81081542d0 2023-08-25T10:03:53.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810d245640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81081edce0 con 0x7f81080c4fb0 2023-08-25T10:03:53.428 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 0x7f810815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:53.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 0x7f810815ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55502/0 (socket says 172.21.15.71:55502) 2023-08-25T10:03:53.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 -- 172.21.15.71:0/3075256475 learned_addr learned my addr 172.21.15.71:0/3075256475 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:53.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810ca44640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81081542d0 0x7f81080c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:53.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f81077fe640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81080c4fb0 0x7f810814ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:53.429 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 -- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81080c4fb0 msgr2=0x7f810814ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81080c4fb0 0x7f810814ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 -- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81081542d0 msgr2=0x7f81080c2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81081542d0 0x7f81080c2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f8107fff640 1 -- 172.21.15.71:0/3075256475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100009b30 con 0x7f8108069850 2023-08-25T10:03:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.425+0000 7f810ca44640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81081542d0 0x7f81080c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.426+0000 7f8107fff640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 0x7f810815ab30 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c460 tx=0x7f80f800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:53.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.426+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8012830 con 0x7f8108069850 2023-08-25T10:03:53.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.426+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f80129d0 con 0x7f8108069850 2023-08-25T10:03:53.431 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.426+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8012d80 con 0x7f8108069850 2023-08-25T10:03:53.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.426+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810815b070 con 0x7f8108069850 2023-08-25T10:03:53.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.426+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81081edf60 con 0x7f8108069850 2023-08-25T10:03:53.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.428+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810804e420 con 0x7f8108069850 2023-08-25T10:03:53.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.429+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f80f801a3b0 con 0x7f8108069850 2023-08-25T10:03:53.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.429+0000 7f8104ff9640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f80f406a050 0x7f80f406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:53.432 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.430+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f80f8088a50 con 0x7f8108069850 2023-08-25T10:03:53.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.430+0000 7f81077fe640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f80f406a050 0x7f80f406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:53.433 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.431+0000 7f81077fe640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f80f406a050 0x7f80f406c510 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f80fc006fd0 tx=0x7f80fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:53.436 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.435+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f80f8053be0 con 0x7f8108069850 2023-08-25T10:03:53.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.697+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8108069f10 con 0x7f8108069850 2023-08-25T10:03:53.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.698+0000 7f8104ff9640 1 -- 172.21.15.71:0/3075256475 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f80f8057200 con 0x7f8108069850 2023-08-25T10:03:53.701 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:53.701 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:53.701 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:53.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f80f406a050 msgr2=0x7f80f406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:53.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f80f406a050 0x7f80f406c510 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f80fc006fd0 tx=0x7f80fc008040 comp rx=0 tx=0).stop 2023-08-25T10:03:53.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 msgr2=0x7f810815ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 0x7f810815ab30 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c460 tx=0x7f80f800c920 comp rx=0 tx=0).stop 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 shutdown_connections 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81081542d0 0x7f81080c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f80f406a050 0x7f80f406c510 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81080c4fb0 0x7f810814ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 --2- 172.21.15.71:0/3075256475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8108069850 0x7f810815ab30 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:53.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.701+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 >> 172.21.15.71:0/3075256475 conn(0x7f810806c280 msgr2=0x7f81080ad8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:53.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.702+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 shutdown_connections 2023-08-25T10:03:53.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:53.702+0000 7f810d245640 1 -- 172.21.15.71:0/3075256475 wait complete. 2023-08-25T10:03:53.728 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:55.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:55.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.955+0000 7ff2ecf60640 1 Processor -- start 2023-08-25T10:03:55.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.955+0000 7ff2ecf60640 1 -- start start 2023-08-25T10:03:55.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.956+0000 7ff2ecf60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 0x7ff2e80b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2ecf60640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2e8158d80 0x7ff2e8159210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2ecf60640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e8159750 con 0x7ff2e80ca010 2023-08-25T10:03:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2ecf60640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e815d010 con 0x7ff2e8158d80 2023-08-25T10:03:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2ecf60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e80b14f0 con 0x7ff2e80c8310 2023-08-25T10:03:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2e6ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff2e80ca010 0x7ff2e80c9290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47618/0 (socket says 172.21.15.71:47618) 2023-08-25T10:03:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/4257225502 learned_addr learned my addr 172.21.15.71:0/4257225502 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2e77fe640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 0x7ff2e80b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2e7fff640 1 --2- 172.21.15.71:0/4257225502 >> v2:172.21.15.164:3300/0 conn(0x7ff2e8158d80 0x7ff2e8159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766501756 0 0) 0x7ff2e8159750 con 0x7ff2e80ca010 2023-08-25T10:03:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.957+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d0003520 con 0x7ff2e80ca010 2023-08-25T10:03:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 487821372 0 0) 0x7ff2d0003520 con 0x7ff2e80ca010 2023-08-25T10:03:55.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 msgr2=0x7ff2e80b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 0x7ff2e80b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 >> v2:172.21.15.164:3300/0 conn(0x7ff2e8158d80 msgr2=0x7ff2e8159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 --2- 172.21.15.71:0/4257225502 >> v2:172.21.15.164:3300/0 conn(0x7ff2e8158d80 0x7ff2e8159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e80ca550 con 0x7ff2e80ca010 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff2dc001670 con 0x7ff2e80ca010 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 >> v1:172.21.15.71:6789/0 conn(0x7ff2e80ca010 legacy=0x7ff2e80c9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 0x7ff2d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2d0003f60 0x7ff2d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d0004d00 0x7ff2d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e815d010 con 0x7ff2d0003f60 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e80b14f0 con 0x7ff2d0004d00 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d000ba30 con 0x7ff2e80c8310 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.958+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2d0003f60 0x7ff2d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e7fff640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d0004d00 0x7ff2d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e77fe640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 0x7ff2d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 msgr2=0x7ff2d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:55.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2e80c8310 0x7ff2d00039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d0004d00 msgr2=0x7ff2d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:55.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d0004d00 0x7ff2d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.959+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/4257225502 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0003520 con 0x7ff2d0003f60 2023-08-25T10:03:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.960+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2d0003f60 0x7ff2d00047c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c290 tx=0x7ff2d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c264d933cc545aa server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:55.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.960+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc005ad0 con 0x7ff2d0003f60 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.960+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc005c30 con 0x7ff2d0003f60 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.960+0000 7ff2e67fc640 1 -- 172.21.15.71:0/4257225502 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc02c8d0 con 0x7ff2d0003f60 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2ecf60640 1 -- 172.21.15.71:0/4257225502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2d0003f60 msgr2=0x7ff2d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/4257225502 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2d0003f60 0x7ff2d00047c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c290 tx=0x7ff2d00057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2e7fff640 1 -- 172.21.15.71:0/4257225502 reap_dead start 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2ecf60640 1 -- 172.21.15.71:0/4257225502 shutdown_connections 2023-08-25T10:03:55.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2ecf60640 1 -- 172.21.15.71:0/4257225502 >> 172.21.15.71:0/4257225502 conn(0x7ff2e806ae20 msgr2=0x7ff2e806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:55.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2ecf60640 1 -- 172.21.15.71:0/4257225502 shutdown_connections 2023-08-25T10:03:55.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.961+0000 7ff2ecf60640 1 -- 172.21.15.71:0/4257225502 wait complete. 2023-08-25T10:03:55.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.963+0000 7ff2ecf60640 1 Processor -- start 2023-08-25T10:03:55.967 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.963+0000 7ff2ecf60640 1 -- start start 2023-08-25T10:03:55.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.964+0000 7ff2ecf60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d0003f60 0x7ff2e8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.964+0000 7ff2ecf60640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 0x7ff2e81f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.964+0000 7ff2ecf60640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e81f2780 0x7ff2e81487a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.964+0000 7ff2ecf60640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e80c7ae0 con 0x7ff2e80cdc00 2023-08-25T10:03:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.964+0000 7ff2ecf60640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81f1ad0 con 0x7ff2e81f2780 2023-08-25T10:03:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.964+0000 7ff2ecf60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81f1db0 con 0x7ff2d0003f60 2023-08-25T10:03:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 0x7ff2e81f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 0x7ff2e81f2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55524/0 (socket says 172.21.15.71:55524) 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/2256531789 learned_addr learned my addr 172.21.15.71:0/2256531789 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e77fe640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d0003f60 0x7ff2e8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e7fff640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e81f2780 0x7ff2e81487a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d0003f60 msgr2=0x7ff2e8157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d0003f60 0x7ff2e8157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e81f2780 msgr2=0x7ff2e81487a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:55.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e81f2780 0x7ff2e81487a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.965+0000 7ff2e6ffd640 1 -- 172.21.15.71:0/2256531789 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e80c5810 con 0x7ff2e80cdc00 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.966+0000 7ff2e6ffd640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 0x7ff2e81f2240 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc02c470 tx=0x7ff2dc000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.966+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc02cce0 con 0x7ff2e80cdc00 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.966+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc02ce60 con 0x7ff2e80cdc00 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.966+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc0072b0 con 0x7ff2e80cdc00 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.966+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e80c7800 con 0x7ff2e80cdc00 2023-08-25T10:03:55.971 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.966+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e815be90 con 0x7ff2e80cdc00 2023-08-25T10:03:55.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.967+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e80cd000 con 0x7ff2e80cdc00 2023-08-25T10:03:55.972 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.968+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff2dc032070 con 0x7ff2e80cdc00 2023-08-25T10:03:55.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.969+0000 7ff2cffff640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d406a080 0x7ff2d406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:55.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.970+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff2dc03c040 con 0x7ff2e80cdc00 2023-08-25T10:03:55.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.970+0000 7ff2e77fe640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d406a080 0x7ff2d406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:55.973 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.970+0000 7ff2e77fe640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d406a080 0x7ff2d406c540 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009930 tx=0x7ff2d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:55.977 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:55.975+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff2dc0750c0 con 0x7ff2e80cdc00 2023-08-25T10:03:56.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.240+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2e804e420 con 0x7ff2e80cdc00 2023-08-25T10:03:56.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.241+0000 7ff2cffff640 1 -- 172.21.15.71:0/2256531789 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7ff2dc045070 con 0x7ff2e80cdc00 2023-08-25T10:03:56.243 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:56.244 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:56.244 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d406a080 msgr2=0x7ff2d406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d406a080 0x7ff2d406c540 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009930 tx=0x7ff2d8008040 comp rx=0 tx=0).stop 2023-08-25T10:03:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 msgr2=0x7ff2e81f2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 0x7ff2e81f2240 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc02c470 tx=0x7ff2dc000fe0 comp rx=0 tx=0).stop 2023-08-25T10:03:56.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 shutdown_connections 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2e81f2780 0x7ff2e81487a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff2d406a080 0x7ff2d406c540 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff2e80cdc00 0x7ff2e81f2240 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 --2- 172.21.15.71:0/2256531789 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d0003f60 0x7ff2e8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 >> 172.21.15.71:0/2256531789 conn(0x7ff2e806ae20 msgr2=0x7ff2e80c9250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 shutdown_connections 2023-08-25T10:03:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:56.244+0000 7ff2ecf60640 1 -- 172.21.15.71:0/2256531789 wait complete. 2023-08-25T10:03:56.266 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:03:56.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:03:56.638 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:03:58.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:03:58.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.500+0000 7f4dcd9a4640 1 Processor -- start 2023-08-25T10:03:58.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.500+0000 7f4dcd9a4640 1 -- start start 2023-08-25T10:03:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.502+0000 7f4dcd9a4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4dc8152d80 0x7f4dc80c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.502+0000 7f4dcd9a4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc81591c0 con 0x7f4dc815ce50 2023-08-25T10:03:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.502+0000 7f4dcd9a4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc81594a0 con 0x7f4dc80cacb0 2023-08-25T10:03:58.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.502+0000 7f4dcd9a4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc8159780 con 0x7f4dc8152d80 2023-08-25T10:03:58.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.502+0000 7f4dc77fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4dc815ce50 0x7f4dc80cb140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47624/0 (socket says 172.21.15.71:47624) 2023-08-25T10:03:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc77fe640 1 -- 172.21.15.71:0/3806636867 learned_addr learned my addr 172.21.15.71:0/3806636867 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:03:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc6ffd640 1 --2- 172.21.15.71:0/3806636867 >> v2:172.21.15.164:3300/0 conn(0x7f4dc8152d80 0x7f4dc80c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11896841 0 0) 0x7f4dc81591c0 con 0x7f4dc815ce50 2023-08-25T10:03:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db0003520 con 0x7f4dc815ce50 2023-08-25T10:03:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052680689 0 0) 0x7f4db0003520 con 0x7f4dc815ce50 2023-08-25T10:03:58.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 >> v2:172.21.15.164:3300/0 conn(0x7f4dc8152d80 msgr2=0x7f4dc80c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.503+0000 7f4dc5ffb640 1 --2- 172.21.15.71:0/3806636867 >> v2:172.21.15.164:3300/0 conn(0x7f4dc8152d80 0x7f4dc80c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 >> v1:172.21.15.164:6789/0 conn(0x7f4dc80cacb0 legacy=0x7f4dc815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8159a60 con 0x7f4dc815ce50 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4dc00019f0 con 0x7f4dc815ce50 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 >> v1:172.21.15.71:6789/0 conn(0x7f4dc815ce50 legacy=0x7f4dc80cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4db00039e0 0x7f4db0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4db0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4db0005180 0x7f4db000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc8159780 con 0x7f4db00047c0 2023-08-25T10:03:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db000c090 con 0x7f4db0005180 2023-08-25T10:03:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db000c250 con 0x7f4db00039e0 2023-08-25T10:03:58.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4db0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc77fe640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4db0005180 0x7f4db000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc6ffd640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4db00039e0 0x7f4db0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc67fc640 1 -- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4db00039e0 msgr2=0x7f4db0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4db00039e0 0x7f4db0003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc67fc640 1 -- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4db0005180 msgr2=0x7f4db000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:03:58.510 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.504+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4db0005180 0x7f4db000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.505+0000 7f4dc67fc640 1 -- 172.21.15.71:0/3806636867 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0003520 con 0x7f4db00047c0 2023-08-25T10:03:58.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.505+0000 7f4dc77fe640 1 -- 172.21.15.71:0/3806636867 reap_dead start 2023-08-25T10:03:58.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.506+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4db0004c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc001750 tx=0x7f4db00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe2d8196c2b4c12 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:58.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.506+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0052d0 con 0x7f4db00047c0 2023-08-25T10:03:58.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.506+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc02e050 con 0x7f4db00047c0 2023-08-25T10:03:58.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.506+0000 7f4dc5ffb640 1 -- 172.21.15.71:0/3806636867 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc003830 con 0x7f4db00047c0 2023-08-25T10:03:58.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/3806636867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 msgr2=0x7f4db0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4db0004c40 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc001750 tx=0x7f4db00057f0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/3806636867 shutdown_connections 2023-08-25T10:03:58.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/3806636867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4db0004c40 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/3806636867 >> 172.21.15.71:0/3806636867 conn(0x7f4dc806b8f0 msgr2=0x7f4dc80654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:58.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/3806636867 shutdown_connections 2023-08-25T10:03:58.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.507+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/3806636867 wait complete. 2023-08-25T10:03:58.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.509+0000 7f4dcd9a4640 1 Processor -- start 2023-08-25T10:03:58.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.509+0000 7f4dcd9a4640 1 -- start start 2023-08-25T10:03:58.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dcd9a4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4dc8156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dcd9a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc814d290 0x7f4dc81eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dcd9a4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc814d6d0 0x7f4dc81ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dcd9a4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc81ee6f0 con 0x7f4db00047c0 2023-08-25T10:03:58.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dcd9a4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc80c4c40 con 0x7f4dc814d6d0 2023-08-25T10:03:58.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dcd9a4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc81f2980 con 0x7f4dc814d290 2023-08-25T10:03:58.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dc6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4dc8156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dc6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4dc8156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:55538/0 (socket says 172.21.15.71:55538) 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dc6ffd640 1 -- 172.21.15.71:0/219496208 learned_addr learned my addr 172.21.15.71:0/219496208 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc814d290 0x7f4dc81eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.510+0000 7f4dc77fe640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc814d6d0 0x7f4dc81ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc6ffd640 1 -- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc814d290 msgr2=0x7f4dc81eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc6ffd640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc814d290 0x7f4dc81eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc6ffd640 1 -- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc814d6d0 msgr2=0x7f4dc81ef0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc6ffd640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc814d6d0 0x7f4dc81ef0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc6ffd640 1 -- 172.21.15.71:0/219496208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc81f1e00 con 0x7f4db00047c0 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc77fe640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc814d6d0 0x7f4dc81ef0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dc6ffd640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4dc8156110 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4db8010790 tx=0x7f4db8010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8009630 con 0x7f4db00047c0 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db800eb60 con 0x7f4db00047c0 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8010ea0 con 0x7f4db00047c0 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.511+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc81f20e0 con 0x7f4db00047c0 2023-08-25T10:03:58.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.512+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc81f26a0 con 0x7f4db00047c0 2023-08-25T10:03:58.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.513+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc80b1a60 con 0x7f4db00047c0 2023-08-25T10:03:58.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.514+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4db801c400 con 0x7f4db00047c0 2023-08-25T10:03:58.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.515+0000 7f4daffff640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:03:58.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.515+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4db80890d0 con 0x7f4db00047c0 2023-08-25T10:03:58.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.515+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:03:58.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.516+0000 7f4dc67fc640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc033000 tx=0x7f4dbc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:03:58.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.520+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4db80541d0 con 0x7f4db00047c0 2023-08-25T10:03:58.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.784+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dc81576f0 con 0x7f4db00047c0 2023-08-25T10:03:58.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.785+0000 7f4daffff640 1 -- 172.21.15.71:0/219496208 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4db80577f0 con 0x7f4db00047c0 2023-08-25T10:03:58.787 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:03:58.788 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:03:58.788 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:03:58.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 msgr2=0x7f4db406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc033000 tx=0x7f4dbc009030 comp rx=0 tx=0).stop 2023-08-25T10:03:58.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 msgr2=0x7f4dc8156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:03:58.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4dc8156110 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4db8010790 tx=0x7f4db8010c50 comp rx=0 tx=0).stop 2023-08-25T10:03:58.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 shutdown_connections 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc814d6d0 0x7f4dc81ef0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4db406a050 0x7f4db406c510 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc814d290 0x7f4dc81eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 --2- 172.21.15.71:0/219496208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4db00047c0 0x7f4dc8156110 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 >> 172.21.15.71:0/219496208 conn(0x7f4dc806b8f0 msgr2=0x7f4dc8156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 shutdown_connections 2023-08-25T10:03:58.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:03:58.788+0000 7f4dcd9a4640 1 -- 172.21.15.71:0/219496208 wait complete. 2023-08-25T10:03:58.807 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:00.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:01.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.053+0000 7f3c9e43f640 1 Processor -- start 2023-08-25T10:04:01.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.053+0000 7f3c9e43f640 1 -- start start 2023-08-25T10:04:01.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.055+0000 7f3c9e43f640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.055+0000 7f3c9e43f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c981591c0 con 0x7f3c980cacb0 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.055+0000 7f3c9e43f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c981594a0 con 0x7f3c98152d80 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.055+0000 7f3c9e43f640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c98159780 con 0x7f3c9815ce50 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c9c9dc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c9c9dc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50592/0 (socket says 172.21.15.71:50592) 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c977fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3c980cacb0 0x7f3c9815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50662/0 (socket says 172.21.15.71:50662) 2023-08-25T10:04:01.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c9c9dc640 1 -- 172.21.15.71:0/1590623210 learned_addr learned my addr 172.21.15.71:0/1590623210 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:01.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c9c9dc640 1 -- 172.21.15.71:0/1590623210 >> v1:172.21.15.164:6789/0 conn(0x7f3c98152d80 legacy=0x7f3c980c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:01.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c9c9dc640 1 -- 172.21.15.71:0/1590623210 >> v1:172.21.15.71:6789/0 conn(0x7f3c980cacb0 legacy=0x7f3c9815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:01.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.056+0000 7f3c9c9dc640 1 -- 172.21.15.71:0/1590623210 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c98159a60 con 0x7f3c9815ce50 2023-08-25T10:04:01.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.058+0000 7f3c9c9dc640 1 --2- 172.21.15.71:0/1590623210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c9815ce50 0x7f3c980cb140 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009e50 tx=0x7f3c9002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288109b0bd42cb5d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:01.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.058+0000 7f3c96ffd640 1 -- 172.21.15.71:0/1590623210 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9002cb90 con 0x7f3c9815ce50 2023-08-25T10:04:01.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.058+0000 7f3c96ffd640 1 -- 172.21.15.71:0/1590623210 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9002ccf0 con 0x7f3c9815ce50 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.058+0000 7f3c96ffd640 1 -- 172.21.15.71:0/1590623210 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90034610 con 0x7f3c9815ce50 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1590623210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c9815ce50 msgr2=0x7f3c980cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1590623210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c9815ce50 0x7f3c980cb140 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009e50 tx=0x7f3c9002bff0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1590623210 shutdown_connections 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1590623210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1590623210 >> 172.21.15.71:0/1590623210 conn(0x7f3c9806b8f0 msgr2=0x7f3c980654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:01.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1590623210 shutdown_connections 2023-08-25T10:04:01.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.059+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1590623210 wait complete. 2023-08-25T10:04:01.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.061+0000 7f3c9e43f640 1 Processor -- start 2023-08-25T10:04:01.063 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.061+0000 7f3c9e43f640 1 -- start start 2023-08-25T10:04:01.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c9e43f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 0x7f3c980cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c9e43f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c98152d80 0x7f3c981f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c9e43f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c9e43f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980ce500 con 0x7f3c980cacb0 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c9e43f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c981f1ad0 con 0x7f3c98152d80 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c9e43f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980b45b0 con 0x7f3c9815ce50 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c97fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 0x7f3c980cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:01.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c97fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 0x7f3c980cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50600/0 (socket says 172.21.15.71:50600) 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.062+0000 7f3c97fff640 1 -- 172.21.15.71:0/1083348578 learned_addr learned my addr 172.21.15.71:0/1083348578 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c977fe640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c98152d80 0x7f3c981f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c9c9dc640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c97fff640 1 -- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c9815ce50 msgr2=0x7f3c980cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c97fff640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c97fff640 1 -- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c98152d80 msgr2=0x7f3c981f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c97fff640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c98152d80 0x7f3c981f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c97fff640 1 -- 172.21.15.71:0/1083348578 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90009b30 con 0x7f3c980cacb0 2023-08-25T10:04:01.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c97fff640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 0x7f3c980cdfc0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800c430 tx=0x7f3c8800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:01.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.063+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88011790 con 0x7f3c980cacb0 2023-08-25T10:04:01.068 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.064+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c880118f0 con 0x7f3c980cacb0 2023-08-25T10:04:01.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.064+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c981478f0 con 0x7f3c980cacb0 2023-08-25T10:04:01.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.064+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c88011ca0 con 0x7f3c980cacb0 2023-08-25T10:04:01.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.064+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c981f1760 con 0x7f3c980cacb0 2023-08-25T10:04:01.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.065+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c981f1920 con 0x7f3c980cacb0 2023-08-25T10:04:01.069 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.066+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3c8801c460 con 0x7f3c980cacb0 2023-08-25T10:04:01.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.067+0000 7f3c94ff9640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c8406a050 0x7f3c8406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:01.070 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.067+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f3c88088a20 con 0x7f3c980cacb0 2023-08-25T10:04:01.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.067+0000 7f3c977fe640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c8406a050 0x7f3c8406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:01.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.068+0000 7f3c977fe640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c8406a050 0x7f3c8406c510 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3c98065840 tx=0x7f3c8c007c80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:01.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.073+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3c88053bb0 con 0x7f3c980cacb0 2023-08-25T10:04:01.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.329+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c980b1a60 con 0x7f3c980cacb0 2023-08-25T10:04:01.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.330+0000 7f3c94ff9640 1 -- 172.21.15.71:0/1083348578 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f3c880571d0 con 0x7f3c980cacb0 2023-08-25T10:04:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:01.333 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:01.333 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:01.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.333+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c8406a050 msgr2=0x7f3c8406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:01.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.333+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c8406a050 0x7f3c8406c510 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3c98065840 tx=0x7f3c8c007c80 comp rx=0 tx=0).stop 2023-08-25T10:04:01.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.333+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 msgr2=0x7f3c980cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:01.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.333+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 0x7f3c980cdfc0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800c430 tx=0x7f3c8800c8f0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 shutdown_connections 2023-08-25T10:04:01.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3c98152d80 0x7f3c981f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f3c8406a050 0x7f3c8406c510 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c9815ce50 0x7f3c980cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 --2- 172.21.15.71:0/1083348578 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3c980cacb0 0x7f3c980cdfc0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:01.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 >> 172.21.15.71:0/1083348578 conn(0x7f3c9806b8f0 msgr2=0x7f3c980c3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:01.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 shutdown_connections 2023-08-25T10:04:01.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:01.334+0000 7f3c9e43f640 1 -- 172.21.15.71:0/1083348578 wait complete. 2023-08-25T10:04:01.356 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:04:01.358 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:04:03.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:03.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.596+0000 7fe59f2c0640 1 Processor -- start 2023-08-25T10:04:03.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.596+0000 7fe59f2c0640 1 -- start start 2023-08-25T10:04:03.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.598+0000 7fe59f2c0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe59814fc20 0x7fe5981598c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:03.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.598+0000 7fe59f2c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe598071590 0x7fe59806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:03.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.598+0000 7fe59f2c0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5980c5260 con 0x7fe59806f720 2023-08-25T10:04:03.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.598+0000 7fe59f2c0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe598159e00 con 0x7fe59814fc20 2023-08-25T10:04:03.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.598+0000 7fe59f2c0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5980c5500 con 0x7fe598071590 2023-08-25T10:04:03.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.598+0000 7fe59e2be640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe59814fc20 0x7fe5981598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:03.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59e2be640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe59814fc20 0x7fe5981598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50602/0 (socket says 172.21.15.71:50602) 2023-08-25T10:04:03.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59e2be640 1 -- 172.21.15.71:0/3043341079 learned_addr learned my addr 172.21.15.71:0/3043341079 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:03.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59dabd640 1 --1- 172.21.15.71:0/3043341079 >> v1:172.21.15.71:6789/0 conn(0x7fe59806f720 0x7fe5980c65a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50674/0 (socket says 172.21.15.71:50674) 2023-08-25T10:04:03.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59eabf640 1 --2- 172.21.15.71:0/3043341079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe598071590 0x7fe59806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:03.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59e2be640 1 -- 172.21.15.71:0/3043341079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe598071590 msgr2=0x7fe59806a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:03.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59e2be640 1 --2- 172.21.15.71:0/3043341079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe598071590 0x7fe59806a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59e2be640 1 -- 172.21.15.71:0/3043341079 >> v1:172.21.15.71:6789/0 conn(0x7fe59806f720 legacy=0x7fe5980c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:03.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.599+0000 7fe59e2be640 1 -- 172.21.15.71:0/3043341079 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5980c57e0 con 0x7fe59814fc20 2023-08-25T10:04:03.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.601+0000 7fe59e2be640 1 --2- 172.21.15.71:0/3043341079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59814fc20 0x7fe5981598c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe59400bb40 tx=0x7fe59402e1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23d16e34862d2e0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:03.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.601+0000 7fe59d2bc640 1 -- 172.21.15.71:0/3043341079 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594036440 con 0x7fe59814fc20 2023-08-25T10:04:03.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.601+0000 7fe59d2bc640 1 -- 172.21.15.71:0/3043341079 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5940365a0 con 0x7fe59814fc20 2023-08-25T10:04:03.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.601+0000 7fe59d2bc640 1 -- 172.21.15.71:0/3043341079 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594036930 con 0x7fe59814fc20 2023-08-25T10:04:03.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 -- 172.21.15.71:0/3043341079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59814fc20 msgr2=0x7fe5981598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/3043341079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59814fc20 0x7fe5981598c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe59400bb40 tx=0x7fe59402e1e0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 -- 172.21.15.71:0/3043341079 shutdown_connections 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/3043341079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe598071590 0x7fe59806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/3043341079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe59814fc20 0x7fe5981598c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 -- 172.21.15.71:0/3043341079 >> 172.21.15.71:0/3043341079 conn(0x7fe59806d0d0 msgr2=0x7fe5980ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 -- 172.21.15.71:0/3043341079 shutdown_connections 2023-08-25T10:04:03.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.602+0000 7fe59f2c0640 1 -- 172.21.15.71:0/3043341079 wait complete. 2023-08-25T10:04:03.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.604+0000 7fe59f2c0640 1 Processor -- start 2023-08-25T10:04:03.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.604+0000 7fe59f2c0640 1 -- start start 2023-08-25T10:04:03.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.605+0000 7fe59f2c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59806f720 0x7fe598150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:03.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.605+0000 7fe59f2c0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 0x7fe598148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:03.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.605+0000 7fe59f2c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59814fc20 0x7fe5981f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:03.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.605+0000 7fe59f2c0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe598148c70 con 0x7fe598071590 2023-08-25T10:04:03.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.605+0000 7fe59f2c0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe598153750 con 0x7fe59814fc20 2023-08-25T10:04:03.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.605+0000 7fe59f2c0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe598148e00 con 0x7fe59806f720 2023-08-25T10:04:03.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 0x7fe598148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:03.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 0x7fe598148730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50606/0 (socket says 172.21.15.71:50606) 2023-08-25T10:04:03.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 -- 172.21.15.71:0/1932010175 learned_addr learned my addr 172.21.15.71:0/1932010175 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:03.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59e2be640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59806f720 0x7fe598150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:03.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59eabf640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59814fc20 0x7fe5981f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:03.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 -- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59806f720 msgr2=0x7fe598150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:03.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59806f720 0x7fe598150d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 -- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59814fc20 msgr2=0x7fe5981f1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:03.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59814fc20 0x7fe5981f1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 -- 172.21.15.71:0/1932010175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59400b770 con 0x7fe598071590 2023-08-25T10:04:03.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.606+0000 7fe59dabd640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 0x7fe598148730 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe58800c4d0 tx=0x7fe58800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:03.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.607+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe588007480 con 0x7fe598071590 2023-08-25T10:04:03.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.607+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5880075e0 con 0x7fe598071590 2023-08-25T10:04:03.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.607+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5880049e0 con 0x7fe598071590 2023-08-25T10:04:03.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.607+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5981f22f0 con 0x7fe598071590 2023-08-25T10:04:03.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.607+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe598153c80 con 0x7fe598071590 2023-08-25T10:04:03.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.608+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe598148450 con 0x7fe598071590 2023-08-25T10:04:03.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.609+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe588004b40 con 0x7fe598071590 2023-08-25T10:04:03.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.611+0000 7fe58effd640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe57c06a050 0x7fe57c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:03.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.611+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe588087bc0 con 0x7fe598071590 2023-08-25T10:04:03.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.611+0000 7fe59e2be640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe57c06a050 0x7fe57c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:03.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.612+0000 7fe59e2be640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe57c06a050 0x7fe57c06c510 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe594038000 tx=0x7fe594008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:03.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.616+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe598148450 con 0x7fe598071590 2023-08-25T10:04:03.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.882+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe598148450 con 0x7fe598071590 2023-08-25T10:04:03.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.883+0000 7fe58effd640 1 -- 172.21.15.71:0/1932010175 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fe598148450 con 0x7fe598071590 2023-08-25T10:04:03.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:03.886 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:03.886 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:03.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.885+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe57c06a050 msgr2=0x7fe57c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:03.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.885+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe57c06a050 0x7fe57c06c510 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe594038000 tx=0x7fe594008040 comp rx=0 tx=0).stop 2023-08-25T10:04:03.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.885+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 msgr2=0x7fe598148730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:03.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.885+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 0x7fe598148730 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fe58800c4d0 tx=0x7fe58800c990 comp rx=0 tx=0).stop 2023-08-25T10:04:03.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 shutdown_connections 2023-08-25T10:04:03.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe59814fc20 0x7fe5981f1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe57c06a050 0x7fe57c06c510 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe598071590 0x7fe598148730 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 --2- 172.21.15.71:0/1932010175 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe59806f720 0x7fe598150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:03.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 >> 172.21.15.71:0/1932010175 conn(0x7fe59806d0d0 msgr2=0x7fe598158200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:03.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 shutdown_connections 2023-08-25T10:04:03.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:03.886+0000 7fe59f2c0640 1 -- 172.21.15.71:0/1932010175 wait complete. 2023-08-25T10:04:03.910 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:05.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:06.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.151+0000 7fad9b59e640 1 Processor -- start 2023-08-25T10:04:06.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.151+0000 7fad9b59e640 1 -- start start 2023-08-25T10:04:06.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.152+0000 7fad9b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c0cb190 0x7fad9c0ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.152+0000 7fad9b59e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fad9c15cd00 0x7fad9c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.152+0000 7fad9b59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9c0b1af0 con 0x7fad9c0cb570 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.152+0000 7fad9b59e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9c0c98f0 con 0x7fad9c15cd00 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.152+0000 7fad9b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c15d190 con 0x7fad9c0cb190 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad99d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fad9c0cb570 0x7fad9c0c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50676/0 (socket says 172.21.15.71:50676) 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad99d9b640 1 -- 172.21.15.71:0/3353362250 learned_addr learned my addr 172.21.15.71:0/3353362250 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9ad9d640 1 --2- 172.21.15.71:0/3353362250 >> v2:172.21.15.71:3300/0 conn(0x7fad9c15cd00 0x7fad9c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9a59c640 1 --2- 172.21.15.71:0/3353362250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c0cb190 0x7fad9c0ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9ad9d640 1 -- 172.21.15.71:0/3353362250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c0cb190 msgr2=0x7fad9c0ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9ad9d640 1 --2- 172.21.15.71:0/3353362250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c0cb190 0x7fad9c0ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9ad9d640 1 -- 172.21.15.71:0/3353362250 >> v1:172.21.15.71:6789/0 conn(0x7fad9c0cb570 legacy=0x7fad9c0c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9ad9d640 1 -- 172.21.15.71:0/3353362250 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c0b0fb0 con 0x7fad9c15cd00 2023-08-25T10:04:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.153+0000 7fad9959a640 1 -- 172.21.15.71:0/3353362250 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420561140 0 0) 0x7fad9c0b1af0 con 0x7fad9c0cb570 2023-08-25T10:04:06.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.155+0000 7fad9ad9d640 1 --2- 172.21.15.71:0/3353362250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c15cd00 0x7fad9c158d80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fad94009d90 tx=0x7fad9402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bdfbc46ac30c87 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:06.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.155+0000 7fad9959a640 1 -- 172.21.15.71:0/3353362250 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9402cce0 con 0x7fad9c15cd00 2023-08-25T10:04:06.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.155+0000 7fad9959a640 1 -- 172.21.15.71:0/3353362250 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9402ce40 con 0x7fad9c15cd00 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.155+0000 7fad9959a640 1 -- 172.21.15.71:0/3353362250 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad94034660 con 0x7fad9c15cd00 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.155+0000 7fad9b59e640 1 -- 172.21.15.71:0/3353362250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c15cd00 msgr2=0x7fad9c158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.155+0000 7fad9b59e640 1 --2- 172.21.15.71:0/3353362250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c15cd00 0x7fad9c158d80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fad94009d90 tx=0x7fad9402bf30 comp rx=0 tx=0).stop 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.156+0000 7fad9b59e640 1 -- 172.21.15.71:0/3353362250 shutdown_connections 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.156+0000 7fad9b59e640 1 --2- 172.21.15.71:0/3353362250 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c15cd00 0x7fad9c158d80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.156+0000 7fad9b59e640 1 --2- 172.21.15.71:0/3353362250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c0cb190 0x7fad9c0ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.156+0000 7fad9b59e640 1 -- 172.21.15.71:0/3353362250 >> 172.21.15.71:0/3353362250 conn(0x7fad9c06ae20 msgr2=0x7fad9c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:06.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.156+0000 7fad9b59e640 1 -- 172.21.15.71:0/3353362250 shutdown_connections 2023-08-25T10:04:06.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.156+0000 7fad9b59e640 1 -- 172.21.15.71:0/3353362250 wait complete. 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.158+0000 7fad9b59e640 1 Processor -- start 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.158+0000 7fad9b59e640 1 -- start start 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.159+0000 7fad9b59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 0x7fad9c14dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.159+0000 7fad9b59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad9c0cb570 0x7fad9c1ee540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.159+0000 7fad9b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c15cd00 0x7fad9c0cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.159+0000 7fad9b59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c1561d0 con 0x7fad9c0cb190 2023-08-25T10:04:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.159+0000 7fad9b59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c1ee0b0 con 0x7fad9c0cb570 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.159+0000 7fad9b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad9c1edd80 con 0x7fad9c15cd00 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 0x7fad9c14dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 0x7fad9c14dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50626/0 (socket says 172.21.15.71:50626) 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 -- 172.21.15.71:0/2846468699 learned_addr learned my addr 172.21.15.71:0/2846468699 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad99d9b640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad9c0cb570 0x7fad9c1ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 -- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c15cd00 msgr2=0x7fad9c0cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c15cd00 0x7fad9c0cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 -- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad9c0cb570 msgr2=0x7fad9c1ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:06.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad9c0cb570 0x7fad9c1ee540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 -- 172.21.15.71:0/2846468699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94009a70 con 0x7fad9c0cb190 2023-08-25T10:04:06.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.160+0000 7fad9a59c640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 0x7fad9c14dcf0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00d5f0 tx=0x7fad8c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:06.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.161+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c0125f0 con 0x7fad9c0cb190 2023-08-25T10:04:06.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.161+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c012750 con 0x7fad9c0cb190 2023-08-25T10:04:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.161+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c012b00 con 0x7fad9c0cb190 2023-08-25T10:04:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.161+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9c1f1f30 con 0x7fad9c0cb190 2023-08-25T10:04:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.161+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9c156700 con 0x7fad9c0cb190 2023-08-25T10:04:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.162+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9c04e420 con 0x7fad9c0cb190 2023-08-25T10:04:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.163+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fad8c012c60 con 0x7fad9c0cb190 2023-08-25T10:04:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.165+0000 7fad8affd640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad7c06a080 0x7fad7c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.165+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fad8c088ee0 con 0x7fad9c0cb190 2023-08-25T10:04:06.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.165+0000 7fad99d9b640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad7c06a080 0x7fad7c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:06.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.166+0000 7fad99d9b640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad7c06a080 0x7fad7c06c540 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fad90007fd0 tx=0x7fad90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:06.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.170+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fad8c054100 con 0x7fad9c0cb190 2023-08-25T10:04:06.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.438+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad9c0b1530 con 0x7fad9c0cb190 2023-08-25T10:04:06.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.439+0000 7fad8affd640 1 -- 172.21.15.71:0/2846468699 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fad8c088070 con 0x7fad9c0cb190 2023-08-25T10:04:06.442 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:06.442 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:06.442 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:06.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad7c06a080 msgr2=0x7fad7c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:06.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad7c06a080 0x7fad7c06c540 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fad90007fd0 tx=0x7fad90009040 comp rx=0 tx=0).stop 2023-08-25T10:04:06.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 msgr2=0x7fad9c14dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 0x7fad9c14dcf0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00d5f0 tx=0x7fad8c00dab0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 shutdown_connections 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad9c0cb570 0x7fad9c1ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fad7c06a080 0x7fad7c06c540 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad9c15cd00 0x7fad9c0cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 --2- 172.21.15.71:0/2846468699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad9c0cb190 0x7fad9c14dcf0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.442+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 >> 172.21.15.71:0/2846468699 conn(0x7fad9c06ae20 msgr2=0x7fad9c15b470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:06.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.443+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 shutdown_connections 2023-08-25T10:04:06.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:06.443+0000 7fad9b59e640 1 -- 172.21.15.71:0/2846468699 wait complete. 2023-08-25T10:04:06.465 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:08.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:08.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.681+0000 7f324ffff640 1 Processor -- start 2023-08-25T10:04:08.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.682+0000 7f324ffff640 1 -- start start 2023-08-25T10:04:08.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.684+0000 7f324ffff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3250152db0 0x7f32500c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.684+0000 7f324ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32500c60e0 0x7f32500c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.684+0000 7f324ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3250071be0 0x7f3250158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.684+0000 7f324ffff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3250158b90 con 0x7f3250152db0 2023-08-25T10:04:08.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324ffff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32500c5260 con 0x7f3250071be0 2023-08-25T10:04:08.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324ffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32500c6580 con 0x7f32500c60e0 2023-08-25T10:04:08.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324effd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3250152db0 0x7f32500c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324effd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f3250152db0 0x7f32500c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50632/0 (socket says 172.21.15.71:50632) 2023-08-25T10:04:08.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324effd640 1 -- 172.21.15.71:0/2643930143 learned_addr learned my addr 172.21.15.71:0/2643930143 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:08.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324f7fe640 1 --2- 172.21.15.71:0/2643930143 >> v2:172.21.15.164:3300/0 conn(0x7f3250071be0 0x7f3250158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.685+0000 7f324e7fc640 1 --2- 172.21.15.71:0/2643930143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32500c60e0 0x7f32500c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.686+0000 7f324effd640 1 -- 172.21.15.71:0/2643930143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32500c60e0 msgr2=0x7f32500c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:08.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.686+0000 7f324effd640 1 --2- 172.21.15.71:0/2643930143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32500c60e0 0x7f32500c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.686+0000 7f324effd640 1 -- 172.21.15.71:0/2643930143 >> v2:172.21.15.164:3300/0 conn(0x7f3250071be0 msgr2=0x7f3250158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:08.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.686+0000 7f324effd640 1 --2- 172.21.15.71:0/2643930143 >> v2:172.21.15.164:3300/0 conn(0x7f3250071be0 0x7f3250158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.686+0000 7f324effd640 1 -- 172.21.15.71:0/2643930143 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325015cc30 con 0x7f3250152db0 2023-08-25T10:04:08.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.687+0000 7f324effd640 1 --2- 172.21.15.71:0/2643930143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3250152db0 0x7f32500c67c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f325006eb50 tx=0x7f324002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203824d68b150dda server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:08.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.688+0000 7f324dffb640 1 -- 172.21.15.71:0/2643930143 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240031870 con 0x7f3250152db0 2023-08-25T10:04:08.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.688+0000 7f324dffb640 1 -- 172.21.15.71:0/2643930143 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32400319d0 con 0x7f3250152db0 2023-08-25T10:04:08.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.688+0000 7f324dffb640 1 -- 172.21.15.71:0/2643930143 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240031d60 con 0x7f3250152db0 2023-08-25T10:04:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 -- 172.21.15.71:0/2643930143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3250152db0 msgr2=0x7f32500c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 --2- 172.21.15.71:0/2643930143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3250152db0 0x7f32500c67c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f325006eb50 tx=0x7f324002b910 comp rx=0 tx=0).stop 2023-08-25T10:04:08.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 -- 172.21.15.71:0/2643930143 shutdown_connections 2023-08-25T10:04:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 --2- 172.21.15.71:0/2643930143 >> v2:172.21.15.164:3300/0 conn(0x7f3250071be0 0x7f3250158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 --2- 172.21.15.71:0/2643930143 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32500c60e0 0x7f32500c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 --2- 172.21.15.71:0/2643930143 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3250152db0 0x7f32500c67c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 -- 172.21.15.71:0/2643930143 >> 172.21.15.71:0/2643930143 conn(0x7f325006d0d0 msgr2=0x7f32500ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 -- 172.21.15.71:0/2643930143 shutdown_connections 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.689+0000 7f324ffff640 1 -- 172.21.15.71:0/2643930143 wait complete. 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.691+0000 7f324ffff640 1 Processor -- start 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.691+0000 7f324ffff640 1 -- start start 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.692+0000 7f324ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3250071be0 0x7f32500c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.692+0000 7f324ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 0x7f3250148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.692+0000 7f324ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3250152db0 0x7f32501f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.692+0000 7f324ffff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3250154930 con 0x7f32500c60e0 2023-08-25T10:04:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.692+0000 7f324ffff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3250153500 con 0x7f3250152db0 2023-08-25T10:04:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.692+0000 7f324ffff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3250154f20 con 0x7f3250071be0 2023-08-25T10:04:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 0x7f3250148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 0x7f3250148910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:50636/0 (socket says 172.21.15.71:50636) 2023-08-25T10:04:08.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 -- 172.21.15.71:0/587203683 learned_addr learned my addr 172.21.15.71:0/587203683 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324effd640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3250071be0 0x7f32500c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324f7fe640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3250152db0 0x7f32501f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 -- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3250071be0 msgr2=0x7f32500c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3250071be0 0x7f32500c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 -- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3250152db0 msgr2=0x7f32501f11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3250152db0 0x7f32501f11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 -- 172.21.15.71:0/587203683 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32400094d0 con 0x7f32500c60e0 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.693+0000 7f324e7fc640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 0x7f3250148910 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f324400e870 tx=0x7f324400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.694+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32440094f0 con 0x7f32500c60e0 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.694+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324400c8e0 con 0x7f32500c60e0 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.694+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3250155e80 con 0x7f32500c60e0 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.694+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324400cc90 con 0x7f32500c60e0 2023-08-25T10:04:08.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.694+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32501536f0 con 0x7f32500c60e0 2023-08-25T10:04:08.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.695+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325004e420 con 0x7f32500c60e0 2023-08-25T10:04:08.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.696+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3244013470 con 0x7f32500c60e0 2023-08-25T10:04:08.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.698+0000 7f32377fe640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f323c06a080 0x7f323c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:08.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.698+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f3244088aa0 con 0x7f32500c60e0 2023-08-25T10:04:08.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.698+0000 7f324effd640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f323c06a080 0x7f323c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:08.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.699+0000 7f324effd640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f323c06a080 0x7f323c06c540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3240036000 tx=0x7f324003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:08.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.703+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3244053c30 con 0x7f32500c60e0 2023-08-25T10:04:08.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.980+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32500c5c00 con 0x7f32500c60e0 2023-08-25T10:04:08.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.981+0000 7f32377fe640 1 -- 172.21.15.71:0/587203683 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f3244057250 con 0x7f32500c60e0 2023-08-25T10:04:08.983 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:08.983 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:08.983 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:08.985 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f323c06a080 msgr2=0x7f323c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:08.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f323c06a080 0x7f323c06c540 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3240036000 tx=0x7f324003c040 comp rx=0 tx=0).stop 2023-08-25T10:04:08.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 msgr2=0x7f3250148910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:08.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 0x7f3250148910 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f324400e870 tx=0x7f324400ed30 comp rx=0 tx=0).stop 2023-08-25T10:04:08.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 shutdown_connections 2023-08-25T10:04:08.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3250152db0 0x7f32501f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f323c06a080 0x7f323c06c540 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f32500c60e0 0x7f3250148910 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 --2- 172.21.15.71:0/587203683 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3250071be0 0x7f32500c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:08.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 >> 172.21.15.71:0/587203683 conn(0x7f325006d0d0 msgr2=0x7f3250157590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:08.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 shutdown_connections 2023-08-25T10:04:08.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:08.984+0000 7f324ffff640 1 -- 172.21.15.71:0/587203683 wait complete. 2023-08-25T10:04:09.005 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:11.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:11.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.231+0000 7f92a185d640 1 Processor -- start 2023-08-25T10:04:11.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.232+0000 7f92a185d640 1 -- start start 2023-08-25T10:04:11.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f92a185d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f929c0c0260 0x7f929c06fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:11.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f92a185d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c070290 0x7f929c1634a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:11.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f92a185d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929c06f8c0 con 0x7f929c152bc0 2023-08-25T10:04:11.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f92a185d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c0706e0 con 0x7f929c0c0260 2023-08-25T10:04:11.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f92a185d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06f360 con 0x7f929c070290 2023-08-25T10:04:11.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f929bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f929c0c0260 0x7f929c06fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:11.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f929bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f929c0c0260 0x7f929c06fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35986/0 (socket says 172.21.15.71:35986) 2023-08-25T10:04:11.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f929bfff640 1 -- 172.21.15.71:0/4182287167 learned_addr learned my addr 172.21.15.71:0/4182287167 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:11.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.234+0000 7f92a085b640 1 --1- 172.21.15.71:0/4182287167 >> v1:172.21.15.71:6789/0 conn(0x7f929c152bc0 0x7f929c0c0ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60712/0 (socket says 172.21.15.71:60712) 2023-08-25T10:04:11.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.235+0000 7f92a105c640 1 --2- 172.21.15.71:0/4182287167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c070290 0x7f929c1634a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:11.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.235+0000 7f929bfff640 1 -- 172.21.15.71:0/4182287167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c070290 msgr2=0x7f929c1634a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:11.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.235+0000 7f929bfff640 1 --2- 172.21.15.71:0/4182287167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c070290 0x7f929c1634a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.235+0000 7f929bfff640 1 -- 172.21.15.71:0/4182287167 >> v1:172.21.15.71:6789/0 conn(0x7f929c152bc0 legacy=0x7f929c0c0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:11.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.235+0000 7f929bfff640 1 -- 172.21.15.71:0/4182287167 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c158ed0 con 0x7f929c0c0260 2023-08-25T10:04:11.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.235+0000 7f929b7fe640 1 -- 172.21.15.71:0/4182287167 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398563582 0 0) 0x7f929c06f8c0 con 0x7f929c152bc0 2023-08-25T10:04:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.236+0000 7f929bfff640 1 --2- 172.21.15.71:0/4182287167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c0c0260 0x7f929c06fd50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9288009870 tx=0x7f928802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b05a90e155cfac server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f929b7fe640 1 -- 172.21.15.71:0/4182287167 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928802f840 con 0x7f929c0c0260 2023-08-25T10:04:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f929b7fe640 1 -- 172.21.15.71:0/4182287167 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928802f9a0 con 0x7f929c0c0260 2023-08-25T10:04:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f929b7fe640 1 -- 172.21.15.71:0/4182287167 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928802fd30 con 0x7f929c0c0260 2023-08-25T10:04:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f92a185d640 1 -- 172.21.15.71:0/4182287167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c0c0260 msgr2=0x7f929c06fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f92a185d640 1 --2- 172.21.15.71:0/4182287167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c0c0260 0x7f929c06fd50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9288009870 tx=0x7f928802ba20 comp rx=0 tx=0).stop 2023-08-25T10:04:11.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f92a185d640 1 -- 172.21.15.71:0/4182287167 shutdown_connections 2023-08-25T10:04:11.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f92a185d640 1 --2- 172.21.15.71:0/4182287167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c070290 0x7f929c1634a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f92a185d640 1 --2- 172.21.15.71:0/4182287167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c0c0260 0x7f929c06fd50 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.237+0000 7f92a185d640 1 -- 172.21.15.71:0/4182287167 >> 172.21.15.71:0/4182287167 conn(0x7f929c0cb260 msgr2=0x7f929c0cb640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:11.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.238+0000 7f92a185d640 1 -- 172.21.15.71:0/4182287167 shutdown_connections 2023-08-25T10:04:11.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.238+0000 7f92a185d640 1 -- 172.21.15.71:0/4182287167 wait complete. 2023-08-25T10:04:11.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.239+0000 7f92a185d640 1 Processor -- start 2023-08-25T10:04:11.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.240+0000 7f92a185d640 1 -- start start 2023-08-25T10:04:11.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a185d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 0x7f929c1515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a185d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c0c0260 0x7f929c1ef260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a185d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f929c152bc0 0x7f929c1ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a185d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06fd50 con 0x7f929c070290 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a185d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06ff80 con 0x7f929c152bc0 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a185d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0c2eb0 con 0x7f929c0c0260 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a085b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 0x7f929c1515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:11.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a085b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 0x7f929c1515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35992/0 (socket says 172.21.15.71:35992) 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f92a085b640 1 -- 172.21.15.71:0/619216525 learned_addr learned my addr 172.21.15.71:0/619216525 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.241+0000 7f929bfff640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c0c0260 0x7f929c1ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a085b640 1 -- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c0c0260 msgr2=0x7f929c1ef260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a085b640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c0c0260 0x7f929c1ef260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a105c640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f929c152bc0 0x7f929c1ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a085b640 1 -- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f929c152bc0 msgr2=0x7f929c1ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a085b640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f929c152bc0 0x7f929c1ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a085b640 1 -- 172.21.15.71:0/619216525 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880094d0 con 0x7f929c070290 2023-08-25T10:04:11.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a085b640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 0x7f929c1515a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f929000c430 tx=0x7f929000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:11.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92900074e0 con 0x7f929c070290 2023-08-25T10:04:11.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9290007660 con 0x7f929c070290 2023-08-25T10:04:11.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c0c3190 con 0x7f929c070290 2023-08-25T10:04:11.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290004a20 con 0x7f929c070290 2023-08-25T10:04:11.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.242+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c0c3750 con 0x7f929c070290 2023-08-25T10:04:11.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.244+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c0c3da0 con 0x7f929c070290 2023-08-25T10:04:11.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.245+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9290004b80 con 0x7f929c070290 2023-08-25T10:04:11.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.246+0000 7f92997fa640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f928006a050 0x7f928006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:11.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.246+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9290088980 con 0x7f929c070290 2023-08-25T10:04:11.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.246+0000 7f929bfff640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f928006a050 0x7f928006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:11.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.247+0000 7f929bfff640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f928006a050 0x7f928006c510 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9288036000 tx=0x7f9288008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:11.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.251+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9290053b20 con 0x7f929c070290 2023-08-25T10:04:11.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.507+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f929c0713f0 con 0x7f929c070290 2023-08-25T10:04:11.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.508+0000 7f92997fa640 1 -- 172.21.15.71:0/619216525 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f9290057140 con 0x7f929c070290 2023-08-25T10:04:11.510 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:11.510 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:11.510 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:11.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.511+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f928006a050 msgr2=0x7f928006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:11.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.511+0000 7f92a185d640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f928006a050 0x7f928006c510 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9288036000 tx=0x7f9288008030 comp rx=0 tx=0).stop 2023-08-25T10:04:11.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.511+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 msgr2=0x7f929c1515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:11.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.511+0000 7f92a185d640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 0x7f929c1515a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f929000c430 tx=0x7f929000c8f0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 shutdown_connections 2023-08-25T10:04:11.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f929c152bc0 0x7f929c1ef7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f928006a050 0x7f928006c510 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f929c0c0260 0x7f929c1ef260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 --2- 172.21.15.71:0/619216525 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f929c070290 0x7f929c1515a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:11.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 >> 172.21.15.71:0/619216525 conn(0x7f929c0cb260 msgr2=0x7f929c0ca820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:11.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 shutdown_connections 2023-08-25T10:04:11.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:11.512+0000 7f92a185d640 1 -- 172.21.15.71:0/619216525 wait complete. 2023-08-25T10:04:11.531 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:13.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:13.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.774+0000 7f513a9ab640 1 Processor -- start 2023-08-25T10:04:13.776 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.774+0000 7f513a9ab640 1 -- start start 2023-08-25T10:04:13.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.776+0000 7f513a9ab640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5134152fc0 0x7f51340cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:13.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.776+0000 7f513a9ab640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f51340cad40 0x7f51340c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:13.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.776+0000 7f513a9ab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513415c6c0 con 0x7f5134158d60 2023-08-25T10:04:13.778 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.776+0000 7f513a9ab640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513415d260 con 0x7f51340cad40 2023-08-25T10:04:13.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.776+0000 7f513a9ab640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51341592c0 con 0x7f5134152fc0 2023-08-25T10:04:13.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.777+0000 7f513a1aa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f51340cad40 0x7f51340c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:13.779 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.777+0000 7f51399a9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5134152fc0 0x7f51340cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:13.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.777+0000 7f513a1aa640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f51340cad40 0x7f51340c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36006/0 (socket says 172.21.15.71:36006) 2023-08-25T10:04:13.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.777+0000 7f513a1aa640 1 -- 172.21.15.71:0/773967098 learned_addr learned my addr 172.21.15.71:0/773967098 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:13.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.777+0000 7f513a1aa640 1 -- 172.21.15.71:0/773967098 >> v2:172.21.15.164:3300/0 conn(0x7f5134152fc0 msgr2=0x7f51340cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:13.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.778+0000 7f513a1aa640 1 --2- 172.21.15.71:0/773967098 >> v2:172.21.15.164:3300/0 conn(0x7f5134152fc0 0x7f51340cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:13.780 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.778+0000 7f513a1aa640 1 -- 172.21.15.71:0/773967098 >> v1:172.21.15.71:6789/0 conn(0x7f5134158d60 legacy=0x7f513415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:13.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.778+0000 7f51389a7640 1 -- 172.21.15.71:0/773967098 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393737527 0 0) 0x7f513415c6c0 con 0x7f5134158d60 2023-08-25T10:04:13.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.778+0000 7f513a1aa640 1 -- 172.21.15.71:0/773967098 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51341595a0 con 0x7f51340cad40 2023-08-25T10:04:13.781 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.780+0000 7f513a1aa640 1 --2- 172.21.15.71:0/773967098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f51340c97a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f51340cf730 tx=0x7f512c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac5b1c0871d9f81 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:13.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.780+0000 7f51389a7640 1 -- 172.21.15.71:0/773967098 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c02cbb0 con 0x7f51340cad40 2023-08-25T10:04:13.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.780+0000 7f51389a7640 1 -- 172.21.15.71:0/773967098 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c02cd10 con 0x7f51340cad40 2023-08-25T10:04:13.782 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.780+0000 7f51389a7640 1 -- 172.21.15.71:0/773967098 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c034610 con 0x7f51340cad40 2023-08-25T10:04:13.783 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.781+0000 7f513a9ab640 1 -- 172.21.15.71:0/773967098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 msgr2=0x7f51340c97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:13.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.781+0000 7f513a9ab640 1 --2- 172.21.15.71:0/773967098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f51340c97a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f51340cf730 tx=0x7f512c02c010 comp rx=0 tx=0).stop 2023-08-25T10:04:13.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.782+0000 7f513a9ab640 1 -- 172.21.15.71:0/773967098 shutdown_connections 2023-08-25T10:04:13.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.782+0000 7f513a9ab640 1 --2- 172.21.15.71:0/773967098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f51340c97a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:13.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.782+0000 7f513a9ab640 1 --2- 172.21.15.71:0/773967098 >> v2:172.21.15.164:3300/0 conn(0x7f5134152fc0 0x7f51340cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:13.784 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.782+0000 7f513a9ab640 1 -- 172.21.15.71:0/773967098 >> 172.21.15.71:0/773967098 conn(0x7f513406a3a0 msgr2=0x7f5134070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:13.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.782+0000 7f513a9ab640 1 -- 172.21.15.71:0/773967098 shutdown_connections 2023-08-25T10:04:13.785 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.782+0000 7f513a9ab640 1 -- 172.21.15.71:0/773967098 wait complete. 2023-08-25T10:04:13.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.784+0000 7f513a9ab640 1 Processor -- start 2023-08-25T10:04:13.786 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.784+0000 7f513a9ab640 1 -- start start 2023-08-25T10:04:13.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.785+0000 7f513a9ab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f513415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:13.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f513a9ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5134152fc0 0x7f513414db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:13.787 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f513a9ab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5134158d60 0x7f51341f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:13.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f513a9ab640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51341587b0 con 0x7f51340cad40 2023-08-25T10:04:13.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f513a9ab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134157c00 con 0x7f5134152fc0 2023-08-25T10:04:13.789 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f513a9ab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5134158bd0 con 0x7f5134158d60 2023-08-25T10:04:13.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51399a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f513415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:13.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51399a9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f513415b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36014/0 (socket says 172.21.15.71:36014) 2023-08-25T10:04:13.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51399a9640 1 -- 172.21.15.71:0/1877266538 learned_addr learned my addr 172.21.15.71:0/1877266538 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:13.790 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51399a9640 1 -- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5134158d60 msgr2=0x7f51341f1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:13.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51399a9640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5134158d60 0x7f51341f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:13.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51391a8640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5134152fc0 0x7f513414db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:13.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.786+0000 7f51399a9640 1 -- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5134152fc0 msgr2=0x7f513414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:13.791 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f51399a9640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5134152fc0 0x7f513414db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:13.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f51399a9640 1 -- 172.21.15.71:0/1877266538 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c009b30 con 0x7f51340cad40 2023-08-25T10:04:13.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f51399a9640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f513415b930 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f513000e9b0 tx=0x7f513000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:13.792 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5130009670 con 0x7f51340cad40 2023-08-25T10:04:13.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513000c9a0 con 0x7f51340cad40 2023-08-25T10:04:13.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513000cd50 con 0x7f51340cad40 2023-08-25T10:04:13.793 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513414e050 con 0x7f51340cad40 2023-08-25T10:04:13.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.787+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5134157e10 con 0x7f51340cad40 2023-08-25T10:04:13.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.789+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513404e420 con 0x7f51340cad40 2023-08-25T10:04:13.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.790+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5130013470 con 0x7f51340cad40 2023-08-25T10:04:13.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.791+0000 7f512a7fc640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f511806a050 0x7f511806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:13.796 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.791+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f5130088d90 con 0x7f51340cad40 2023-08-25T10:04:13.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.792+0000 7f51391a8640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f511806a050 0x7f511806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:13.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.792+0000 7f51391a8640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f511806a050 0x7f511806c510 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f513404dca0 tx=0x7f5124009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:13.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:13.796+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5130053f20 con 0x7f51340cad40 2023-08-25T10:04:14.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.059+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51340c9580 con 0x7f51340cad40 2023-08-25T10:04:14.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.059+0000 7f512a7fc640 1 -- 172.21.15.71:0/1877266538 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f5130057540 con 0x7f51340cad40 2023-08-25T10:04:14.061 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:14.062 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:14.062 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:14.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f511806a050 msgr2=0x7f511806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:14.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f511806a050 0x7f511806c510 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f513404dca0 tx=0x7f5124009040 comp rx=0 tx=0).stop 2023-08-25T10:04:14.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 msgr2=0x7f513415b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:14.064 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f513415b930 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f513000e9b0 tx=0x7f513000ee70 comp rx=0 tx=0).stop 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 shutdown_connections 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5134152fc0 0x7f513414db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f511806a050 0x7f511806c510 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5134158d60 0x7f51341f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 --2- 172.21.15.71:0/1877266538 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51340cad40 0x7f513415b930 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 >> 172.21.15.71:0/1877266538 conn(0x7f513406a3a0 msgr2=0x7f51340cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:14.065 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 shutdown_connections 2023-08-25T10:04:14.066 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:14.062+0000 7f513a9ab640 1 -- 172.21.15.71:0/1877266538 wait complete. 2023-08-25T10:04:14.083 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:04:14.084 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:04:16.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:16.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.318+0000 7f070ac21640 1 Processor -- start 2023-08-25T10:04:16.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.318+0000 7f070ac21640 1 -- start start 2023-08-25T10:04:16.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070ac21640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07041532a0 0x7f070406ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070ac21640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f0704158f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070ac21640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f07040c7520 0x7f070406e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070ac21640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070406f250 con 0x7f07040c7520 2023-08-25T10:04:16.322 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070ac21640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f070415d350 con 0x7f07041532a0 2023-08-25T10:04:16.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070ac21640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0704159490 con 0x7f07040c6df0 2023-08-25T10:04:16.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070a420640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f07040c7520 0x7f070406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070a420640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f07040c7520 0x7f070406e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36022/0 (socket says 172.21.15.71:36022) 2023-08-25T10:04:16.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070a420640 1 -- 172.21.15.71:0/1181676777 learned_addr learned my addr 172.21.15.71:0/1181676777 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:16.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f070941e640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f0704158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.323 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.320+0000 7f0709c1f640 1 --2- 172.21.15.71:0/1181676777 >> v2:172.21.15.164:3300/0 conn(0x7f07041532a0 0x7f070406ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.321+0000 7f070a420640 1 -- 172.21.15.71:0/1181676777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 msgr2=0x7f0704158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:16.324 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.321+0000 7f070a420640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f0704158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.321+0000 7f070a420640 1 -- 172.21.15.71:0/1181676777 >> v2:172.21.15.164:3300/0 conn(0x7f07041532a0 msgr2=0x7f070406ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.321+0000 7f070a420640 1 --2- 172.21.15.71:0/1181676777 >> v2:172.21.15.164:3300/0 conn(0x7f07041532a0 0x7f070406ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.321+0000 7f070a420640 1 -- 172.21.15.71:0/1181676777 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070406ef70 con 0x7f07040c7520 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.321+0000 7f070941e640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f0704158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.322+0000 7f070a420640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07040c7520 0x7f070406e050 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0704071860 tx=0x7f06fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4fada1c8e5ede server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.322+0000 7f0708c1d640 1 -- 172.21.15.71:0/1181676777 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc02cb00 con 0x7f07040c7520 2023-08-25T10:04:16.325 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f0708c1d640 1 -- 172.21.15.71:0/1181676777 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc02cc60 con 0x7f07040c7520 2023-08-25T10:04:16.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f0708c1d640 1 -- 172.21.15.71:0/1181676777 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0343f0 con 0x7f07040c7520 2023-08-25T10:04:16.326 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 -- 172.21.15.71:0/1181676777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07040c7520 msgr2=0x7f070406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07040c7520 0x7f070406e050 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0704071860 tx=0x7f06fc02bf60 comp rx=0 tx=0).stop 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 -- 172.21.15.71:0/1181676777 shutdown_connections 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07040c7520 0x7f070406e050 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 --2- 172.21.15.71:0/1181676777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f0704158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 --2- 172.21.15.71:0/1181676777 >> v2:172.21.15.164:3300/0 conn(0x7f07041532a0 0x7f070406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 -- 172.21.15.71:0/1181676777 >> 172.21.15.71:0/1181676777 conn(0x7f07040654f0 msgr2=0x7f0704065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:16.327 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 -- 172.21.15.71:0/1181676777 shutdown_connections 2023-08-25T10:04:16.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.323+0000 7f070ac21640 1 -- 172.21.15.71:0/1181676777 wait complete. 2023-08-25T10:04:16.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.325+0000 7f070ac21640 1 Processor -- start 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.325+0000 7f070ac21640 1 -- start start 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.326+0000 7f070ac21640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f07041ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.326+0000 7f070ac21640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07040c7520 0x7f07041f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.326+0000 7f070ac21640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 0x7f070415b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.326+0000 7f070ac21640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0704154de0 con 0x7f07041532a0 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.326+0000 7f070ac21640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07041f1330 con 0x7f07040c7520 2023-08-25T10:04:16.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.326+0000 7f070ac21640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040bf280 con 0x7f07040c6df0 2023-08-25T10:04:16.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 0x7f070415b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 0x7f070415b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36024/0 (socket says 172.21.15.71:36024) 2023-08-25T10:04:16.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 -- 172.21.15.71:0/2656434960 learned_addr learned my addr 172.21.15.71:0/2656434960 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:16.330 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f0709c1f640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f07041ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070941e640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07040c7520 0x7f07041f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 -- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 msgr2=0x7f07041ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:16.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f07041ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 -- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07040c7520 msgr2=0x7f07041f17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07040c7520 0x7f07041f17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.327+0000 7f070a420640 1 -- 172.21.15.71:0/2656434960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc009b30 con 0x7f07041532a0 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.328+0000 7f070a420640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 0x7f070415b820 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f06fc036040 tx=0x7f06fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.328+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc009510 con 0x7f07041532a0 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.328+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc03e450 con 0x7f07041532a0 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.328+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc03e820 con 0x7f07041532a0 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.328+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0704154b00 con 0x7f07041532a0 2023-08-25T10:04:16.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.328+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07040cd190 con 0x7f07041532a0 2023-08-25T10:04:16.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.330+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070415cf50 con 0x7f07041532a0 2023-08-25T10:04:16.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.331+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f06fc03e980 con 0x7f07041532a0 2023-08-25T10:04:16.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.332+0000 7f06fa7fc640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f06e806a080 0x7f06e806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:16.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.332+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f06fc0ab510 con 0x7f07041532a0 2023-08-25T10:04:16.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.333+0000 7f0709c1f640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f06e806a080 0x7f06e806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:16.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.333+0000 7f0709c1f640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f06e806a080 0x7f06e806c540 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f070000b9b0 tx=0x7f0700008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:16.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.338+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f070415cf50 con 0x7f07041532a0 2023-08-25T10:04:16.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.609+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f070415cf50 con 0x7f07041532a0 2023-08-25T10:04:16.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.610+0000 7f06fa7fc640 1 -- 172.21.15.71:0/2656434960 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f070415cf50 con 0x7f07041532a0 2023-08-25T10:04:16.612 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:16.612 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:16.612 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:16.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f06e806a080 msgr2=0x7f06e806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:16.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f06e806a080 0x7f06e806c540 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f070000b9b0 tx=0x7f0700008040 comp rx=0 tx=0).stop 2023-08-25T10:04:16.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 msgr2=0x7f070415b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 0x7f070415b820 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f06fc036040 tx=0x7f06fc004a40 comp rx=0 tx=0).stop 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 shutdown_connections 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07040c7520 0x7f07041f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f06e806a080 0x7f06e806c540 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07041532a0 0x7f070415b820 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 --2- 172.21.15.71:0/2656434960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07040c6df0 0x7f07041ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:16.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.612+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 >> 172.21.15.71:0/2656434960 conn(0x7f07040654f0 msgr2=0x7f0704156f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:16.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.613+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 shutdown_connections 2023-08-25T10:04:16.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:16.613+0000 7f070ac21640 1 -- 172.21.15.71:0/2656434960 wait complete. 2023-08-25T10:04:16.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:18.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:18.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.872+0000 7fb849883640 1 Processor -- start 2023-08-25T10:04:18.874 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.872+0000 7fb849883640 1 -- start start 2023-08-25T10:04:18.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.874+0000 7fb849883640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 0x7fb8440c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.874+0000 7fb849883640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb844158e40 con 0x7fb8441592d0 2023-08-25T10:04:18.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.874+0000 7fb849883640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8441534a0 con 0x7fb8440c7740 2023-08-25T10:04:18.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.874+0000 7fb849883640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84406bc80 con 0x7fb8441530c0 2023-08-25T10:04:18.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.874+0000 7fb8437fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb8441592d0 0x7fb844159730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60734/0 (socket says 172.21.15.71:60734) 2023-08-25T10:04:18.877 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.874+0000 7fb8437fe640 1 -- 172.21.15.71:0/2837433624 learned_addr learned my addr 172.21.15.71:0/2837433624 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:18.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 0x7fb8440c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.878 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486602436 0 0) 0x7fb844158e40 con 0x7fb8441592d0 2023-08-25T10:04:18.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb830003520 con 0x7fb8441592d0 2023-08-25T10:04:18.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371573487 0 0) 0x7fb830003520 con 0x7fb8441592d0 2023-08-25T10:04:18.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 msgr2=0x7fb8440c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:18.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 0x7fb8440c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:18.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 >> v1:172.21.15.164:6789/0 conn(0x7fb8440c7740 legacy=0x7fb84406a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:18.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8440c7380 con 0x7fb8441592d0 2023-08-25T10:04:18.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fb83c0019f0 con 0x7fb8441592d0 2023-08-25T10:04:18.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 >> v1:172.21.15.71:6789/0 conn(0x7fb8441592d0 legacy=0x7fb844159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:18.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb830003a10 0x7fb830003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 0x7fb8300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb830004d00 0x7fb830005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83000baf0 con 0x7fb830003a10 2023-08-25T10:04:18.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83000bc60 con 0x7fb830004d00 2023-08-25T10:04:18.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.875+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83000be20 con 0x7fb8441530c0 2023-08-25T10:04:18.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb830003a10 0x7fb830003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb83bfff640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 0x7fb8300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb842ffd640 1 -- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 msgr2=0x7fb8300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb8437fe640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb830004d00 0x7fb830005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8441530c0 0x7fb8300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb842ffd640 1 -- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb830004d00 msgr2=0x7fb830005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb830004d00 0x7fb830005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:18.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb842ffd640 1 -- 172.21.15.71:0/2837433624 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830003520 con 0x7fb830003a10 2023-08-25T10:04:18.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.876+0000 7fb8437fe640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb830004d00 0x7fb830005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:04:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.878+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb830003a10 0x7fb830003e30 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fb834009a90 tx=0x7fb8300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c3012acbf9666e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.878+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83402c5c0 con 0x7fb830003a10 2023-08-25T10:04:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.878+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83402c720 con 0x7fb830003a10 2023-08-25T10:04:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.878+0000 7fb8427fc640 1 -- 172.21.15.71:0/2837433624 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83402cab0 con 0x7fb830003a10 2023-08-25T10:04:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb849883640 1 -- 172.21.15.71:0/2837433624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb830003a10 msgr2=0x7fb830003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:18.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb849883640 1 --2- 172.21.15.71:0/2837433624 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb830003a10 0x7fb830003e30 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fb834009a90 tx=0x7fb8300057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:18.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb8437fe640 1 -- 172.21.15.71:0/2837433624 reap_dead start 2023-08-25T10:04:18.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb849883640 1 -- 172.21.15.71:0/2837433624 shutdown_connections 2023-08-25T10:04:18.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb849883640 1 -- 172.21.15.71:0/2837433624 >> 172.21.15.71:0/2837433624 conn(0x7fb8440654f0 msgr2=0x7fb844065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:18.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb849883640 1 -- 172.21.15.71:0/2837433624 shutdown_connections 2023-08-25T10:04:18.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.879+0000 7fb849883640 1 -- 172.21.15.71:0/2837433624 wait complete. 2023-08-25T10:04:18.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.881+0000 7fb849883640 1 Processor -- start 2023-08-25T10:04:18.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.881+0000 7fb849883640 1 -- start start 2023-08-25T10:04:18.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb849883640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb830003a10 0x7fb844157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb849883640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 0x7fb8441f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb849883640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8441f05e0 0x7fb8441f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb849883640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440c55c0 con 0x7fb8441e4c90 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb849883640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8441f0a70 con 0x7fb8441f05e0 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb849883640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84414f8a0 con 0x7fb830003a10 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb83bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 0x7fb8441f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb83bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 0x7fb8441f00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36040/0 (socket says 172.21.15.71:36040) 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.882+0000 7fb83bfff640 1 -- 172.21.15.71:0/2867880514 learned_addr learned my addr 172.21.15.71:0/2867880514 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb830003a10 0x7fb844157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb8437fe640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8441f05e0 0x7fb8441f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb83bfff640 1 -- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb830003a10 msgr2=0x7fb844157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:18.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb83bfff640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb830003a10 0x7fb844157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:18.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb83bfff640 1 -- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8441f05e0 msgr2=0x7fb8441f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:18.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb83bfff640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8441f05e0 0x7fb8441f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:18.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb83bfff640 1 -- 172.21.15.71:0/2867880514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8441519f0 con 0x7fb8441e4c90 2023-08-25T10:04:18.889 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb830003a10 0x7fb844157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:04:18.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.883+0000 7fb83bfff640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 0x7fb8441f00a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00e550 tx=0x7fb82c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:18.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.884+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c0052e0 con 0x7fb8441e4c90 2023-08-25T10:04:18.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.884+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c005440 con 0x7fb8441e4c90 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.884+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c003810 con 0x7fb8441e4c90 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.884+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8440c57f0 con 0x7fb8441e4c90 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.884+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb844151710 con 0x7fb8441e4c90 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.885+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb844069160 con 0x7fb8441e4c90 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.886+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb82c003970 con 0x7fb8441e4c90 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.887+0000 7fb840ff9640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb82406a080 0x7fb82406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:18.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.888+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb82c088970 con 0x7fb8441e4c90 2023-08-25T10:04:18.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.888+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb82406a080 0x7fb82406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:18.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.888+0000 7fb842ffd640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb82406a080 0x7fb82406c540 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb834036000 tx=0x7fb834030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:18.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:18.893+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb82c053a70 con 0x7fb8441e4c90 2023-08-25T10:04:19.173 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.172+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb84414f9e0 con 0x7fb8441e4c90 2023-08-25T10:04:19.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.173+0000 7fb840ff9640 1 -- 172.21.15.71:0/2867880514 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fb82c057090 con 0x7fb8441e4c90 2023-08-25T10:04:19.175 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:19.175 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:19.175 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:19.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb82406a080 msgr2=0x7fb82406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:19.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb82406a080 0x7fb82406c540 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb834036000 tx=0x7fb834030000 comp rx=0 tx=0).stop 2023-08-25T10:04:19.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 msgr2=0x7fb8441f00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 0x7fb8441f00a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00e550 tx=0x7fb82c00ea10 comp rx=0 tx=0).stop 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 shutdown_connections 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8441f05e0 0x7fb8441f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb82406a080 0x7fb82406c540 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8441e4c90 0x7fb8441f00a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 --2- 172.21.15.71:0/2867880514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb830003a10 0x7fb844157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:19.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.175+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 >> 172.21.15.71:0/2867880514 conn(0x7fb8440654f0 msgr2=0x7fb8440707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:19.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.176+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 shutdown_connections 2023-08-25T10:04:19.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:19.176+0000 7fb849883640 1 -- 172.21.15.71:0/2867880514 wait complete. 2023-08-25T10:04:19.198 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:21.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:21.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.455+0000 7f9fed6b0640 1 Processor -- start 2023-08-25T10:04:21.457 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.455+0000 7f9fed6b0640 1 -- start start 2023-08-25T10:04:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fed6b0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fe81542d0 0x7f9fe81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fed6b0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fe80c4fb0 0x7f9fe80c5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fed6b0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe8157bf0 con 0x7f9fe8069850 2023-08-25T10:04:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fed6b0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe80c4b60 con 0x7f9fe81542d0 2023-08-25T10:04:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fed6b0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe806a130 con 0x7f9fe80c4fb0 2023-08-25T10:04:21.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fe7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fe81542d0 0x7f9fe81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:21.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fe77fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9fe8069850 0x7f9fe80c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50558/0 (socket says 172.21.15.71:50558) 2023-08-25T10:04:21.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fe77fe640 1 -- 172.21.15.71:0/2475827929 learned_addr learned my addr 172.21.15.71:0/2475827929 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:21.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.457+0000 7f9fe7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9fe81542d0 0x7f9fe81576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59112/0 (socket says 172.21.15.71:59112) 2023-08-25T10:04:21.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9feceaf640 1 --2- 172.21.15.71:0/2475827929 >> v2:172.21.15.164:3300/0 conn(0x7f9fe80c4fb0 0x7f9fe80c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:21.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9fe7fff640 1 -- 172.21.15.71:0/2475827929 >> v2:172.21.15.164:3300/0 conn(0x7f9fe80c4fb0 msgr2=0x7f9fe80c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:21.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9fe7fff640 1 --2- 172.21.15.71:0/2475827929 >> v2:172.21.15.164:3300/0 conn(0x7f9fe80c4fb0 0x7f9fe80c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9fe7fff640 1 -- 172.21.15.71:0/2475827929 >> v1:172.21.15.71:6789/0 conn(0x7f9fe8069850 legacy=0x7f9fe80c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:21.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9fe7fff640 1 -- 172.21.15.71:0/2475827929 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe80c5980 con 0x7f9fe81542d0 2023-08-25T10:04:21.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9feceaf640 1 --2- 172.21.15.71:0/2475827929 >> v2:172.21.15.164:3300/0 conn(0x7f9fe80c4fb0 0x7f9fe80c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:04:21.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.458+0000 7f9fe6ffd640 1 -- 172.21.15.71:0/2475827929 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125120946 0 0) 0x7f9fdc001110 con 0x7f9fe8069850 2023-08-25T10:04:21.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.459+0000 7f9fe7fff640 1 --2- 172.21.15.71:0/2475827929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe81542d0 0x7f9fe81576b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80c1800 tx=0x7f9fd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2a4922e5a7d7cd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:21.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.459+0000 7f9fe6ffd640 1 -- 172.21.15.71:0/2475827929 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8031830 con 0x7f9fe81542d0 2023-08-25T10:04:21.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.459+0000 7f9fe6ffd640 1 -- 172.21.15.71:0/2475827929 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd8031990 con 0x7f9fe81542d0 2023-08-25T10:04:21.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.459+0000 7f9fe6ffd640 1 -- 172.21.15.71:0/2475827929 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8031d20 con 0x7f9fe81542d0 2023-08-25T10:04:21.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 -- 172.21.15.71:0/2475827929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe81542d0 msgr2=0x7f9fe81576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:21.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/2475827929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe81542d0 0x7f9fe81576b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80c1800 tx=0x7f9fd802b8d0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 -- 172.21.15.71:0/2475827929 shutdown_connections 2023-08-25T10:04:21.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/2475827929 >> v2:172.21.15.164:3300/0 conn(0x7f9fe80c4fb0 0x7f9fe80c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/2475827929 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe81542d0 0x7f9fe81576b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 -- 172.21.15.71:0/2475827929 >> 172.21.15.71:0/2475827929 conn(0x7f9fe806c280 msgr2=0x7f9fe806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:21.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 -- 172.21.15.71:0/2475827929 shutdown_connections 2023-08-25T10:04:21.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.460+0000 7f9fed6b0640 1 -- 172.21.15.71:0/2475827929 wait complete. 2023-08-25T10:04:21.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.462+0000 7f9fed6b0640 1 Processor -- start 2023-08-25T10:04:21.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.463+0000 7f9fed6b0640 1 -- start start 2023-08-25T10:04:21.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fed6b0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 0x7f9fe81524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:21.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fed6b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fe80c4fb0 0x7f9fe8152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:21.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fed6b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe81542d0 0x7f9fe814b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:21.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fed6b0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe81f1190 con 0x7f9fe8069850 2023-08-25T10:04:21.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fed6b0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe81f1fb0 con 0x7f9fe81542d0 2023-08-25T10:04:21.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fed6b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe80c9df0 con 0x7f9fe80c4fb0 2023-08-25T10:04:21.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 0x7f9fe81524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:21.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fe7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 0x7f9fe81524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59128/0 (socket says 172.21.15.71:59128) 2023-08-25T10:04:21.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.464+0000 7f9fe7fff640 1 -- 172.21.15.71:0/4180429704 learned_addr learned my addr 172.21.15.71:0/4180429704 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:21.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9feceaf640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe81542d0 0x7f9fe814b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:21.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe77fe640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fe80c4fb0 0x7f9fe8152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:21.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe7fff640 1 -- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fe80c4fb0 msgr2=0x7f9fe8152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:21.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe7fff640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fe80c4fb0 0x7f9fe8152a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe7fff640 1 -- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe81542d0 msgr2=0x7f9fe814b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe7fff640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe81542d0 0x7f9fe814b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe7fff640 1 -- 172.21.15.71:0/4180429704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd8009400 con 0x7f9fe8069850 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9feceaf640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe81542d0 0x7f9fe814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe7fff640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 0x7f9fe81524e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009890 tx=0x7f9fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8004230 con 0x7f9fe8069850 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd8004390 con 0x7f9fe8069850 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd803a920 con 0x7f9fe8069850 2023-08-25T10:04:21.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.465+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe81f13a0 con 0x7f9fe8069850 2023-08-25T10:04:21.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.466+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe81f1960 con 0x7f9fe8069850 2023-08-25T10:04:21.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.467+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe804e420 con 0x7f9fe8069850 2023-08-25T10:04:21.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.468+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9fd803aa80 con 0x7f9fe8069850 2023-08-25T10:04:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.469+0000 7f9fe4ff9640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd406a050 0x7f9fd406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.469+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9fd80ab210 con 0x7f9fe8069850 2023-08-25T10:04:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.469+0000 7f9fe77fe640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd406a050 0x7f9fd406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:21.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.470+0000 7f9fe77fe640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd406a050 0x7f9fd406c510 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9fe81eb850 tx=0x7f9fdc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:21.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.475+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9fd80763a0 con 0x7f9fe8069850 2023-08-25T10:04:21.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.738+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fe8069f10 con 0x7f9fe8069850 2023-08-25T10:04:21.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.740+0000 7f9fe4ff9640 1 -- 172.21.15.71:0/4180429704 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f9fd80799c0 con 0x7f9fe8069850 2023-08-25T10:04:21.742 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:21.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:21.742 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:21.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd406a050 msgr2=0x7f9fd406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:21.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd406a050 0x7f9fd406c510 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9fe81eb850 tx=0x7f9fdc009040 comp rx=0 tx=0).stop 2023-08-25T10:04:21.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 msgr2=0x7f9fe81524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:21.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 0x7f9fe81524e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009890 tx=0x7f9fd80046c0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 shutdown_connections 2023-08-25T10:04:21.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe81542d0 0x7f9fe814b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9fd406a050 0x7f9fd406c510 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fe80c4fb0 0x7f9fe8152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 --2- 172.21.15.71:0/4180429704 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9fe8069850 0x7f9fe81524e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:21.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.742+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 >> 172.21.15.71:0/4180429704 conn(0x7f9fe806c280 msgr2=0x7f9fe806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:21.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.743+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 shutdown_connections 2023-08-25T10:04:21.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:21.743+0000 7f9fed6b0640 1 -- 172.21.15.71:0/4180429704 wait complete. 2023-08-25T10:04:21.815 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:23.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:24.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.070+0000 7f03a7fff640 1 Processor -- start 2023-08-25T10:04:24.073 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.070+0000 7f03a7fff640 1 -- start start 2023-08-25T10:04:24.074 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03a80cb190 0x7f03a80ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:24.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a7fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03a815cd00 0x7f03a8158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:24.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a7fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a80b1af0 con 0x7f03a80cb570 2023-08-25T10:04:24.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a7fff640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a80c98f0 con 0x7f03a815cd00 2023-08-25T10:04:24.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a7fff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a80b0fb0 con 0x7f03a80cb190 2023-08-25T10:04:24.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a77fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03a815cd00 0x7f03a8158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:24.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a77fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f03a815cd00 0x7f03a8158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59132/0 (socket says 172.21.15.71:59132) 2023-08-25T10:04:24.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a77fe640 1 -- 172.21.15.71:0/5474405 learned_addr learned my addr 172.21.15.71:0/5474405 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:24.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.072+0000 7f03a6ffd640 1 --2- 172.21.15.71:0/5474405 >> v2:172.21.15.164:3300/0 conn(0x7f03a80cb190 0x7f03a80ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:24.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.073+0000 7f03a77fe640 1 -- 172.21.15.71:0/5474405 >> v2:172.21.15.164:3300/0 conn(0x7f03a80cb190 msgr2=0x7f03a80ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:24.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.073+0000 7f03a77fe640 1 --2- 172.21.15.71:0/5474405 >> v2:172.21.15.164:3300/0 conn(0x7f03a80cb190 0x7f03a80ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.073+0000 7f03a77fe640 1 -- 172.21.15.71:0/5474405 >> v1:172.21.15.71:6789/0 conn(0x7f03a80cb570 legacy=0x7f03a80c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:24.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.073+0000 7f03a77fe640 1 -- 172.21.15.71:0/5474405 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a815d130 con 0x7f03a815cd00 2023-08-25T10:04:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.074+0000 7f03a77fe640 1 --2- 172.21.15.71:0/5474405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a8158d80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f03a80cc000 tx=0x7f03a002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee61d1e376b1dceb server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.074+0000 7f03a5ffb640 1 -- 172.21.15.71:0/5474405 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a002cbb0 con 0x7f03a815cd00 2023-08-25T10:04:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.074+0000 7f03a5ffb640 1 -- 172.21.15.71:0/5474405 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a002cd10 con 0x7f03a815cd00 2023-08-25T10:04:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.074+0000 7f03a5ffb640 1 -- 172.21.15.71:0/5474405 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0034610 con 0x7f03a815cd00 2023-08-25T10:04:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 -- 172.21.15.71:0/5474405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 msgr2=0x7f03a8158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 --2- 172.21.15.71:0/5474405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a8158d80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f03a80cc000 tx=0x7f03a002c010 comp rx=0 tx=0).stop 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 -- 172.21.15.71:0/5474405 shutdown_connections 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 --2- 172.21.15.71:0/5474405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a8158d80 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 --2- 172.21.15.71:0/5474405 >> v2:172.21.15.164:3300/0 conn(0x7f03a80cb190 0x7f03a80ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 -- 172.21.15.71:0/5474405 >> 172.21.15.71:0/5474405 conn(0x7f03a806ae20 msgr2=0x7f03a806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 -- 172.21.15.71:0/5474405 shutdown_connections 2023-08-25T10:04:24.079 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.075+0000 7f03a7fff640 1 -- 172.21.15.71:0/5474405 wait complete. 2023-08-25T10:04:24.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.077+0000 7f03a7fff640 1 Processor -- start 2023-08-25T10:04:24.080 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.077+0000 7f03a7fff640 1 -- start start 2023-08-25T10:04:24.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.078+0000 7f03a7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a80cb190 0x7f03a80c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:24.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.078+0000 7f03a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a80cb570 0x7f03a81f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:24.082 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.078+0000 7f03a7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.078+0000 7f03a7fff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8159ac0 con 0x7f03a815cd00 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.078+0000 7f03a7fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8159d10 con 0x7f03a80cb570 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.078+0000 7f03a7fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a814c780 con 0x7f03a80cb190 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a814b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59134/0 (socket says 172.21.15.71:59134) 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 -- 172.21.15.71:0/645647386 learned_addr learned my addr 172.21.15.71:0/645647386 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:24.083 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a6ffd640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a80cb190 0x7f03a80c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:24.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 -- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a80cb190 msgr2=0x7f03a80c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:24.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a67fc640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a80cb570 0x7f03a81f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:24.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a80cb190 0x7f03a80c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 -- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a80cb570 msgr2=0x7f03a81f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:24.084 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a80cb570 0x7f03a81f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a77fe640 1 -- 172.21.15.71:0/645647386 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0009b30 con 0x7f03a815cd00 2023-08-25T10:04:24.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.079+0000 7f03a67fc640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a80cb570 0x7f03a81f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:04:24.085 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.080+0000 7f03a77fe640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a814b760 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009f40 tx=0x7f03a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:24.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.080+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0009510 con 0x7f03a815cd00 2023-08-25T10:04:24.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.080+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a003e450 con 0x7f03a815cd00 2023-08-25T10:04:24.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.080+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a814ca60 con 0x7f03a815cd00 2023-08-25T10:04:24.086 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.080+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a003e7e0 con 0x7f03a815cd00 2023-08-25T10:04:24.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.080+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a80ce800 con 0x7f03a815cd00 2023-08-25T10:04:24.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.081+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a80ce9f0 con 0x7f03a815cd00 2023-08-25T10:04:24.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.082+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f03a003e940 con 0x7f03a815cd00 2023-08-25T10:04:24.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.083+0000 7f038ffff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f039406a050 0x7f039406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:24.087 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.083+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f03a00ab8d0 con 0x7f03a815cd00 2023-08-25T10:04:24.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.084+0000 7f03a6ffd640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f039406a050 0x7f039406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:24.088 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.084+0000 7f03a6ffd640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f039406a050 0x7f039406c510 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0398006fd0 tx=0x7f0398008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:24.091 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.088+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f03a0076b80 con 0x7f03a815cd00 2023-08-25T10:04:24.331 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.328+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03a804e420 con 0x7f03a815cd00 2023-08-25T10:04:24.332 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.329+0000 7f038ffff640 1 -- 172.21.15.71:0/645647386 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f03a007a1a0 con 0x7f03a815cd00 2023-08-25T10:04:24.332 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:24.333 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:24.333 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:24.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.331+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f039406a050 msgr2=0x7f039406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:24.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.331+0000 7f03a7fff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f039406a050 0x7f039406c510 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0398006fd0 tx=0x7f0398008040 comp rx=0 tx=0).stop 2023-08-25T10:04:24.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.331+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 msgr2=0x7f03a814b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:24.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.331+0000 7f03a7fff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a814b760 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009f40 tx=0x7f03a0004a40 comp rx=0 tx=0).stop 2023-08-25T10:04:24.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 shutdown_connections 2023-08-25T10:04:24.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a80cb570 0x7f03a81f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f039406a050 0x7f039406c510 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03a815cd00 0x7f03a814b760 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 --2- 172.21.15.71:0/645647386 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a80cb190 0x7f03a80c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:24.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 >> 172.21.15.71:0/645647386 conn(0x7f03a806ae20 msgr2=0x7f03a815a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:24.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 shutdown_connections 2023-08-25T10:04:24.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:24.332+0000 7f03a7fff640 1 -- 172.21.15.71:0/645647386 wait complete. 2023-08-25T10:04:24.357 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:26.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:26.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.597+0000 7f23f69f3640 1 Processor -- start 2023-08-25T10:04:26.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.597+0000 7f23f69f3640 1 -- start start 2023-08-25T10:04:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.600+0000 7f23f69f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f23f01510e0 0x7f23f00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.600+0000 7f23f69f3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f00c1dc0 con 0x7f23f00c3660 2023-08-25T10:04:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.600+0000 7f23f69f3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f0156e00 con 0x7f23f00c3a40 2023-08-25T10:04:26.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.600+0000 7f23f69f3640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f0157a80 con 0x7f23f01510e0 2023-08-25T10:04:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.601+0000 7f23effff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f23f00c3660 0x7f23f006ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:50582/0 (socket says 172.21.15.71:50582) 2023-08-25T10:04:26.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.601+0000 7f23effff640 1 -- 172.21.15.71:0/3584830423 learned_addr learned my addr 172.21.15.71:0/3584830423 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:26.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.601+0000 7f23f4f90640 1 --2- 172.21.15.71:0/3584830423 >> v2:172.21.15.71:3300/0 conn(0x7f23f01510e0 0x7f23f00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:26.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.601+0000 7f23eeffd640 1 -- 172.21.15.71:0/3584830423 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669397827 0 0) 0x7f23f00c1dc0 con 0x7f23f00c3660 2023-08-25T10:04:26.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.601+0000 7f23eeffd640 1 -- 172.21.15.71:0/3584830423 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d8003520 con 0x7f23f00c3660 2023-08-25T10:04:26.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.602+0000 7f23f4f90640 1 -- 172.21.15.71:0/3584830423 >> v1:172.21.15.164:6789/0 conn(0x7f23f00c3a40 legacy=0x7f23f01509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.602+0000 7f23f4f90640 1 -- 172.21.15.71:0/3584830423 >> v1:172.21.15.71:6789/0 conn(0x7f23f00c3660 legacy=0x7f23f006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.602+0000 7f23f4f90640 1 -- 172.21.15.71:0/3584830423 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f01570e0 con 0x7f23f01510e0 2023-08-25T10:04:26.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.603+0000 7f23f4f90640 1 --2- 172.21.15.71:0/3584830423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f01510e0 0x7f23f00c1710 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009d90 tx=0x7f23e802bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996fe1c19adff103 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:26.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.603+0000 7f23eeffd640 1 -- 172.21.15.71:0/3584830423 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e802cd80 con 0x7f23f01510e0 2023-08-25T10:04:26.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.603+0000 7f23eeffd640 1 -- 172.21.15.71:0/3584830423 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e8034440 con 0x7f23f01510e0 2023-08-25T10:04:26.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.603+0000 7f23eeffd640 1 -- 172.21.15.71:0/3584830423 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80347d0 con 0x7f23f01510e0 2023-08-25T10:04:26.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.604+0000 7f23f69f3640 1 -- 172.21.15.71:0/3584830423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f01510e0 msgr2=0x7f23f00c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.604+0000 7f23f69f3640 1 --2- 172.21.15.71:0/3584830423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f01510e0 0x7f23f00c1710 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009d90 tx=0x7f23e802bfd0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.604+0000 7f23f69f3640 1 -- 172.21.15.71:0/3584830423 shutdown_connections 2023-08-25T10:04:26.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.604+0000 7f23f69f3640 1 --2- 172.21.15.71:0/3584830423 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f01510e0 0x7f23f00c1710 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.604+0000 7f23f69f3640 1 -- 172.21.15.71:0/3584830423 >> 172.21.15.71:0/3584830423 conn(0x7f23f00654f0 msgr2=0x7f23f0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:26.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.604+0000 7f23f69f3640 1 -- 172.21.15.71:0/3584830423 shutdown_connections 2023-08-25T10:04:26.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.605+0000 7f23f69f3640 1 -- 172.21.15.71:0/3584830423 wait complete. 2023-08-25T10:04:26.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.606+0000 7f23f69f3640 1 Processor -- start 2023-08-25T10:04:26.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.607+0000 7f23f69f3640 1 -- start start 2023-08-25T10:04:26.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23f69f3640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 0x7f23f014b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:26.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23f69f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f00c3a40 0x7f23f0155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:26.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23f69f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f01510e0 0x7f23f014a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:26.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23f69f3640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0155370 con 0x7f23f00c3660 2023-08-25T10:04:26.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23f69f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0155930 con 0x7f23f00c3a40 2023-08-25T10:04:26.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23f69f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0148b30 con 0x7f23f01510e0 2023-08-25T10:04:26.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23effff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 0x7f23f014b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:26.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23effff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 0x7f23f014b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59156/0 (socket says 172.21.15.71:59156) 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23effff640 1 -- 172.21.15.71:0/1065656551 learned_addr learned my addr 172.21.15.71:0/1065656551 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23ef7fe640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f00c3a40 0x7f23f0155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23effff640 1 -- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f01510e0 msgr2=0x7f23f014a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.608+0000 7f23effff640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f01510e0 0x7f23f014a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23effff640 1 -- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f00c3a40 msgr2=0x7f23f0155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23effff640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f00c3a40 0x7f23f0155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23effff640 1 -- 172.21.15.71:0/1065656551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8009a70 con 0x7f23f00c3660 2023-08-25T10:04:26.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23effff640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 0x7f23f014b1a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f23e000c410 tx=0x7f23e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:26.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0005ce0 con 0x7f23f00c3660 2023-08-25T10:04:26.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e0005e40 con 0x7f23f00c3660 2023-08-25T10:04:26.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e00072f0 con 0x7f23f00c3660 2023-08-25T10:04:26.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f0148da0 con 0x7f23f00c3660 2023-08-25T10:04:26.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.609+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f01555c0 con 0x7f23f00c3660 2023-08-25T10:04:26.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.611+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f0155780 con 0x7f23f00c3660 2023-08-25T10:04:26.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.612+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f23e0007450 con 0x7f23f00c3660 2023-08-25T10:04:26.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.613+0000 7f23ecff9640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f23dc06a080 0x7f23dc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:26.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.613+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f23e0087ff0 con 0x7f23f00c3660 2023-08-25T10:04:26.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.613+0000 7f23ef7fe640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f23dc06a080 0x7f23dc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:26.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.614+0000 7f23ef7fe640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f23dc06a080 0x7f23dc06c540 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f23f0154650 tx=0x7f23e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:26.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.618+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f23e0056e00 con 0x7f23f00c3660 2023-08-25T10:04:26.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.899+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23f0069160 con 0x7f23f00c3660 2023-08-25T10:04:26.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.900+0000 7f23ecff9640 1 -- 172.21.15.71:0/1065656551 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f23e0056990 con 0x7f23f00c3660 2023-08-25T10:04:26.904 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:26.904 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:26.904 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:26.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.903+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f23dc06a080 msgr2=0x7f23dc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.903+0000 7f23f69f3640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f23dc06a080 0x7f23dc06c540 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f23f0154650 tx=0x7f23e4009040 comp rx=0 tx=0).stop 2023-08-25T10:04:26.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.903+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 msgr2=0x7f23f014b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:26.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.903+0000 7f23f69f3640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 0x7f23f014b1a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f23e000c410 tx=0x7f23e000c8d0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 shutdown_connections 2023-08-25T10:04:26.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f00c3a40 0x7f23f0155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f23dc06a080 0x7f23dc06c540 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f01510e0 0x7f23f014a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 --2- 172.21.15.71:0/1065656551 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23f00c3660 0x7f23f014b1a0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:26.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 >> 172.21.15.71:0/1065656551 conn(0x7f23f00654f0 msgr2=0x7f23f00c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:26.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 shutdown_connections 2023-08-25T10:04:26.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:26.904+0000 7f23f69f3640 1 -- 172.21.15.71:0/1065656551 wait complete. 2023-08-25T10:04:26.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:04:26.923 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:04:26.925 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:04:26.928 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:04:28.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:29.154 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.151+0000 7f8ccd871640 1 Processor -- start 2023-08-25T10:04:29.155 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.151+0000 7f8ccd871640 1 -- start start 2023-08-25T10:04:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.153+0000 7f8ccd871640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8cc80cb190 0x7f8cc80ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:29.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.153+0000 7f8ccd871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80cb570 0x7f8cc815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:29.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.153+0000 7f8ccd871640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc80b1af0 con 0x7f8cc80c7f10 2023-08-25T10:04:29.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.153+0000 7f8ccd871640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc80c98f0 con 0x7f8cc80cb190 2023-08-25T10:04:29.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.153+0000 7f8ccd871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc80b0fb0 con 0x7f8cc80cb570 2023-08-25T10:04:29.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.154+0000 7f8ccc86f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8cc80cb190 0x7f8cc80ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:29.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.154+0000 7f8cc7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80cb570 0x7f8cc815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:29.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.154+0000 7f8ccc86f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8cc80cb190 0x7f8cc80ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:49504/0 (socket says 172.21.15.71:49504) 2023-08-25T10:04:29.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.154+0000 7f8ccc86f640 1 -- 172.21.15.71:0/3253395724 learned_addr learned my addr 172.21.15.71:0/3253395724 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:29.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.155+0000 7f8ccc86f640 1 -- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80cb570 msgr2=0x7f8cc815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:29.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.155+0000 7f8ccc86f640 1 --2- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80cb570 0x7f8cc815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.155+0000 7f8ccc86f640 1 -- 172.21.15.71:0/3253395724 >> v1:172.21.15.164:6789/0 conn(0x7f8cc80c7f10 legacy=0x7f8cc8158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:29.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.155+0000 7f8ccc86f640 1 -- 172.21.15.71:0/3253395724 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc80b1530 con 0x7f8cc80cb190 2023-08-25T10:04:29.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.156+0000 7f8ccc86f640 1 --2- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc80ca910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009870 tx=0x7f8cbc02ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d2db969751946a3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:29.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.157+0000 7f8cc77fe640 1 -- 172.21.15.71:0/3253395724 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc02f830 con 0x7f8cc80cb190 2023-08-25T10:04:29.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.157+0000 7f8cc77fe640 1 -- 172.21.15.71:0/3253395724 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc02f990 con 0x7f8cc80cb190 2023-08-25T10:04:29.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.157+0000 7f8cc77fe640 1 -- 172.21.15.71:0/3253395724 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc02fd20 con 0x7f8cc80cb190 2023-08-25T10:04:29.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 -- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 msgr2=0x7f8cc80ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:29.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 --2- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc80ca910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc009870 tx=0x7f8cbc02ba10 comp rx=0 tx=0).stop 2023-08-25T10:04:29.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 -- 172.21.15.71:0/3253395724 shutdown_connections 2023-08-25T10:04:29.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 --2- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80cb570 0x7f8cc815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 --2- 172.21.15.71:0/3253395724 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc80ca910 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 -- 172.21.15.71:0/3253395724 >> 172.21.15.71:0/3253395724 conn(0x7f8cc806ae20 msgr2=0x7f8cc806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:29.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 -- 172.21.15.71:0/3253395724 shutdown_connections 2023-08-25T10:04:29.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.158+0000 7f8ccd871640 1 -- 172.21.15.71:0/3253395724 wait complete. 2023-08-25T10:04:29.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.160+0000 7f8ccd871640 1 Processor -- start 2023-08-25T10:04:29.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.160+0000 7f8ccd871640 1 -- start start 2023-08-25T10:04:29.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd871640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80c7f10 0x7f8cc814c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:29.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd871640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:29.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd871640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 0x7f8cc81f1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:29.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd871640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc814b470 con 0x7f8cc80cb570 2023-08-25T10:04:29.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd871640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc814c600 con 0x7f8cc80cb190 2023-08-25T10:04:29.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd871640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc8149960 con 0x7f8cc80c7f10 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.161+0000 7f8ccd070640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 0x7f8cc81f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 0x7f8cc81f1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59160/0 (socket says 172.21.15.71:59160) 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 -- 172.21.15.71:0/512265284 learned_addr learned my addr 172.21.15.71:0/512265284 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccc86f640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80c7f10 0x7f8cc814c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8cc7fff640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 -- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80c7f10 msgr2=0x7f8cc814c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:29.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80c7f10 0x7f8cc814c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 -- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 msgr2=0x7f8cc814e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:29.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc814e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 -- 172.21.15.71:0/512265284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0094d0 con 0x7f8cc80cb570 2023-08-25T10:04:29.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.162+0000 7f8ccd070640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 0x7f8cc81f1670 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c7a0 tx=0x7f8cc000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:29.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.163+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0003b70 con 0x7f8cc80cb570 2023-08-25T10:04:29.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.163+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0003cf0 con 0x7f8cc80cb570 2023-08-25T10:04:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.163+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0003ea0 con 0x7f8cc80cb570 2023-08-25T10:04:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.163+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc8149c40 con 0x7f8cc80cb570 2023-08-25T10:04:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.163+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc814b8c0 con 0x7f8cc80cb570 2023-08-25T10:04:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.164+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc804e420 con 0x7f8cc80cb570 2023-08-25T10:04:29.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.165+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8cc000aa90 con 0x7f8cc80cb570 2023-08-25T10:04:29.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.166+0000 7f8cc57fa640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8cac06a050 0x7f8cac06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:29.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.167+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f8cc0088750 con 0x7f8cc80cb570 2023-08-25T10:04:29.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.167+0000 7f8ccc86f640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8cac06a050 0x7f8cac06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:29.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.168+0000 7f8ccc86f640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8cac06a050 0x7f8cac06c510 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc036000 tx=0x7f8cbc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:29.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.171+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8cc0053a10 con 0x7f8cc80cb570 2023-08-25T10:04:29.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.471+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cc80c9df0 con 0x7f8cc80cb570 2023-08-25T10:04:29.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.473+0000 7f8cc57fa640 1 -- 172.21.15.71:0/512265284 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f8cc0057030 con 0x7f8cc80cb570 2023-08-25T10:04:29.476 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:29.476 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:29.476 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:29.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8cac06a050 msgr2=0x7f8cac06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:29.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8cac06a050 0x7f8cac06c510 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc036000 tx=0x7f8cbc008030 comp rx=0 tx=0).stop 2023-08-25T10:04:29.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 msgr2=0x7f8cc81f1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:29.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 0x7f8cc81f1670 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c7a0 tx=0x7f8cc000cc60 comp rx=0 tx=0).stop 2023-08-25T10:04:29.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 shutdown_connections 2023-08-25T10:04:29.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cc80cb190 0x7f8cc814e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8cac06a050 0x7f8cac06c510 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8cc80cb570 0x7f8cc81f1670 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 --2- 172.21.15.71:0/512265284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cc80c7f10 0x7f8cc814c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:29.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.475+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 >> 172.21.15.71:0/512265284 conn(0x7f8cc806ae20 msgr2=0x7f8cc80c60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:29.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.476+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 shutdown_connections 2023-08-25T10:04:29.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:29.476+0000 7f8ccd871640 1 -- 172.21.15.71:0/512265284 wait complete. 2023-08-25T10:04:29.497 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:31.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:31.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.739+0000 7f7d3e86a640 1 Processor -- start 2023-08-25T10:04:31.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.740+0000 7f7d3e86a640 1 -- start start 2023-08-25T10:04:31.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.742+0000 7f7d3e86a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 0x7f7d38069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3e86a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d380c1e30 con 0x7f7d380c2070 2023-08-25T10:04:31.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3e86a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d38069f00 con 0x7f7d3814f4a0 2023-08-25T10:04:31.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3e86a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c4900 con 0x7f7d38155040 2023-08-25T10:04:31.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3e069640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7d380c2070 0x7f7d3815f9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35876/0 (socket says 172.21.15.71:35876) 2023-08-25T10:04:31.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3e069640 1 -- 172.21.15.71:0/2313787192 learned_addr learned my addr 172.21.15.71:0/2313787192 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:31.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3d067640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 0x7f7d38069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.743+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503760859 0 0) 0x7f7d380c1e30 con 0x7f7d380c2070 2023-08-25T10:04:31.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d24003520 con 0x7f7d380c2070 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518907292 0 0) 0x7f7d38069f00 con 0x7f7d3814f4a0 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d380c1e30 con 0x7f7d3814f4a0 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1784228842 0 0) 0x7f7d24003520 con 0x7f7d380c2070 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 msgr2=0x7f7d38069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 0x7f7d38069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 >> v1:172.21.15.164:6789/0 conn(0x7f7d3814f4a0 legacy=0x7f7d380c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:31.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3806a1e0 con 0x7f7d380c2070 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f7d30001670 con 0x7f7d380c2070 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 >> v1:172.21.15.71:6789/0 conn(0x7f7d380c2070 legacy=0x7f7d3815f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24003ba0 0x7f7d24003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 0x7f7d240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d24004d00 0x7f7d24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400dc90 con 0x7f7d24004d00 2023-08-25T10:04:31.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400de00 con 0x7f7d24003ba0 2023-08-25T10:04:31.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2400dfc0 con 0x7f7d38155040 2023-08-25T10:04:31.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.744+0000 7f7d3e069640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d24004d00 0x7f7d24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3d067640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 0x7f7d240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3d868640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24003ba0 0x7f7d24003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3e069640 1 -- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 msgr2=0x7f7d240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:31.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3e069640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d38155040 0x7f7d240047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:31.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3e069640 1 -- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24003ba0 msgr2=0x7f7d24003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:31.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3e069640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24003ba0 0x7f7d24003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:31.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3e069640 1 -- 172.21.15.71:0/2313787192 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24003520 con 0x7f7d24004d00 2023-08-25T10:04:31.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.745+0000 7f7d3d868640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24003ba0 0x7f7d24003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:04:31.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.746+0000 7f7d3e069640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d24004d00 0x7f7d24005180 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7d30007f90 tx=0x7f7d240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d839ff2c2e7b04c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:31.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30005ad0 con 0x7f7d24004d00 2023-08-25T10:04:31.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30005c30 con 0x7f7d24004d00 2023-08-25T10:04:31.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3c866640 1 -- 172.21.15.71:0/2313787192 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3002c790 con 0x7f7d24004d00 2023-08-25T10:04:31.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e86a640 1 -- 172.21.15.71:0/2313787192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d24004d00 msgr2=0x7f7d24005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:31.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/2313787192 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d24004d00 0x7f7d24005180 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7d30007f90 tx=0x7f7d240057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:31.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e069640 1 -- 172.21.15.71:0/2313787192 reap_dead start 2023-08-25T10:04:31.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e86a640 1 -- 172.21.15.71:0/2313787192 shutdown_connections 2023-08-25T10:04:31.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e86a640 1 -- 172.21.15.71:0/2313787192 >> 172.21.15.71:0/2313787192 conn(0x7f7d3806d0d0 msgr2=0x7f7d380ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:31.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e86a640 1 -- 172.21.15.71:0/2313787192 shutdown_connections 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.747+0000 7f7d3e86a640 1 -- 172.21.15.71:0/2313787192 wait complete. 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.749+0000 7f7d3e86a640 1 Processor -- start 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.749+0000 7f7d3e86a640 1 -- start start 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.750+0000 7f7d3e86a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24004d00 0x7f7d381f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.750+0000 7f7d3e86a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d380caaa0 0x7f7d38145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.750+0000 7f7d3e86a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 0x7f7d381ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.750+0000 7f7d3e86a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381eec00 con 0x7f7d381ee240 2023-08-25T10:04:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.750+0000 7f7d3e86a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381f20d0 con 0x7f7d24004d00 2023-08-25T10:04:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.750+0000 7f7d3e86a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c9a90 con 0x7f7d380caaa0 2023-08-25T10:04:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 0x7f7d381ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 0x7f7d381ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46508/0 (socket says 172.21.15.71:46508) 2023-08-25T10:04:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 -- 172.21.15.71:0/886189098 learned_addr learned my addr 172.21.15.71:0/886189098 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:31.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3d067640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d380caaa0 0x7f7d38145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3d868640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24004d00 0x7f7d381f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 -- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d380caaa0 msgr2=0x7f7d38145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:31.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d380caaa0 0x7f7d38145470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:31.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 -- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24004d00 msgr2=0x7f7d381f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:31.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24004d00 0x7f7d381f1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:31.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.751+0000 7f7d3e069640 1 -- 172.21.15.71:0/886189098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d380ca640 con 0x7f7d381ee240 2023-08-25T10:04:31.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.752+0000 7f7d3e069640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 0x7f7d381ee6c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7d30007f90 tx=0x7f7d3002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:31.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.752+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3003c040 con 0x7f7d381ee240 2023-08-25T10:04:31.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.752+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3002cba0 con 0x7f7d381ee240 2023-08-25T10:04:31.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.752+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d380c9d30 con 0x7f7d381ee240 2023-08-25T10:04:31.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.752+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30005a80 con 0x7f7d381ee240 2023-08-25T10:04:31.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.752+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d381edd20 con 0x7f7d381ee240 2023-08-25T10:04:31.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.754+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3804e420 con 0x7f7d381ee240 2023-08-25T10:04:31.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.754+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7d30032070 con 0x7f7d381ee240 2023-08-25T10:04:31.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.756+0000 7f7d2e7fc640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7d1c06a050 0x7f7d1c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:31.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.756+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f7d300a9c50 con 0x7f7d381ee240 2023-08-25T10:04:31.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.756+0000 7f7d3d868640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7d1c06a050 0x7f7d1c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:31.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.757+0000 7f7d3d868640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7d1c06a050 0x7f7d1c06c510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7d340012e0 tx=0x7f7d34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:31.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:31.760+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7d30075040 con 0x7f7d381ee240 2023-08-25T10:04:32.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.003+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d380c63f0 con 0x7f7d381ee240 2023-08-25T10:04:32.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.004+0000 7f7d2e7fc640 1 -- 172.21.15.71:0/886189098 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f7d30046090 con 0x7f7d381ee240 2023-08-25T10:04:32.007 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:32.007 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:32.007 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:32.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7d1c06a050 msgr2=0x7f7d1c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:32.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7d1c06a050 0x7f7d1c06c510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7d340012e0 tx=0x7f7d34009040 comp rx=0 tx=0).stop 2023-08-25T10:04:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 msgr2=0x7f7d381ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 0x7f7d381ee6c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7d30007f90 tx=0x7f7d3002e070 comp rx=0 tx=0).stop 2023-08-25T10:04:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 shutdown_connections 2023-08-25T10:04:32.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7d1c06a050 0x7f7d1c06c510 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7d381ee240 0x7f7d381ee6c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d380caaa0 0x7f7d38145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 --2- 172.21.15.71:0/886189098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d24004d00 0x7f7d381f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 >> 172.21.15.71:0/886189098 conn(0x7f7d3806d0d0 msgr2=0x7f7d38155a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 shutdown_connections 2023-08-25T10:04:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:32.008+0000 7f7d3e86a640 1 -- 172.21.15.71:0/886189098 wait complete. 2023-08-25T10:04:32.032 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:34.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:34.273 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.270+0000 7fbf54a9b640 1 Processor -- start 2023-08-25T10:04:34.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.270+0000 7fbf54a9b640 1 -- start start 2023-08-25T10:04:34.274 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf54a9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf500c3660 0x7fbf500c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:34.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf54a9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fbf500c3a40 0x7fbf5006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:34.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf54a9b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf500c1dc0 con 0x7fbf501509d0 2023-08-25T10:04:34.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf54a9b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf50157a80 con 0x7fbf500c3a40 2023-08-25T10:04:34.275 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf54a9b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5006cf70 con 0x7fbf500c3660 2023-08-25T10:04:34.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf4ed9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fbf501509d0 0x7fbf50150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35890/0 (socket says 172.21.15.71:35890) 2023-08-25T10:04:34.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf4ed9d640 1 -- 172.21.15.71:0/1887367674 learned_addr learned my addr 172.21.15.71:0/1887367674 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:34.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf4dd9b640 1 --2- 172.21.15.71:0/1887367674 >> v2:172.21.15.71:3300/0 conn(0x7fbf500c3a40 0x7fbf5006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:34.276 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.272+0000 7fbf4e59c640 1 --2- 172.21.15.71:0/1887367674 >> v2:172.21.15.164:3300/0 conn(0x7fbf500c3660 0x7fbf500c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.273+0000 7fbf4d59a640 1 -- 172.21.15.71:0/1887367674 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701188943 0 0) 0x7fbf500c1dc0 con 0x7fbf501509d0 2023-08-25T10:04:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.273+0000 7fbf4d59a640 1 -- 172.21.15.71:0/1887367674 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf38003520 con 0x7fbf501509d0 2023-08-25T10:04:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.273+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/1887367674 >> v2:172.21.15.164:3300/0 conn(0x7fbf500c3660 msgr2=0x7fbf500c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.273+0000 7fbf4dd9b640 1 --2- 172.21.15.71:0/1887367674 >> v2:172.21.15.164:3300/0 conn(0x7fbf500c3660 0x7fbf500c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.273+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/1887367674 >> v1:172.21.15.71:6789/0 conn(0x7fbf501509d0 legacy=0x7fbf50150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:34.277 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.273+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/1887367674 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50156e00 con 0x7fbf500c3a40 2023-08-25T10:04:34.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.274+0000 7fbf4dd9b640 1 --2- 172.21.15.71:0/1887367674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5006ca30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbf50066440 tx=0x7fbf4402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0e84f7203f2e94 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:34.278 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.274+0000 7fbf4d59a640 1 -- 172.21.15.71:0/1887367674 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44030610 con 0x7fbf500c3a40 2023-08-25T10:04:34.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.274+0000 7fbf4d59a640 1 -- 172.21.15.71:0/1887367674 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf44030770 con 0x7fbf500c3a40 2023-08-25T10:04:34.279 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.274+0000 7fbf4d59a640 1 -- 172.21.15.71:0/1887367674 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44030b00 con 0x7fbf500c3a40 2023-08-25T10:04:34.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 -- 172.21.15.71:0/1887367674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 msgr2=0x7fbf5006ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:34.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/1887367674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5006ca30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fbf50066440 tx=0x7fbf4402b960 comp rx=0 tx=0).stop 2023-08-25T10:04:34.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 -- 172.21.15.71:0/1887367674 shutdown_connections 2023-08-25T10:04:34.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/1887367674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5006ca30 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/1887367674 >> v2:172.21.15.164:3300/0 conn(0x7fbf500c3660 0x7fbf500c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.280 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 -- 172.21.15.71:0/1887367674 >> 172.21.15.71:0/1887367674 conn(0x7fbf500654f0 msgr2=0x7fbf50065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:34.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 -- 172.21.15.71:0/1887367674 shutdown_connections 2023-08-25T10:04:34.281 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.275+0000 7fbf54a9b640 1 -- 172.21.15.71:0/1887367674 wait complete. 2023-08-25T10:04:34.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.277+0000 7fbf54a9b640 1 Processor -- start 2023-08-25T10:04:34.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.277+0000 7fbf54a9b640 1 -- start start 2023-08-25T10:04:34.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.278+0000 7fbf54a9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500c3660 0x7fbf50153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:34.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf54a9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5015a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:34.283 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf54a9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf501509d0 0x7fbf5006f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf54a9b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501ee480 con 0x7fbf500c3a40 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf54a9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501ee740 con 0x7fbf501509d0 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf54a9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf501eea20 con 0x7fbf500c3660 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5015a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46522/0 (socket says 172.21.15.71:46522) 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/4036471477 learned_addr learned my addr 172.21.15.71:0/4036471477 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4ed9d640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf501509d0 0x7fbf5006f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4e59c640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500c3660 0x7fbf50153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500c3660 msgr2=0x7fbf50153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:34.285 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500c3660 0x7fbf50153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.286 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf501509d0 msgr2=0x7fbf5006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf501509d0 0x7fbf5006f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.279+0000 7fbf4dd9b640 1 -- 172.21.15.71:0/4036471477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf44009400 con 0x7fbf500c3a40 2023-08-25T10:04:34.287 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.280+0000 7fbf4dd9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5015a940 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009890 tx=0x7fbf44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.280+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4403e040 con 0x7fbf500c3a40 2023-08-25T10:04:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.280+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf44039cb0 con 0x7fbf500c3a40 2023-08-25T10:04:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.280+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf44044630 con 0x7fbf500c3a40 2023-08-25T10:04:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.280+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf501eed00 con 0x7fbf500c3a40 2023-08-25T10:04:34.288 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.280+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf501ef2c0 con 0x7fbf500c3a40 2023-08-25T10:04:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.282+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf500c29c0 con 0x7fbf500c3a40 2023-08-25T10:04:34.289 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.283+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fbf44044790 con 0x7fbf500c3a40 2023-08-25T10:04:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.284+0000 7fbf3effd640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf3006a050 0x7fbf3006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.284+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fbf440aaf80 con 0x7fbf500c3a40 2023-08-25T10:04:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.284+0000 7fbf4e59c640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf3006a050 0x7fbf3006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:34.290 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.285+0000 7fbf4e59c640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf3006a050 0x7fbf3006c510 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e90 tx=0x7fbf4000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:34.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.288+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fbf44076080 con 0x7fbf500c3a40 2023-08-25T10:04:34.545 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.542+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf50069160 con 0x7fbf500c3a40 2023-08-25T10:04:34.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.544+0000 7fbf3effd640 1 -- 172.21.15.71:0/4036471477 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fbf44034020 con 0x7fbf500c3a40 2023-08-25T10:04:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:34.547 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:34.547 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:34.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf3006a050 msgr2=0x7fbf3006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:34.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf3006a050 0x7fbf3006c510 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009e90 tx=0x7fbf4000b040 comp rx=0 tx=0).stop 2023-08-25T10:04:34.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 msgr2=0x7fbf5015a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:34.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5015a940 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009890 tx=0x7fbf44004730 comp rx=0 tx=0).stop 2023-08-25T10:04:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 shutdown_connections 2023-08-25T10:04:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf501509d0 0x7fbf5006f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fbf3006a050 0x7fbf3006c510 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf500c3a40 0x7fbf5015a940 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 --2- 172.21.15.71:0/4036471477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf500c3660 0x7fbf50153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:34.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.546+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 >> 172.21.15.71:0/4036471477 conn(0x7fbf500654f0 msgr2=0x7fbf500c87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.547+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 shutdown_connections 2023-08-25T10:04:34.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:34.547+0000 7fbf54a9b640 1 -- 172.21.15.71:0/4036471477 wait complete. 2023-08-25T10:04:34.569 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:36.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:36.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.792+0000 7ff9c6107640 1 Processor -- start 2023-08-25T10:04:36.795 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.792+0000 7ff9c6107640 1 -- start start 2023-08-25T10:04:36.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.794+0000 7ff9c6107640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff9c00c3660 0x7ff9c00c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:36.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.794+0000 7ff9c6107640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c006ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:36.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.794+0000 7ff9c6107640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c00c1dc0 con 0x7ff9c01509d0 2023-08-25T10:04:36.797 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.794+0000 7ff9c6107640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c0157a80 con 0x7ff9c00c3660 2023-08-25T10:04:36.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.794+0000 7ff9c6107640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c006cf70 con 0x7ff9c00c3a40 2023-08-25T10:04:36.798 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9beffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c006ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:36.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9bf7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff9c00c3660 0x7ff9c00c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:36.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9beffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c006ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:55614/0 (socket says 172.21.15.71:55614) 2023-08-25T10:04:36.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9bf7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff9c00c3660 0x7ff9c00c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:35522/0 (socket says 172.21.15.71:35522) 2023-08-25T10:04:36.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9beffd640 1 -- 172.21.15.71:0/4195693072 learned_addr learned my addr 172.21.15.71:0/4195693072 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:36.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9bf7fe640 1 -- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 msgr2=0x7ff9c006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:36.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9bf7fe640 1 --2- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c006ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:36.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9bf7fe640 1 -- 172.21.15.71:0/4195693072 >> v1:172.21.15.164:6789/0 conn(0x7ff9c01509d0 legacy=0x7ff9c0150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:36.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.795+0000 7ff9bf7fe640 1 -- 172.21.15.71:0/4195693072 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0156e00 con 0x7ff9c00c3660 2023-08-25T10:04:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.797+0000 7ff9bf7fe640 1 --2- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c00c1710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff9b00097a0 tx=0x7ff9b002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a9fe24d23058047 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.797+0000 7ff9be7fc640 1 -- 172.21.15.71:0/4195693072 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0030610 con 0x7ff9c00c3660 2023-08-25T10:04:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.797+0000 7ff9be7fc640 1 -- 172.21.15.71:0/4195693072 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b0030770 con 0x7ff9c00c3660 2023-08-25T10:04:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.797+0000 7ff9be7fc640 1 -- 172.21.15.71:0/4195693072 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0030b00 con 0x7ff9c00c3660 2023-08-25T10:04:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 -- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 msgr2=0x7ff9c00c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:36.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 --2- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c00c1710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff9b00097a0 tx=0x7ff9b002b960 comp rx=0 tx=0).stop 2023-08-25T10:04:36.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 -- 172.21.15.71:0/4195693072 shutdown_connections 2023-08-25T10:04:36.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 --2- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c006ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:36.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 --2- 172.21.15.71:0/4195693072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c00c1710 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:36.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 -- 172.21.15.71:0/4195693072 >> 172.21.15.71:0/4195693072 conn(0x7ff9c00654f0 msgr2=0x7ff9c0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 -- 172.21.15.71:0/4195693072 shutdown_connections 2023-08-25T10:04:36.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.798+0000 7ff9c6107640 1 -- 172.21.15.71:0/4195693072 wait complete. 2023-08-25T10:04:36.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.800+0000 7ff9c6107640 1 Processor -- start 2023-08-25T10:04:36.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.800+0000 7ff9c6107640 1 -- start start 2023-08-25T10:04:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9c6107640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c0157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9c6107640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c00c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9c6107640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 0x7ff9c01ecdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:36.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9c6107640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01591c0 con 0x7ff9c01509d0 2023-08-25T10:04:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9c6107640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01ec5e0 con 0x7ff9c00c3660 2023-08-25T10:04:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9c6107640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01f0040 con 0x7ff9c00c3a40 2023-08-25T10:04:36.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.801+0000 7ff9bffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 0x7ff9c01ecdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 0x7ff9c01ecdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46524/0 (socket says 172.21.15.71:46524) 2023-08-25T10:04:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 -- 172.21.15.71:0/2552005573 learned_addr learned my addr 172.21.15.71:0/2552005573 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bf7fe640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c0157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9beffd640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c00c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 -- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 msgr2=0x7ff9c00c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:36.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c00c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 -- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 msgr2=0x7ff9c0157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c0157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 -- 172.21.15.71:0/2552005573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b0009400 con 0x7ff9c01509d0 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.802+0000 7ff9bffff640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 0x7ff9c01ecdc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800c7a0 tx=0x7ff9b800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.803+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8003b70 con 0x7ff9c01509d0 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.803+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b8003cf0 con 0x7ff9c01509d0 2023-08-25T10:04:36.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.803+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c01f0320 con 0x7ff9c01509d0 2023-08-25T10:04:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.803+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8003ea0 con 0x7ff9c01509d0 2023-08-25T10:04:36.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.803+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c01f08e0 con 0x7ff9c01509d0 2023-08-25T10:04:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.805+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c00c29c0 con 0x7ff9c01509d0 2023-08-25T10:04:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.806+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff9b800aa90 con 0x7ff9c01509d0 2023-08-25T10:04:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.807+0000 7ff9a7fff640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff9ac06a080 0x7ff9ac06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:36.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.807+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff9b8088780 con 0x7ff9c01509d0 2023-08-25T10:04:36.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.807+0000 7ff9bf7fe640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff9ac06a080 0x7ff9ac06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:36.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.808+0000 7ff9bf7fe640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff9ac06a080 0x7ff9ac06c540 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0036000 tx=0x7ff9b003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:36.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:36.813+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff9b80539b0 con 0x7ff9c01509d0 2023-08-25T10:04:37.071 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.069+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9c0069160 con 0x7ff9c01509d0 2023-08-25T10:04:37.072 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.069+0000 7ff9a7fff640 1 -- 172.21.15.71:0/2552005573 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7ff9b8056fd0 con 0x7ff9c01509d0 2023-08-25T10:04:37.072 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:37.073 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:37.073 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:37.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff9ac06a080 msgr2=0x7ff9ac06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:37.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff9ac06a080 0x7ff9ac06c540 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0036000 tx=0x7ff9b003b040 comp rx=0 tx=0).stop 2023-08-25T10:04:37.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 msgr2=0x7ff9c01ecdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:37.075 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 0x7ff9c01ecdc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800c7a0 tx=0x7ff9b800cc60 comp rx=0 tx=0).stop 2023-08-25T10:04:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 shutdown_connections 2023-08-25T10:04:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff9ac06a080 0x7ff9ac06c540 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff9c01509d0 0x7ff9c01ecdc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff9c00c3a40 0x7ff9c00c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:37.076 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 --2- 172.21.15.71:0/2552005573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff9c00c3660 0x7ff9c0157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:37.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 >> 172.21.15.71:0/2552005573 conn(0x7ff9c00654f0 msgr2=0x7ff9c0069850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:37.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 shutdown_connections 2023-08-25T10:04:37.077 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:37.072+0000 7ff9c6107640 1 -- 172.21.15.71:0/2552005573 wait complete. 2023-08-25T10:04:37.094 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:39.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:39.328 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.325+0000 7fe92f59e640 1 Processor -- start 2023-08-25T10:04:39.329 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.325+0000 7fe92f59e640 1 -- start start 2023-08-25T10:04:39.333 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92f59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe930150d50 0x7fe930070d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 0x7fe9300c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92f59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93006ae90 con 0x7fe93006b2a0 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92f59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe930070880 con 0x7fe930150d50 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93006ebc0 con 0x7fe9300c5260 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92ed9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe93006b2a0 0x7fe930159f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35898/0 (socket says 172.21.15.71:35898) 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.327+0000 7fe92ed9d640 1 -- 172.21.15.71:0/3138228858 learned_addr learned my addr 172.21.15.71:0/3138228858 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:39.334 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92dd9b640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 0x7fe9300c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92e59c640 1 --2- 172.21.15.71:0/3138228858 >> v2:172.21.15.164:3300/0 conn(0x7fe930150d50 0x7fe930070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212451087 0 0) 0x7fe93006ae90 con 0x7fe93006b2a0 2023-08-25T10:04:39.335 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe918003520 con 0x7fe93006b2a0 2023-08-25T10:04:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 568532212 0 0) 0x7fe918003520 con 0x7fe93006b2a0 2023-08-25T10:04:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 msgr2=0x7fe9300c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:39.336 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 0x7fe9300c3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 >> v2:172.21.15.164:3300/0 conn(0x7fe930150d50 msgr2=0x7fe930070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 --2- 172.21.15.71:0/3138228858 >> v2:172.21.15.164:3300/0 conn(0x7fe930150d50 0x7fe930070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930071270 con 0x7fe93006b2a0 2023-08-25T10:04:39.337 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe9280019f0 con 0x7fe93006b2a0 2023-08-25T10:04:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 >> v1:172.21.15.71:6789/0 conn(0x7fe93006b2a0 legacy=0x7fe930159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.328+0000 7fe92d59a640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9180039e0 0x7fe918003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92d59a640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180047c0 0x7fe918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92d59a640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 0x7fe918005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930070880 con 0x7fe9180039e0 2023-08-25T10:04:39.338 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93006ebc0 con 0x7fe9180047c0 2023-08-25T10:04:39.339 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91800b980 con 0x7fe9300c5260 2023-08-25T10:04:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92e59c640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9180039e0 0x7fe918003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92ed9d640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 0x7fe918005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92e59c640 1 -- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 msgr2=0x7fe918005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92e59c640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9300c5260 0x7fe918005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92e59c640 1 -- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180047c0 msgr2=0x7fe918004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:39.340 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92e59c640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180047c0 0x7fe918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.329+0000 7fe92e59c640 1 -- 172.21.15.71:0/3138228858 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918003520 con 0x7fe9180039e0 2023-08-25T10:04:39.341 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.331+0000 7fe92e59c640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9180039e0 0x7fe918003e00 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe920009af0 tx=0x7fe9180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd18d696ff13264 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.331+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92002c710 con 0x7fe9180039e0 2023-08-25T10:04:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.331+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92002c870 con 0x7fe9180039e0 2023-08-25T10:04:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.331+0000 7fe92d59a640 1 -- 172.21.15.71:0/3138228858 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92002cc00 con 0x7fe9180039e0 2023-08-25T10:04:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92f59e640 1 -- 172.21.15.71:0/3138228858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9180039e0 msgr2=0x7fe918003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:39.344 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92f59e640 1 --2- 172.21.15.71:0/3138228858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe9180039e0 0x7fe918003e00 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fe920009af0 tx=0x7fe9180057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92ed9d640 1 -- 172.21.15.71:0/3138228858 reap_dead start 2023-08-25T10:04:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92f59e640 1 -- 172.21.15.71:0/3138228858 shutdown_connections 2023-08-25T10:04:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92f59e640 1 -- 172.21.15.71:0/3138228858 >> 172.21.15.71:0/3138228858 conn(0x7fe93006d0d0 msgr2=0x7fe9300ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92f59e640 1 -- 172.21.15.71:0/3138228858 shutdown_connections 2023-08-25T10:04:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.332+0000 7fe92f59e640 1 -- 172.21.15.71:0/3138228858 wait complete. 2023-08-25T10:04:39.345 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.334+0000 7fe92f59e640 1 Processor -- start 2023-08-25T10:04:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.335+0000 7fe92f59e640 1 -- start start 2023-08-25T10:04:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180039e0 0x7fe9301e3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 0x7fe93014f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9301e2480 0x7fe9301e28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92f59e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9301e21a0 con 0x7fe93014e290 2023-08-25T10:04:39.346 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92f59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93014e7f0 con 0x7fe9180039e0 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92f59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9301ee1a0 con 0x7fe9301e2480 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 0x7fe93014f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92dd9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 0x7fe93014f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:46538/0 (socket says 172.21.15.71:46538) 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92dd9b640 1 -- 172.21.15.71:0/700076669 learned_addr learned my addr 172.21.15.71:0/700076669 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92e59c640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180039e0 0x7fe9301e3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92ed9d640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9301e2480 0x7fe9301e28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92dd9b640 1 -- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9301e2480 msgr2=0x7fe9301e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:39.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92dd9b640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9301e2480 0x7fe9301e28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.336+0000 7fe92dd9b640 1 -- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180039e0 msgr2=0x7fe9301e3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe92dd9b640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180039e0 0x7fe9301e3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe92dd9b640 1 -- 172.21.15.71:0/700076669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9301ee480 con 0x7fe93014e290 2023-08-25T10:04:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe92ed9d640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9301e2480 0x7fe9301e28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:04:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe92dd9b640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 0x7fe93014f320 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe92400e7c0 tx=0x7fe92400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:39.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92400eed0 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92400c8c0 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92400cc70 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9301ee760 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.337+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93014ead0 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.339+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93004e420 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.340+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe924013470 con 0x7fe93014e290 2023-08-25T10:04:39.349 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.340+0000 7fe91effd640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91006a080 0x7fe91006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.341+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe924088ad0 con 0x7fe93014e290 2023-08-25T10:04:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.341+0000 7fe92e59c640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91006a080 0x7fe91006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:39.350 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.342+0000 7fe92e59c640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91006a080 0x7fe91006c540 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe920030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.345+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe924053cf0 con 0x7fe93014e290 2023-08-25T10:04:39.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.616+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9300700a0 con 0x7fe93014e290 2023-08-25T10:04:39.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.617+0000 7fe91effd640 1 -- 172.21.15.71:0/700076669 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fe924057310 con 0x7fe93014e290 2023-08-25T10:04:39.620 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:39.620 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:39.620 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:39.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.619+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91006a080 msgr2=0x7fe91006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:39.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.619+0000 7fe92f59e640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91006a080 0x7fe91006c540 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe920036000 tx=0x7fe920030000 comp rx=0 tx=0).stop 2023-08-25T10:04:39.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 msgr2=0x7fe93014f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:39.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 0x7fe93014f320 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe92400e7c0 tx=0x7fe92400ec80 comp rx=0 tx=0).stop 2023-08-25T10:04:39.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 shutdown_connections 2023-08-25T10:04:39.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe91006a080 0x7fe91006c540 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9301e2480 0x7fe9301e28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe93014e290 0x7fe93014f320 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 --2- 172.21.15.71:0/700076669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9180039e0 0x7fe9301e3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:39.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 >> 172.21.15.71:0/700076669 conn(0x7fe93006d0d0 msgr2=0x7fe9300c86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:39.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 shutdown_connections 2023-08-25T10:04:39.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:39.620+0000 7fe92f59e640 1 -- 172.21.15.71:0/700076669 wait complete. 2023-08-25T10:04:39.641 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:04:39.642 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:04:41.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:41.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.895+0000 7fd24ccc0640 1 Processor -- start 2023-08-25T10:04:41.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.896+0000 7fd24ccc0640 1 -- start start 2023-08-25T10:04:41.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.897+0000 7fd24ccc0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fd2480cb190 0x7fd2480ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:41.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.897+0000 7fd24ccc0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2480b1af0 con 0x7fd2480cb570 2023-08-25T10:04:41.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.897+0000 7fd24ccc0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2480b0fb0 con 0x7fd24815cd00 2023-08-25T10:04:41.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.897+0000 7fd24ccc0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2480c98f0 con 0x7fd2480cb190 2023-08-25T10:04:41.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.897+0000 7fd246ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd2480cb570 0x7fd2480c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46624/0 (socket says 172.21.15.71:46624) 2023-08-25T10:04:41.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.898+0000 7fd246ffd640 1 -- 172.21.15.71:0/3399983620 learned_addr learned my addr 172.21.15.71:0/3399983620 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:41.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.898+0000 7fd2477fe640 1 --2- 172.21.15.71:0/3399983620 >> v2:172.21.15.71:3300/0 conn(0x7fd2480cb190 0x7fd2480ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:41.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.898+0000 7fd2477fe640 1 -- 172.21.15.71:0/3399983620 >> v1:172.21.15.164:6789/0 conn(0x7fd24815cd00 legacy=0x7fd248158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:41.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.898+0000 7fd2477fe640 1 -- 172.21.15.71:0/3399983620 >> v1:172.21.15.71:6789/0 conn(0x7fd2480cb570 legacy=0x7fd2480c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:41.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.898+0000 7fd2477fe640 1 -- 172.21.15.71:0/3399983620 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24815d130 con 0x7fd2480cb190 2023-08-25T10:04:41.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.898+0000 7fd2467fc640 1 -- 172.21.15.71:0/3399983620 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762932433 0 0) 0x7fd2480b1af0 con 0x7fd2480cb570 2023-08-25T10:04:41.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.900+0000 7fd2477fe640 1 --2- 172.21.15.71:0/3399983620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb190 0x7fd2480ca910 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd2380097a0 tx=0x7fd23802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2a309b67136501 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:41.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.900+0000 7fd2467fc640 1 -- 172.21.15.71:0/3399983620 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23802f810 con 0x7fd2480cb190 2023-08-25T10:04:41.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.900+0000 7fd2467fc640 1 -- 172.21.15.71:0/3399983620 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23802f970 con 0x7fd2480cb190 2023-08-25T10:04:41.903 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.900+0000 7fd2467fc640 1 -- 172.21.15.71:0/3399983620 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23802fd00 con 0x7fd2480cb190 2023-08-25T10:04:41.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.901+0000 7fd24ccc0640 1 -- 172.21.15.71:0/3399983620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb190 msgr2=0x7fd2480ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:41.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.901+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/3399983620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb190 0x7fd2480ca910 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd2380097a0 tx=0x7fd23802b9f0 comp rx=0 tx=0).stop 2023-08-25T10:04:41.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.901+0000 7fd24ccc0640 1 -- 172.21.15.71:0/3399983620 shutdown_connections 2023-08-25T10:04:41.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.901+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/3399983620 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb190 0x7fd2480ca910 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:41.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.901+0000 7fd24ccc0640 1 -- 172.21.15.71:0/3399983620 >> 172.21.15.71:0/3399983620 conn(0x7fd24806ae20 msgr2=0x7fd24806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:41.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.902+0000 7fd24ccc0640 1 -- 172.21.15.71:0/3399983620 shutdown_connections 2023-08-25T10:04:41.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.902+0000 7fd24ccc0640 1 -- 172.21.15.71:0/3399983620 wait complete. 2023-08-25T10:04:41.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.904+0000 7fd24ccc0640 1 Processor -- start 2023-08-25T10:04:41.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.904+0000 7fd24ccc0640 1 -- start start 2023-08-25T10:04:41.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.905+0000 7fd24ccc0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2480cb190 0x7fd2480c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:41.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.905+0000 7fd24ccc0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 0x7fd24814a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:41.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.905+0000 7fd24ccc0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24815cd00 0x7fd24814ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:41.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.905+0000 7fd24ccc0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24814d3d0 con 0x7fd2480cb570 2023-08-25T10:04:41.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.905+0000 7fd24ccc0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2480c87c0 con 0x7fd2480cb190 2023-08-25T10:04:41.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.905+0000 7fd24ccc0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2480c5fe0 con 0x7fd24815cd00 2023-08-25T10:04:41.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 0x7fd24814a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:41.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 0x7fd24814a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37370/0 (socket says 172.21.15.71:37370) 2023-08-25T10:04:41.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 -- 172.21.15.71:0/818447799 learned_addr learned my addr 172.21.15.71:0/818447799 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:41.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd2477fe640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2480cb190 0x7fd2480c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:41.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd247fff640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24815cd00 0x7fd24814ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 -- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24815cd00 msgr2=0x7fd24814ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24815cd00 0x7fd24814ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 -- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2480cb190 msgr2=0x7fd2480c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2480cb190 0x7fd2480c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.906+0000 7fd246ffd640 1 -- 172.21.15.71:0/818447799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd238009400 con 0x7fd2480cb570 2023-08-25T10:04:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.907+0000 7fd246ffd640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 0x7fd24814a5e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00c490 tx=0x7fd23c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:41.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.907+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0074e0 con 0x7fd2480cb570 2023-08-25T10:04:41.911 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.907+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c007640 con 0x7fd2480cb570 2023-08-25T10:04:41.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.907+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c004a20 con 0x7fd2480cb570 2023-08-25T10:04:41.912 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.907+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2480c7f80 con 0x7fd2480cb570 2023-08-25T10:04:41.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.908+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2480c84e0 con 0x7fd2480cb570 2023-08-25T10:04:41.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.910+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24814d510 con 0x7fd2480cb570 2023-08-25T10:04:41.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.910+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd23c004b80 con 0x7fd2480cb570 2023-08-25T10:04:41.913 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.911+0000 7fd22ffff640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23406a0a0 0x7fd23406c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:41.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.911+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fd23c087f90 con 0x7fd2480cb570 2023-08-25T10:04:41.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.912+0000 7fd2477fe640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23406a0a0 0x7fd23406c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:41.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.913+0000 7fd2477fe640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23406a0a0 0x7fd23406c560 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd238036000 tx=0x7fd238008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:41.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:41.918+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd24814d510 con 0x7fd2480cb570 2023-08-25T10:04:42.161 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.159+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd24814d510 con 0x7fd2480cb570 2023-08-25T10:04:42.162 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.160+0000 7fd22ffff640 1 -- 172.21.15.71:0/818447799 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fd24814d510 con 0x7fd2480cb570 2023-08-25T10:04:42.163 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:42.163 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:42.164 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:42.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23406a0a0 msgr2=0x7fd23406c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:42.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23406a0a0 0x7fd23406c560 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd238036000 tx=0x7fd238008030 comp rx=0 tx=0).stop 2023-08-25T10:04:42.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 msgr2=0x7fd24814a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:42.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 0x7fd24814a5e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00c490 tx=0x7fd23c00c950 comp rx=0 tx=0).stop 2023-08-25T10:04:42.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 shutdown_connections 2023-08-25T10:04:42.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd23406a0a0 0x7fd23406c560 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:42.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd24815cd00 0x7fd24814ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:42.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd2480cb570 0x7fd24814a5e0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:42.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 --2- 172.21.15.71:0/818447799 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd2480cb190 0x7fd2480c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:42.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 >> 172.21.15.71:0/818447799 conn(0x7fd24806ae20 msgr2=0x7fd2480c6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:42.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 shutdown_connections 2023-08-25T10:04:42.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:42.163+0000 7fd24ccc0640 1 -- 172.21.15.71:0/818447799 wait complete. 2023-08-25T10:04:42.184 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:44.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.407+0000 7fc50d7f5640 1 Processor -- start 2023-08-25T10:04:44.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.408+0000 7fc50d7f5640 1 -- start start 2023-08-25T10:04:44.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.409+0000 7fc50d7f5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5081542d0 0x7fc5081576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.409+0000 7fc50d7f5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc508069850 0x7fc5080c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.409+0000 7fc50d7f5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508157bf0 con 0x7fc5080c75c0 2023-08-25T10:04:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.409+0000 7fc50d7f5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5080c4b60 con 0x7fc508069850 2023-08-25T10:04:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.409+0000 7fc50d7f5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50806a130 con 0x7fc5081542d0 2023-08-25T10:04:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc508069850 0x7fc5080c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:44.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc508069850 0x7fc5080c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37398/0 (socket says 172.21.15.71:37398) 2023-08-25T10:04:44.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 -- 172.21.15.71:0/41345727 learned_addr learned my addr 172.21.15.71:0/41345727 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:44.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc50cff4640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc5080c75c0 0x7fc5080c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46634/0 (socket says 172.21.15.71:46634) 2023-08-25T10:04:44.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc507fff640 1 --2- 172.21.15.71:0/41345727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5081542d0 0x7fc5081576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:44.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 -- 172.21.15.71:0/41345727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5081542d0 msgr2=0x7fc5081576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:44.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 --2- 172.21.15.71:0/41345727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5081542d0 0x7fc5081576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 -- 172.21.15.71:0/41345727 >> v1:172.21.15.71:6789/0 conn(0x7fc5080c75c0 legacy=0x7fc5080c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.410+0000 7fc5077fe640 1 -- 172.21.15.71:0/41345727 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5081546b0 con 0x7fc508069850 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.411+0000 7fc5077fe640 1 --2- 172.21.15.71:0/41345727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc508069850 0x7fc5080c7080 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009870 tx=0x7fc4fc02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f5a81002041509 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc506ffd640 1 -- 172.21.15.71:0/41345727 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc02f830 con 0x7fc508069850 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc506ffd640 1 -- 172.21.15.71:0/41345727 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc02f990 con 0x7fc508069850 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc506ffd640 1 -- 172.21.15.71:0/41345727 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc02fd20 con 0x7fc508069850 2023-08-25T10:04:44.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 -- 172.21.15.71:0/41345727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc508069850 msgr2=0x7fc5080c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:44.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/41345727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc508069850 0x7fc5080c7080 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009870 tx=0x7fc4fc02ba10 comp rx=0 tx=0).stop 2023-08-25T10:04:44.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 -- 172.21.15.71:0/41345727 shutdown_connections 2023-08-25T10:04:44.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/41345727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc508069850 0x7fc5080c7080 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/41345727 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5081542d0 0x7fc5081576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.416 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 -- 172.21.15.71:0/41345727 >> 172.21.15.71:0/41345727 conn(0x7fc50806c280 msgr2=0x7fc50806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:44.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 -- 172.21.15.71:0/41345727 shutdown_connections 2023-08-25T10:04:44.417 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.412+0000 7fc50d7f5640 1 -- 172.21.15.71:0/41345727 wait complete. 2023-08-25T10:04:44.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.414+0000 7fc50d7f5640 1 Processor -- start 2023-08-25T10:04:44.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.414+0000 7fc50d7f5640 1 -- start start 2023-08-25T10:04:44.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50d7f5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc508069850 0x7fc5081ea100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:44.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50d7f5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5080c75c0 0x7fc508148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:44.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50d7f5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 0x7fc508148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:44.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50d7f5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50814e700 con 0x7fc5081542d0 2023-08-25T10:04:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50d7f5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5081ef430 con 0x7fc5080c75c0 2023-08-25T10:04:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50d7f5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5081ee0b0 con 0x7fc508069850 2023-08-25T10:04:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50cff4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 0x7fc508148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50cff4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 0x7fc508148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37412/0 (socket says 172.21.15.71:37412) 2023-08-25T10:04:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.415+0000 7fc50cff4640 1 -- 172.21.15.71:0/1750563530 learned_addr learned my addr 172.21.15.71:0/1750563530 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc507fff640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc508069850 0x7fc5081ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc5077fe640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5080c75c0 0x7fc508148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc50cff4640 1 -- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc508069850 msgr2=0x7fc5081ea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc50cff4640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc508069850 0x7fc5081ea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc50cff4640 1 -- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5080c75c0 msgr2=0x7fc508148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc50cff4640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5080c75c0 0x7fc508148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc50cff4640 1 -- 172.21.15.71:0/1750563530 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0094d0 con 0x7fc5081542d0 2023-08-25T10:04:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc50cff4640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 0x7fc508148d80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc50000c780 tx=0x7fc50000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50000ce90 con 0x7fc5081542d0 2023-08-25T10:04:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50000a640 con 0x7fc5081542d0 2023-08-25T10:04:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.416+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50000a9f0 con 0x7fc5081542d0 2023-08-25T10:04:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.417+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5081ee390 con 0x7fc5081542d0 2023-08-25T10:04:44.423 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.417+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5081ee950 con 0x7fc5081542d0 2023-08-25T10:04:44.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.418+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50804e420 con 0x7fc5081542d0 2023-08-25T10:04:44.424 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.419+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc50000ab50 con 0x7fc5081542d0 2023-08-25T10:04:44.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.420+0000 7fc504ff9640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4f406a050 0x7fc4f406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:44.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.420+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc500088100 con 0x7fc5081542d0 2023-08-25T10:04:44.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.420+0000 7fc507fff640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4f406a050 0x7fc4f406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:44.425 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.421+0000 7fc507fff640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4f406a050 0x7fc4f406c510 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006fd0 tx=0x7fc4f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:44.426 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.424+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc500053340 con 0x7fc5081542d0 2023-08-25T10:04:44.651 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.649+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc508157bf0 con 0x7fc5081542d0 2023-08-25T10:04:44.652 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.651+0000 7fc504ff9640 1 -- 172.21.15.71:0/1750563530 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fc508157bf0 con 0x7fc5081542d0 2023-08-25T10:04:44.653 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:44.653 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:44.654 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:44.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4f406a050 msgr2=0x7fc4f406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:44.655 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4f406a050 0x7fc4f406c510 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006fd0 tx=0x7fc4f8008040 comp rx=0 tx=0).stop 2023-08-25T10:04:44.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 msgr2=0x7fc508148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:44.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 0x7fc508148d80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc50000c780 tx=0x7fc50000cc40 comp rx=0 tx=0).stop 2023-08-25T10:04:44.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 shutdown_connections 2023-08-25T10:04:44.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5080c75c0 0x7fc508148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4f406a050 0x7fc4f406c510 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.656 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5081542d0 0x7fc508148d80 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 --2- 172.21.15.71:0/1750563530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc508069850 0x7fc5081ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:44.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.653+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 >> 172.21.15.71:0/1750563530 conn(0x7fc50806c280 msgr2=0x7fc508151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:44.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.654+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 shutdown_connections 2023-08-25T10:04:44.657 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:44.654+0000 7fc50d7f5640 1 -- 172.21.15.71:0/1750563530 wait complete. 2023-08-25T10:04:44.676 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:46.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:46.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.891+0000 7fe6bccd2640 1 Processor -- start 2023-08-25T10:04:46.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.892+0000 7fe6bccd2640 1 -- start start 2023-08-25T10:04:46.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.893+0000 7fe6bccd2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe6b80c3a40 0x7fe6b80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.893+0000 7fe6bccd2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b80c1dc0 con 0x7fe6b80c3660 2023-08-25T10:04:46.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.893+0000 7fe6bccd2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b8156e00 con 0x7fe6b81509d0 2023-08-25T10:04:46.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.893+0000 7fe6bccd2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b8157a80 con 0x7fe6b80c3a40 2023-08-25T10:04:46.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe6b80c3660 0x7fe6b806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46646/0 (socket says 172.21.15.71:46646) 2023-08-25T10:04:46.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b659c640 1 -- 172.21.15.71:0/3403380251 learned_addr learned my addr 172.21.15.71:0/3403380251 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:46.897 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b5d9b640 1 --2- 172.21.15.71:0/3403380251 >> v2:172.21.15.164:3300/0 conn(0x7fe6b80c3a40 0x7fe6b80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837440121 0 0) 0x7fe6b80c1dc0 con 0x7fe6b80c3660 2023-08-25T10:04:46.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a0003520 con 0x7fe6b80c3660 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291855456 0 0) 0x7fe6a0003520 con 0x7fe6b80c3660 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 >> v2:172.21.15.164:3300/0 conn(0x7fe6b80c3a40 msgr2=0x7fe6b80c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 --2- 172.21.15.71:0/3403380251 >> v2:172.21.15.164:3300/0 conn(0x7fe6b80c3a40 0x7fe6b80c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 >> v1:172.21.15.164:6789/0 conn(0x7fe6b81509d0 legacy=0x7fe6b8150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b81570e0 con 0x7fe6b80c3660 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.894+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe6a8001670 con 0x7fe6b80c3660 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 >> v1:172.21.15.71:6789/0 conn(0x7fe6b80c3660 legacy=0x7fe6b806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:46.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6a00047c0 0x7fe6a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6a0005180 0x7fe6a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8157a80 con 0x7fe6a00039e0 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a000c090 con 0x7fe6a0005180 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a000c250 con 0x7fe6a00047c0 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b659c640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b6d9d640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6a0005180 0x7fe6a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b659c640 1 -- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6a00047c0 msgr2=0x7fe6a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b659c640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6a00047c0 0x7fe6a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b659c640 1 -- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6a0005180 msgr2=0x7fe6a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b659c640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6a0005180 0x7fe6a000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b659c640 1 -- 172.21.15.71:0/3403380251 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a0003520 con 0x7fe6a00039e0 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.895+0000 7fe6b6d9d640 1 -- 172.21.15.71:0/3403380251 reap_dead start 2023-08-25T10:04:46.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.896+0000 7fe6b659c640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6a0003e00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800c240 tx=0x7fe6a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f9a967c3db8c5c server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:46.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8005ad0 con 0x7fe6a00039e0 2023-08-25T10:04:46.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a8005c30 con 0x7fe6a00039e0 2023-08-25T10:04:46.902 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6b559a640 1 -- 172.21.15.71:0/3403380251 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a802c8b0 con 0x7fe6a00039e0 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 -- 172.21.15.71:0/3403380251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 msgr2=0x7fe6a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6a0003e00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800c240 tx=0x7fe6a00057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 -- 172.21.15.71:0/3403380251 shutdown_connections 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/3403380251 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6a0003e00 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 -- 172.21.15.71:0/3403380251 >> 172.21.15.71:0/3403380251 conn(0x7fe6b80654f0 msgr2=0x7fe6b8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 -- 172.21.15.71:0/3403380251 shutdown_connections 2023-08-25T10:04:46.904 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.897+0000 7fe6bccd2640 1 -- 172.21.15.71:0/3403380251 wait complete. 2023-08-25T10:04:46.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.899+0000 7fe6bccd2640 1 Processor -- start 2023-08-25T10:04:46.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.899+0000 7fe6bccd2640 1 -- start start 2023-08-25T10:04:46.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6bccd2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6b81eeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6bccd2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6b814e7d0 0x7fe6b806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.905 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6bccd2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6b8159420 0x7fe6b8065b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6bccd2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b81eea60 con 0x7fe6a00039e0 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6bccd2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8157e20 con 0x7fe6b8159420 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6bccd2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8158100 con 0x7fe6b814e7d0 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6b659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6b81eeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6b659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6b81eeef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37434/0 (socket says 172.21.15.71:37434) 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.900+0000 7fe6b659c640 1 -- 172.21.15.71:0/2333556172 learned_addr learned my addr 172.21.15.71:0/2333556172 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:46.906 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b6d9d640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6b8159420 0x7fe6b8065b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b5d9b640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6b814e7d0 0x7fe6b806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b659c640 1 -- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6b814e7d0 msgr2=0x7fe6b806ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b659c640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6b814e7d0 0x7fe6b806ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b659c640 1 -- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6b8159420 msgr2=0x7fe6b8065b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b659c640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6b8159420 0x7fe6b8065b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b659c640 1 -- 172.21.15.71:0/2333556172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b81583e0 con 0x7fe6a00039e0 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6b659c640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6b81eeef0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800c240 tx=0x7fe6a8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:46.907 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a803c070 con 0x7fe6a00039e0 2023-08-25T10:04:46.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a802cb80 con 0x7fe6a00039e0 2023-08-25T10:04:46.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b81ee6f0 con 0x7fe6a00039e0 2023-08-25T10:04:46.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.901+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8007070 con 0x7fe6a00039e0 2023-08-25T10:04:46.908 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.902+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b8066080 con 0x7fe6a00039e0 2023-08-25T10:04:46.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.903+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b8069160 con 0x7fe6a00039e0 2023-08-25T10:04:46.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.904+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe6a8032070 con 0x7fe6a00039e0 2023-08-25T10:04:46.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.905+0000 7fe6a6ffd640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe69806a080 0x7fe69806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:46.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.905+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe6a80a9910 con 0x7fe6a00039e0 2023-08-25T10:04:46.909 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.905+0000 7fe6b5d9b640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe69806a080 0x7fe69806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:46.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.905+0000 7fe6b5d9b640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe69806a080 0x7fe69806c540 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe6b806b260 tx=0x7fe6ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:46.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:46.909+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe6a8074b40 con 0x7fe6a00039e0 2023-08-25T10:04:47.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.139+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6b8154680 con 0x7fe6a00039e0 2023-08-25T10:04:47.141 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.140+0000 7fe6a6ffd640 1 -- 172.21.15.71:0/2333556172 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fe6a8078160 con 0x7fe6a00039e0 2023-08-25T10:04:47.142 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:47.142 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:47.142 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:47.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe69806a080 msgr2=0x7fe69806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:47.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe69806a080 0x7fe69806c540 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe6b806b260 tx=0x7fe6ac008040 comp rx=0 tx=0).stop 2023-08-25T10:04:47.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 msgr2=0x7fe6b81eeef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6b81eeef0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800c240 tx=0x7fe6a8001a00 comp rx=0 tx=0).stop 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 shutdown_connections 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6b8159420 0x7fe6b8065b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe69806a080 0x7fe69806c540 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6b814e7d0 0x7fe6b806ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 --2- 172.21.15.71:0/2333556172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6a00039e0 0x7fe6b81eeef0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:47.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.142+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 >> 172.21.15.71:0/2333556172 conn(0x7fe6b80654f0 msgr2=0x7fe6b80c9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:47.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.143+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 shutdown_connections 2023-08-25T10:04:47.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:47.143+0000 7fe6bccd2640 1 -- 172.21.15.71:0/2333556172 wait complete. 2023-08-25T10:04:47.169 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:49.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:49.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.388+0000 7fe8216f9640 1 Processor -- start 2023-08-25T10:04:49.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.389+0000 7fe8216f9640 1 -- start start 2023-08-25T10:04:49.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.390+0000 7fe8216f9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 0x7fe81c0c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.390+0000 7fe8216f9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c0c1dc0 con 0x7fe81c1509d0 2023-08-25T10:04:49.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.390+0000 7fe8216f9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe81c156e00 con 0x7fe81c0c3660 2023-08-25T10:04:49.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.390+0000 7fe8216f9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c157a80 con 0x7fe81c0c3a40 2023-08-25T10:04:49.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.391+0000 7fe81b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe81c1509d0 0x7fe81c0c5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46648/0 (socket says 172.21.15.71:46648) 2023-08-25T10:04:49.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.391+0000 7fe81b7fe640 1 -- 172.21.15.71:0/2320189364 learned_addr learned my addr 172.21.15.71:0/2320189364 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:49.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.391+0000 7fe8127fc640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 0x7fe81c0c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.391+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186107653 0 0) 0x7fe81c0c1dc0 con 0x7fe81c1509d0 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.391+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808003520 con 0x7fe81c1509d0 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052294305 0 0) 0x7fe808003520 con 0x7fe81c1509d0 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 msgr2=0x7fe81c0c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 0x7fe81c0c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 >> v1:172.21.15.164:6789/0 conn(0x7fe81c0c3660 legacy=0x7fe81c06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c1570e0 con 0x7fe81c1509d0 2023-08-25T10:04:49.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe8140019f0 con 0x7fe81c1509d0 2023-08-25T10:04:49.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 >> v1:172.21.15.71:6789/0 conn(0x7fe81c1509d0 legacy=0x7fe81c0c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe808003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.399 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 0x7fe808005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c157a80 con 0x7fe808003a10 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80800bb40 con 0x7fe8080047c0 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80800bd00 con 0x7fe81c0c3a40 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81affd640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe808003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81b7fe640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 0x7fe808005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe8127fc640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.400 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81affd640 1 -- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 msgr2=0x7fe808005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81affd640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c0c3a40 0x7fe808005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81affd640 1 -- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8080047c0 msgr2=0x7fe808004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81affd640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8080047c0 0x7fe808004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.392+0000 7fe81affd640 1 -- 172.21.15.71:0/2320189364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe808003520 con 0x7fe808003a10 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe81affd640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe808003e30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe80c001750 tx=0x7fe8080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766caea6d8cbb2fe server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0052d0 con 0x7fe808003a10 2023-08-25T10:04:49.401 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c02e050 con 0x7fe808003a10 2023-08-25T10:04:49.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe81a7fc640 1 -- 172.21.15.71:0/2320189364 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0035f0 con 0x7fe808003a10 2023-08-25T10:04:49.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe8216f9640 1 -- 172.21.15.71:0/2320189364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 msgr2=0x7fe808003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe8216f9640 1 --2- 172.21.15.71:0/2320189364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe808003e30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe80c001750 tx=0x7fe8080057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe81b7fe640 1 -- 172.21.15.71:0/2320189364 reap_dead start 2023-08-25T10:04:49.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe8216f9640 1 -- 172.21.15.71:0/2320189364 shutdown_connections 2023-08-25T10:04:49.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe8216f9640 1 -- 172.21.15.71:0/2320189364 >> 172.21.15.71:0/2320189364 conn(0x7fe81c0654f0 msgr2=0x7fe81c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:49.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe8216f9640 1 -- 172.21.15.71:0/2320189364 shutdown_connections 2023-08-25T10:04:49.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.394+0000 7fe8216f9640 1 -- 172.21.15.71:0/2320189364 wait complete. 2023-08-25T10:04:49.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.396+0000 7fe8216f9640 1 Processor -- start 2023-08-25T10:04:49.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.396+0000 7fe8216f9640 1 -- start start 2023-08-25T10:04:49.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.397+0000 7fe8216f9640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe81c150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.397+0000 7fe8216f9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c1f0570 0x7fe81c1f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.397+0000 7fe8216f9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe81c14d8a0 0x7fe81c14dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.397+0000 7fe8216f9640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c0bce90 con 0x7fe808003a10 2023-08-25T10:04:49.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.397+0000 7fe8216f9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c1f0f30 con 0x7fe81c14d8a0 2023-08-25T10:04:49.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.397+0000 7fe8216f9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81c14e260 con 0x7fe81c1f0570 2023-08-25T10:04:49.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe81c150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe81c150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:37464/0 (socket says 172.21.15.71:37464) 2023-08-25T10:04:49.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 -- 172.21.15.71:0/866951043 learned_addr learned my addr 172.21.15.71:0/866951043 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:49.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe8127fc640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c1f0570 0x7fe81c1f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 -- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c1f0570 msgr2=0x7fe81c1f09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81b7fe640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe81c14d8a0 0x7fe81c14dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c1f0570 0x7fe81c1f09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 -- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe81c14d8a0 msgr2=0x7fe81c14dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe81c14d8a0 0x7fe81c14dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 -- 172.21.15.71:0/866951043 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0c9560 con 0x7fe808003a10 2023-08-25T10:04:49.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.398+0000 7fe81affd640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe81c150e60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe80c008fd0 tx=0x7fe80c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:49.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.399+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0076c0 con 0x7fe808003a10 2023-08-25T10:04:49.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.399+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c007a30 con 0x7fe808003a10 2023-08-25T10:04:49.412 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.399+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81c0c9840 con 0x7fe808003a10 2023-08-25T10:04:49.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.399+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c006be0 con 0x7fe808003a10 2023-08-25T10:04:49.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.399+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81c1f0050 con 0x7fe808003a10 2023-08-25T10:04:49.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.400+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81c0c43e0 con 0x7fe808003a10 2023-08-25T10:04:49.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.401+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe80c006d40 con 0x7fe808003a10 2023-08-25T10:04:49.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.402+0000 7fe813fff640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7fc06a080 0x7fe7fc06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:49.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.403+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe80c0aaba0 con 0x7fe808003a10 2023-08-25T10:04:49.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.403+0000 7fe8127fc640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7fc06a080 0x7fe7fc06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:49.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.403+0000 7fe8127fc640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7fc06a080 0x7fe7fc06c540 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0c5cd0 tx=0x7fe804008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:49.415 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.407+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe80c075d40 con 0x7fe808003a10 2023-08-25T10:04:49.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.634+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe81c1efbd0 con 0x7fe808003a10 2023-08-25T10:04:49.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.635+0000 7fe813fff640 1 -- 172.21.15.71:0/866951043 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fe81c1efbd0 con 0x7fe808003a10 2023-08-25T10:04:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:49.637 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:49.638 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:49.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.637+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7fc06a080 msgr2=0x7fe7fc06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.637+0000 7fe8216f9640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7fc06a080 0x7fe7fc06c540 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0c5cd0 tx=0x7fe804008040 comp rx=0 tx=0).stop 2023-08-25T10:04:49.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.637+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 msgr2=0x7fe81c150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.637+0000 7fe8216f9640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe81c150e60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe80c008fd0 tx=0x7fe80c001120 comp rx=0 tx=0).stop 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 shutdown_connections 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe81c14d8a0 0x7fe81c14dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe7fc06a080 0x7fe7fc06c540 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe81c1f0570 0x7fe81c1f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 --2- 172.21.15.71:0/866951043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe808003a10 0x7fe81c150e60 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:49.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 >> 172.21.15.71:0/866951043 conn(0x7fe81c0654f0 msgr2=0x7fe81c0c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:49.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 shutdown_connections 2023-08-25T10:04:49.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:49.638+0000 7fe8216f9640 1 -- 172.21.15.71:0/866951043 wait complete. 2023-08-25T10:04:49.658 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:51.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:51.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.916+0000 7fa6c3a83640 1 Processor -- start 2023-08-25T10:04:51.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.916+0000 7fa6c3a83640 1 -- start start 2023-08-25T10:04:51.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c3a83640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa6bc157850 0x7fa6bc0c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:51.921 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c3a83640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa6bc0c58e0 0x7fa6bc15a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c3a83640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc0c7010 con 0x7fa6bc1511e0 2023-08-25T10:04:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c3a83640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc0c5d10 con 0x7fa6bc0c58e0 2023-08-25T10:04:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c3a83640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6bc0c4ae0 con 0x7fa6bc157850 2023-08-25T10:04:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c2020640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa6bc0c58e0 0x7fa6bc15a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:51.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c2020640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa6bc0c58e0 0x7fa6bc15a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49372/0 (socket says 172.21.15.71:49372) 2023-08-25T10:04:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c2020640 1 -- 172.21.15.71:0/4185621363 learned_addr learned my addr 172.21.15.71:0/4185621363 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.918+0000 7fa6c101e640 1 --2- 172.21.15.71:0/4185621363 >> v2:172.21.15.164:3300/0 conn(0x7fa6bc157850 0x7fa6bc0c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.919+0000 7fa6c2020640 1 -- 172.21.15.71:0/4185621363 >> v2:172.21.15.164:3300/0 conn(0x7fa6bc157850 msgr2=0x7fa6bc0c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.919+0000 7fa6c2020640 1 --2- 172.21.15.71:0/4185621363 >> v2:172.21.15.164:3300/0 conn(0x7fa6bc157850 0x7fa6bc0c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:51.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.919+0000 7fa6c2020640 1 -- 172.21.15.71:0/4185621363 >> v1:172.21.15.164:6789/0 conn(0x7fa6bc1511e0 legacy=0x7fa6bc0c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:51.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.919+0000 7fa6c2020640 1 -- 172.21.15.71:0/4185621363 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc157cb0 con 0x7fa6bc0c58e0 2023-08-25T10:04:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.920+0000 7fa6c2020640 1 --2- 172.21.15.71:0/4185621363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc0c58e0 0x7fa6bc15a6f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc154930 tx=0x7fa6b402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8d0d9e6cc0cc94 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.921+0000 7fa6c081d640 1 -- 172.21.15.71:0/4185621363 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b402ca80 con 0x7fa6bc0c58e0 2023-08-25T10:04:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.921+0000 7fa6c081d640 1 -- 172.21.15.71:0/4185621363 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b402cbe0 con 0x7fa6bc0c58e0 2023-08-25T10:04:51.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.921+0000 7fa6c081d640 1 -- 172.21.15.71:0/4185621363 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40343f0 con 0x7fa6bc0c58e0 2023-08-25T10:04:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 -- 172.21.15.71:0/4185621363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc0c58e0 msgr2=0x7fa6bc15a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/4185621363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc0c58e0 0x7fa6bc15a6f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc154930 tx=0x7fa6b402bee0 comp rx=0 tx=0).stop 2023-08-25T10:04:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 -- 172.21.15.71:0/4185621363 shutdown_connections 2023-08-25T10:04:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/4185621363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc0c58e0 0x7fa6bc15a6f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/4185621363 >> v2:172.21.15.164:3300/0 conn(0x7fa6bc157850 0x7fa6bc0c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:51.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 -- 172.21.15.71:0/4185621363 >> 172.21.15.71:0/4185621363 conn(0x7fa6bc0654f0 msgr2=0x7fa6bc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 -- 172.21.15.71:0/4185621363 shutdown_connections 2023-08-25T10:04:51.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.922+0000 7fa6c3a83640 1 -- 172.21.15.71:0/4185621363 wait complete. 2023-08-25T10:04:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.924+0000 7fa6c3a83640 1 Processor -- start 2023-08-25T10:04:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.924+0000 7fa6c3a83640 1 -- start start 2023-08-25T10:04:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.925+0000 7fa6c3a83640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6bc0c58e0 0x7fa6bc147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.925+0000 7fa6c3a83640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6bc1511e0 0x7fa6bc0c0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.925+0000 7fa6c3a83640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 0x7fa6bc1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:51.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.925+0000 7fa6c3a83640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0c83b0 con 0x7fa6bc157850 2023-08-25T10:04:51.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.925+0000 7fa6c3a83640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc147680 con 0x7fa6bc1511e0 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.925+0000 7fa6c3a83640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc14d7e0 con 0x7fa6bc0c58e0 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 0x7fa6bc1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 0x7fa6bc1594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49386/0 (socket says 172.21.15.71:49386) 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 -- 172.21.15.71:0/2763364514 learned_addr learned my addr 172.21.15.71:0/2763364514 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c101e640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6bc1511e0 0x7fa6bc0c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c181f640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6bc0c58e0 0x7fa6bc147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:51.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 -- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6bc0c58e0 msgr2=0x7fa6bc147140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:51.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6bc0c58e0 0x7fa6bc147140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:51.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 -- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6bc1511e0 msgr2=0x7fa6bc0c0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:51.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6bc1511e0 0x7fa6bc0c0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:51.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.926+0000 7fa6c2020640 1 -- 172.21.15.71:0/2763364514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4009b30 con 0x7fa6bc157850 2023-08-25T10:04:51.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.927+0000 7fa6c2020640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 0x7fa6bc1594f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009e20 tx=0x7fa6b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:51.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.927+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4009510 con 0x7fa6bc157850 2023-08-25T10:04:51.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.927+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b403e450 con 0x7fa6bc157850 2023-08-25T10:04:51.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.927+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b403e7e0 con 0x7fa6bc157850 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.927+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc0c8650 con 0x7fa6bc157850 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.927+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc0c94c0 con 0x7fa6bc157850 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.929+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6bc0c66b0 con 0x7fa6bc157850 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.929+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa6b402cd50 con 0x7fa6bc157850 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.931+0000 7fa6b27fc640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa6a006a080 0x7fa6a006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.931+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fa6b40ab280 con 0x7fa6bc157850 2023-08-25T10:04:51.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.931+0000 7fa6c181f640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa6a006a080 0x7fa6a006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:51.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.932+0000 7fa6c181f640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa6a006a080 0x7fa6a006c540 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8004ea0 tx=0x7fa6b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:51.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:51.939+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa6b40764d0 con 0x7fa6bc157850 2023-08-25T10:04:52.225 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.224+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6bc06edf0 con 0x7fa6bc157850 2023-08-25T10:04:52.227 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.225+0000 7fa6b27fc640 1 -- 172.21.15.71:0/2763364514 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fa6b4079af0 con 0x7fa6bc157850 2023-08-25T10:04:52.227 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:52.227 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:52.228 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:52.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa6a006a080 msgr2=0x7fa6a006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:52.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa6a006a080 0x7fa6a006c540 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8004ea0 tx=0x7fa6b8009040 comp rx=0 tx=0).stop 2023-08-25T10:04:52.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 msgr2=0x7fa6bc1594f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:52.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 0x7fa6bc1594f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4009e20 tx=0x7fa6b4004a40 comp rx=0 tx=0).stop 2023-08-25T10:04:52.230 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 shutdown_connections 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa6bc1511e0 0x7fa6bc0c0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa6a006a080 0x7fa6a006c540 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6bc157850 0x7fa6bc1594f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 --2- 172.21.15.71:0/2763364514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa6bc0c58e0 0x7fa6bc147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 >> 172.21.15.71:0/2763364514 conn(0x7fa6bc0654f0 msgr2=0x7fa6bc0cb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 shutdown_connections 2023-08-25T10:04:52.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:52.228+0000 7fa6c3a83640 1 -- 172.21.15.71:0/2763364514 wait complete. 2023-08-25T10:04:52.258 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:04:52.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:04:54.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:54.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.497+0000 7f9f7b857640 1 Processor -- start 2023-08-25T10:04:54.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.497+0000 7f9f7b857640 1 -- start start 2023-08-25T10:04:54.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f7b857640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9f740c3660 0x7f9f740c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:54.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f7b857640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f740c3a40 0x7f9f7406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:54.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f7b857640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f740c1dc0 con 0x7f9f741509d0 2023-08-25T10:04:54.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f7b857640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f74157a80 con 0x7f9f740c3660 2023-08-25T10:04:54.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f7b857640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7406cf70 con 0x7f9f740c3a40 2023-08-25T10:04:54.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f795f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9f740c3660 0x7f9f740c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:54.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f795f3640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9f740c3660 0x7f9f740c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49398/0 (socket says 172.21.15.71:49398) 2023-08-25T10:04:54.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.499+0000 7f9f795f3640 1 -- 172.21.15.71:0/165256203 learned_addr learned my addr 172.21.15.71:0/165256203 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:54.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.500+0000 7f9f795f3640 1 -- 172.21.15.71:0/165256203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f740c3a40 msgr2=0x7f9f7406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:54.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.500+0000 7f9f78df2640 1 --2- 172.21.15.71:0/165256203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f740c3a40 0x7f9f7406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:54.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.500+0000 7f9f795f3640 1 --2- 172.21.15.71:0/165256203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f740c3a40 0x7f9f7406ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.500+0000 7f9f795f3640 1 -- 172.21.15.71:0/165256203 >> v1:172.21.15.164:6789/0 conn(0x7f9f741509d0 legacy=0x7f9f74150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:54.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.500+0000 7f9f795f3640 1 -- 172.21.15.71:0/165256203 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f74156e00 con 0x7f9f740c3660 2023-08-25T10:04:54.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.501+0000 7f9f795f3640 1 --2- 172.21.15.71:0/165256203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3660 0x7f9f740c1710 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009870 tx=0x7f9f7002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53466a929ae521dd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:54.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.501+0000 7f9f6bfff640 1 -- 172.21.15.71:0/165256203 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70030840 con 0x7f9f740c3660 2023-08-25T10:04:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.501+0000 7f9f6bfff640 1 -- 172.21.15.71:0/165256203 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f700309a0 con 0x7f9f740c3660 2023-08-25T10:04:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.501+0000 7f9f6bfff640 1 -- 172.21.15.71:0/165256203 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70030d30 con 0x7f9f740c3660 2023-08-25T10:04:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 -- 172.21.15.71:0/165256203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3660 msgr2=0x7f9f740c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 --2- 172.21.15.71:0/165256203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3660 0x7f9f740c1710 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009870 tx=0x7f9f7002b980 comp rx=0 tx=0).stop 2023-08-25T10:04:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 -- 172.21.15.71:0/165256203 shutdown_connections 2023-08-25T10:04:54.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 --2- 172.21.15.71:0/165256203 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f740c3a40 0x7f9f7406ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 --2- 172.21.15.71:0/165256203 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3660 0x7f9f740c1710 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 -- 172.21.15.71:0/165256203 >> 172.21.15.71:0/165256203 conn(0x7f9f740654f0 msgr2=0x7f9f74065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 -- 172.21.15.71:0/165256203 shutdown_connections 2023-08-25T10:04:54.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.502+0000 7f9f7b857640 1 -- 172.21.15.71:0/165256203 wait complete. 2023-08-25T10:04:54.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.504+0000 7f9f7b857640 1 Processor -- start 2023-08-25T10:04:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.504+0000 7f9f7b857640 1 -- start start 2023-08-25T10:04:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.505+0000 7f9f7b857640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f740c3660 0x7f9f741517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:54.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.505+0000 7f9f7b857640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 0x7f9f741ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.505+0000 7f9f7b857640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f741509d0 0x7f9f741eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.505+0000 7f9f7b857640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f74152310 con 0x7f9f740c3a40 2023-08-25T10:04:54.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.505+0000 7f9f7b857640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f741ef030 con 0x7f9f740c3660 2023-08-25T10:04:54.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.505+0000 7f9f7b857640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f741f2730 con 0x7f9f741509d0 2023-08-25T10:04:54.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 0x7f9f741ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:54.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 0x7f9f741ee5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49400/0 (socket says 172.21.15.71:49400) 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 -- 172.21.15.71:0/1529212191 learned_addr learned my addr 172.21.15.71:0/1529212191 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f795f3640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f740c3660 0x7f9f741517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f79df4640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f741509d0 0x7f9f741eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 -- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f741509d0 msgr2=0x7f9f741eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f741509d0 0x7f9f741eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 -- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f740c3660 msgr2=0x7f9f741517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:54.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f740c3660 0x7f9f741517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.506+0000 7f9f78df2640 1 -- 172.21.15.71:0/1529212191 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700094d0 con 0x7f9f740c3a40 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.507+0000 7f9f78df2640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 0x7f9f741ee5b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c820 tx=0x7f9f6400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.507+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64004020 con 0x7f9f740c3a40 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.507+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64004180 con 0x7f9f740c3a40 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.507+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f74070520 con 0x7f9f740c3a40 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.507+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64007830 con 0x7f9f740c3a40 2023-08-25T10:04:54.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.507+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f741f1bd0 con 0x7f9f740c3a40 2023-08-25T10:04:54.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.508+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f740c29c0 con 0x7f9f740c3a40 2023-08-25T10:04:54.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.510+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9f64007990 con 0x7f9f740c3a40 2023-08-25T10:04:54.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.511+0000 7f9f69ffb640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9f5806a050 0x7f9f5806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:54.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.511+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9f64088be0 con 0x7f9f740c3a40 2023-08-25T10:04:54.520 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.512+0000 7f9f795f3640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9f5806a050 0x7f9f5806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.512+0000 7f9f795f3640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9f5806a050 0x7f9f5806c510 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9f70036000 tx=0x7f9f7003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:54.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.514+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9f64053e60 con 0x7f9f740c3a40 2023-08-25T10:04:54.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.747+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f741f28a0 con 0x7f9f740c3a40 2023-08-25T10:04:54.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.748+0000 7f9f69ffb640 1 -- 172.21.15.71:0/1529212191 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f9f6401e030 con 0x7f9f740c3a40 2023-08-25T10:04:54.751 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:54.751 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:54.751 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:54.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.750+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9f5806a050 msgr2=0x7f9f5806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:54.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.750+0000 7f9f7b857640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9f5806a050 0x7f9f5806c510 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9f70036000 tx=0x7f9f7003b040 comp rx=0 tx=0).stop 2023-08-25T10:04:54.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 msgr2=0x7f9f741ee5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:54.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 0x7f9f741ee5b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c820 tx=0x7f9f6400cce0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 shutdown_connections 2023-08-25T10:04:54.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9f5806a050 0x7f9f5806c510 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9f741509d0 0x7f9f741eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9f740c3a40 0x7f9f741ee5b0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 --2- 172.21.15.71:0/1529212191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9f740c3660 0x7f9f741517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:54.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 >> 172.21.15.71:0/1529212191 conn(0x7f9f740654f0 msgr2=0x7f9f740c5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:54.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 shutdown_connections 2023-08-25T10:04:54.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:54.751+0000 7f9f7b857640 1 -- 172.21.15.71:0/1529212191 wait complete. 2023-08-25T10:04:54.771 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:56.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:57.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.002+0000 7f92f960d640 1 Processor -- start 2023-08-25T10:04:57.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.002+0000 7f92f960d640 1 -- start start 2023-08-25T10:04:57.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.004+0000 7f92f960d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f92f4159fc0 0x7f92f40c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.004+0000 7f92f960d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f406a960 con 0x7f92f406cc70 2023-08-25T10:04:57.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.004+0000 7f92f960d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f406ac20 con 0x7f92f41594d0 2023-08-25T10:04:57.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.004+0000 7f92f960d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f406af00 con 0x7f92f4159fc0 2023-08-25T10:04:57.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.004+0000 7f92ea7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f92f406cc70 0x7f92f41598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46608/0 (socket says 172.21.15.71:46608) 2023-08-25T10:04:57.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.004+0000 7f92ea7fc640 1 -- 172.21.15.71:0/886949774 learned_addr learned my addr 172.21.15.71:0/886949774 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:57.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f37fe640 1 --2- 172.21.15.71:0/886949774 >> v2:172.21.15.164:3300/0 conn(0x7f92f4159fc0 0x7f92f40c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185592001 0 0) 0x7f92f406a960 con 0x7f92f406cc70 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4003520 con 0x7f92f406cc70 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984397410 0 0) 0x7f92d4003520 con 0x7f92f406cc70 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 >> v2:172.21.15.164:3300/0 conn(0x7f92f4159fc0 msgr2=0x7f92f40c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 --2- 172.21.15.71:0/886949774 >> v2:172.21.15.164:3300/0 conn(0x7f92f4159fc0 0x7f92f40c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 >> v1:172.21.15.164:6789/0 conn(0x7f92f41594d0 legacy=0x7f92f414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f40c2e60 con 0x7f92f406cc70 2023-08-25T10:04:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f92dc001670 con 0x7f92f406cc70 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 >> v1:172.21.15.71:6789/0 conn(0x7f92f406cc70 legacy=0x7f92f41598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92d4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d40047c0 0x7f92d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d40051f0 0x7f92d400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f406af00 con 0x7f92d40039e0 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d400e350 con 0x7f92d40047c0 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.005+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d400e510 con 0x7f92d40051f0 2023-08-25T10:04:57.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92d4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92ea7fc640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d40047c0 0x7f92d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f2ffd640 1 -- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d40051f0 msgr2=0x7f92d400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92d40051f0 0x7f92d400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f2ffd640 1 -- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d40047c0 msgr2=0x7f92d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d40047c0 0x7f92d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f2ffd640 1 -- 172.21.15.71:0/886949774 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d4003520 con 0x7f92d40039e0 2023-08-25T10:04:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.006+0000 7f92f37fe640 1 -- 172.21.15.71:0/886949774 reap_dead start 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.007+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92d4003e00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f92e4001750 tx=0x7f92d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d99786d573ae4a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.007+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40052d0 con 0x7f92d40039e0 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.007+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e402e050 con 0x7f92d40039e0 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.007+0000 7f92f27fc640 1 -- 172.21.15.71:0/886949774 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4003830 con 0x7f92d40039e0 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 -- 172.21.15.71:0/886949774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 msgr2=0x7f92d4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92d4003e00 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f92e4001750 tx=0x7f92d40057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 -- 172.21.15.71:0/886949774 shutdown_connections 2023-08-25T10:04:57.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 --2- 172.21.15.71:0/886949774 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92d4003e00 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 -- 172.21.15.71:0/886949774 >> 172.21.15.71:0/886949774 conn(0x7f92f40654f0 msgr2=0x7f92f4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:57.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 -- 172.21.15.71:0/886949774 shutdown_connections 2023-08-25T10:04:57.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.008+0000 7f92f960d640 1 -- 172.21.15.71:0/886949774 wait complete. 2023-08-25T10:04:57.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.010+0000 7f92f960d640 1 Processor -- start 2023-08-25T10:04:57.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.010+0000 7f92f960d640 1 -- start start 2023-08-25T10:04:57.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f960d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92f4145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f960d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92f41ee430 0x7f92f40c0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f960d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92f41ee870 0x7f92f40c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f960d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f406f650 con 0x7f92d40039e0 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f960d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f41f22d0 con 0x7f92f41ee870 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f960d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40660a0 con 0x7f92f41ee430 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92f4145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92f4145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49420/0 (socket says 172.21.15.71:49420) 2023-08-25T10:04:57.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.011+0000 7f92f2ffd640 1 -- 172.21.15.71:0/534528114 learned_addr learned my addr 172.21.15.71:0/534528114 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92ea7fc640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92f41ee430 0x7f92f40c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f37fe640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92f41ee870 0x7f92f40c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f2ffd640 1 -- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92f41ee430 msgr2=0x7f92f40c0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92f41ee430 0x7f92f40c0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f2ffd640 1 -- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92f41ee870 msgr2=0x7f92f40c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92f41ee870 0x7f92f40c0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f2ffd640 1 -- 172.21.15.71:0/534528114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f40c8190 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f2ffd640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92f4145e00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f92e402bc00 tx=0x7f92e4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40076c0 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e4007a30 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4006be0 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f40c1000 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.012+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f41f19a0 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.014+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f4069160 con 0x7f92d40039e0 2023-08-25T10:04:57.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.015+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f92e4006d40 con 0x7f92d40039e0 2023-08-25T10:04:57.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.015+0000 7f92ebfff640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92d806a080 0x7f92d806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:57.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.016+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f92e40aa6b0 con 0x7f92d40039e0 2023-08-25T10:04:57.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.016+0000 7f92ea7fc640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92d806a080 0x7f92d806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:57.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.016+0000 7f92ea7fc640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92d806a080 0x7f92d806c540 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f92dc001050 tx=0x7f92dc001150 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:57.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.020+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f92e4079360 con 0x7f92d40039e0 2023-08-25T10:04:57.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:04:57.072 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:04:57.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.249+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f41444c0 con 0x7f92d40039e0 2023-08-25T10:04:57.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.250+0000 7f92ebfff640 1 -- 172.21.15.71:0/534528114 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f92e4078ef0 con 0x7f92d40039e0 2023-08-25T10:04:57.253 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:57.253 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:57.253 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:57.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92d806a080 msgr2=0x7f92d806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92d806a080 0x7f92d806c540 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f92dc001050 tx=0x7f92dc001150 comp rx=0 tx=0).stop 2023-08-25T10:04:57.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 msgr2=0x7f92f4145e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:57.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92f4145e00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f92e402bc00 tx=0x7f92e4001120 comp rx=0 tx=0).stop 2023-08-25T10:04:57.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 shutdown_connections 2023-08-25T10:04:57.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92f41ee870 0x7f92f40c0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f92d806a080 0x7f92d806c540 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f92f41ee430 0x7f92f40c0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 --2- 172.21.15.71:0/534528114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92d40039e0 0x7f92f4145e00 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:57.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 >> 172.21.15.71:0/534528114 conn(0x7f92f40654f0 msgr2=0x7f92f4146b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:57.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.253+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 shutdown_connections 2023-08-25T10:04:57.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:57.254+0000 7f92f960d640 1 -- 172.21.15.71:0/534528114 wait complete. 2023-08-25T10:04:57.275 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:04:59.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:04:59.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.495+0000 7fc26163f640 1 Processor -- start 2023-08-25T10:04:59.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.495+0000 7fc26163f640 1 -- start start 2023-08-25T10:04:59.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.496+0000 7fc26163f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c066010 0x7fc25c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc26163f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c15db40 con 0x7fc25c157450 2023-08-25T10:04:59.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc26163f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c15d230 con 0x7fc25c070cf0 2023-08-25T10:04:59.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc26163f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c06cb40 con 0x7fc25c066010 2023-08-25T10:04:59.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc25affd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc25c157450 0x7fc25c06e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46612/0 (socket says 172.21.15.71:46612) 2023-08-25T10:04:59.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc25affd640 1 -- 172.21.15.71:0/1240123697 learned_addr learned my addr 172.21.15.71:0/1240123697 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:04:59.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981954703 0 0) 0x7fc25c15db40 con 0x7fc25c157450 2023-08-25T10:04:59.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc248003520 con 0x7fc25c157450 2023-08-25T10:04:59.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.497+0000 7fc253fff640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c066010 0x7fc25c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942639096 0 0) 0x7fc248003520 con 0x7fc25c157450 2023-08-25T10:04:59.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c066010 msgr2=0x7fc25c0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:59.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc25c066010 0x7fc25c0c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 >> v1:172.21.15.164:6789/0 conn(0x7fc25c070cf0 legacy=0x7fc25c0c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c06ce20 con 0x7fc25c157450 2023-08-25T10:04:59.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fc24c001670 con 0x7fc25c157450 2023-08-25T10:04:59.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 >> v1:172.21.15.71:6789/0 conn(0x7fc25c157450 legacy=0x7fc25c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc248003a10 0x7fc248003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2480047c0 0x7fc248004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2480051d0 0x7fc24800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c06cb40 con 0x7fc2480047c0 2023-08-25T10:04:59.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24800c0e0 con 0x7fc2480051d0 2023-08-25T10:04:59.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24800c2a0 con 0x7fc248003a10 2023-08-25T10:04:59.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc253fff640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2480047c0 0x7fc248004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.498+0000 7fc25affd640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc248003a10 0x7fc248003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.499+0000 7fc25b7fe640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2480051d0 0x7fc24800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.499+0000 7fc253fff640 1 -- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc248003a10 msgr2=0x7fc248003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:59.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.499+0000 7fc253fff640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc248003a10 0x7fc248003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.499+0000 7fc253fff640 1 -- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2480051d0 msgr2=0x7fc24800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:04:59.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.499+0000 7fc253fff640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2480051d0 0x7fc24800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.499+0000 7fc253fff640 1 -- 172.21.15.71:0/1240123697 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248003520 con 0x7fc2480047c0 2023-08-25T10:04:59.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc253fff640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2480047c0 0x7fc248004c40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc25c06c950 tx=0x7fc2480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a439f2631ff1e69 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:59.511 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244035680 con 0x7fc2480047c0 2023-08-25T10:04:59.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2440357e0 con 0x7fc2480047c0 2023-08-25T10:04:59.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc25a7fc640 1 -- 172.21.15.71:0/1240123697 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244035b90 con 0x7fc2480047c0 2023-08-25T10:04:59.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc26163f640 1 -- 172.21.15.71:0/1240123697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2480047c0 msgr2=0x7fc248004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc26163f640 1 --2- 172.21.15.71:0/1240123697 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2480047c0 0x7fc248004c40 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fc25c06c950 tx=0x7fc2480057f0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.500+0000 7fc25b7fe640 1 -- 172.21.15.71:0/1240123697 reap_dead start 2023-08-25T10:04:59.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.501+0000 7fc26163f640 1 -- 172.21.15.71:0/1240123697 shutdown_connections 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.501+0000 7fc26163f640 1 -- 172.21.15.71:0/1240123697 >> 172.21.15.71:0/1240123697 conn(0x7fc25c0654f0 msgr2=0x7fc25c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.501+0000 7fc26163f640 1 -- 172.21.15.71:0/1240123697 shutdown_connections 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.501+0000 7fc26163f640 1 -- 172.21.15.71:0/1240123697 wait complete. 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.502+0000 7fc26163f640 1 Processor -- start 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.503+0000 7fc26163f640 1 -- start start 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc26163f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2480047c0 0x7fc25c158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc26163f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c157b20 0x7fc25c160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc26163f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 0x7fc25c1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc26163f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c160590 con 0x7fc25c15e3f0 2023-08-25T10:04:59.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc26163f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c159270 con 0x7fc25c157b20 2023-08-25T10:04:59.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc26163f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25c15ac00 con 0x7fc2480047c0 2023-08-25T10:04:59.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 0x7fc25c1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 0x7fc25c1517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49446/0 (socket says 172.21.15.71:49446) 2023-08-25T10:04:59.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 -- 172.21.15.71:0/166059638 learned_addr learned my addr 172.21.15.71:0/166059638 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 -- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2480047c0 msgr2=0x7fc25c158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25affd640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2480047c0 0x7fc25c158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc253fff640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c157b20 0x7fc25c160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2480047c0 0x7fc25c158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 -- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c157b20 msgr2=0x7fc25c160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c157b20 0x7fc25c160ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.504+0000 7fc25b7fe640 1 -- 172.21.15.71:0/166059638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c15a680 con 0x7fc25c15e3f0 2023-08-25T10:04:59.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.505+0000 7fc25b7fe640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 0x7fc25c1517d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc254000f80 tx=0x7fc25400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.505+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc254003b70 con 0x7fc25c15e3f0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.505+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc254003cd0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.505+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2540052b0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.505+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c15a920 con 0x7fc25c15e3f0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.505+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c1601b0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.506+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c15dfb0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.508+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc254005410 con 0x7fc25c15e3f0 2023-08-25T10:04:59.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.509+0000 7fc2537fe640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc23c06a080 0x7fc23c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:04:59.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.509+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc254088320 con 0x7fc25c15e3f0 2023-08-25T10:04:59.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.509+0000 7fc25affd640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc23c06a080 0x7fc23c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:04:59.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.510+0000 7fc25affd640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc23c06a080 0x7fc23c06c540 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc24c001eb0 tx=0x7fc24c000f50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:04:59.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.516+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc25c15dfb0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.820+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc25c1606d0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.821+0000 7fc2537fe640 1 -- 172.21.15.71:0/166059638 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fc25c1606d0 con 0x7fc25c15e3f0 2023-08-25T10:04:59.823 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:04:59.824 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:04:59.824 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:04:59.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc23c06a080 msgr2=0x7fc23c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc23c06a080 0x7fc23c06c540 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc24c001eb0 tx=0x7fc24c000f50 comp rx=0 tx=0).stop 2023-08-25T10:04:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 msgr2=0x7fc25c1517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:04:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 0x7fc25c1517d0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc254000f80 tx=0x7fc25400dc00 comp rx=0 tx=0).stop 2023-08-25T10:04:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 shutdown_connections 2023-08-25T10:04:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc25c157b20 0x7fc25c160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc23c06a080 0x7fc23c06c540 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc25c15e3f0 0x7fc25c1517d0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 --2- 172.21.15.71:0/166059638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2480047c0 0x7fc25c158d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:04:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 >> 172.21.15.71:0/166059638 conn(0x7fc25c0654f0 msgr2=0x7fc25c0c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:04:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 shutdown_connections 2023-08-25T10:04:59.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:04:59.824+0000 7fc26163f640 1 -- 172.21.15.71:0/166059638 wait complete. 2023-08-25T10:04:59.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:01.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:02.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.101+0000 7f4c64aec640 1 Processor -- start 2023-08-25T10:05:02.103 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.101+0000 7f4c64aec640 1 -- start start 2023-08-25T10:05:02.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c64aec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4c6006ed30 0x7f4c6015a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:02.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c64aec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4c600c6de0 0x7f4c60159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:02.105 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c64aec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c601593c0 con 0x7f4c6006e240 2023-08-25T10:05:02.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c64aec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6014fca0 con 0x7f4c600c6de0 2023-08-25T10:05:02.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c64aec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c600c7320 con 0x7f4c6006ed30 2023-08-25T10:05:02.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c5ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4c600c6de0 0x7f4c60159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:02.106 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c5ed9d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f4c600c6de0 0x7f4c60159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39858/0 (socket says 172.21.15.71:39858) 2023-08-25T10:05:02.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/1364893016 learned_addr learned my addr 172.21.15.71:0/1364893016 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:02.107 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c57fff640 1 --2- 172.21.15.71:0/1364893016 >> v2:172.21.15.164:3300/0 conn(0x7f4c6006ed30 0x7f4c6015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:02.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/1364893016 >> v2:172.21.15.164:3300/0 conn(0x7f4c6006ed30 msgr2=0x7f4c6015a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:02.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.103+0000 7f4c5ed9d640 1 --2- 172.21.15.71:0/1364893016 >> v2:172.21.15.164:3300/0 conn(0x7f4c6006ed30 0x7f4c6015a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.104+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/1364893016 >> v1:172.21.15.164:6789/0 conn(0x7f4c6006e240 legacy=0x7f4c6006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:02.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.104+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/1364893016 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60070dd0 con 0x7f4c600c6de0 2023-08-25T10:05:02.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.105+0000 7f4c5ed9d640 1 --2- 172.21.15.71:0/1364893016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c60159850 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4c6006f250 tx=0x7f4c5802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f63394118fcdc59 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:02.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.105+0000 7f4c5dd9b640 1 -- 172.21.15.71:0/1364893016 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5802cb10 con 0x7f4c600c6de0 2023-08-25T10:05:02.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c5dd9b640 1 -- 172.21.15.71:0/1364893016 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5802cc70 con 0x7f4c600c6de0 2023-08-25T10:05:02.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c5dd9b640 1 -- 172.21.15.71:0/1364893016 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c580343f0 con 0x7f4c600c6de0 2023-08-25T10:05:02.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c64aec640 1 -- 172.21.15.71:0/1364893016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 msgr2=0x7f4c60159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c64aec640 1 --2- 172.21.15.71:0/1364893016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c60159850 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f4c6006f250 tx=0x7f4c5802bf70 comp rx=0 tx=0).stop 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c64aec640 1 -- 172.21.15.71:0/1364893016 shutdown_connections 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c64aec640 1 --2- 172.21.15.71:0/1364893016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c60159850 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c64aec640 1 --2- 172.21.15.71:0/1364893016 >> v2:172.21.15.164:3300/0 conn(0x7f4c6006ed30 0x7f4c6015a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.106+0000 7f4c64aec640 1 -- 172.21.15.71:0/1364893016 >> 172.21.15.71:0/1364893016 conn(0x7f4c60069e80 msgr2=0x7f4c600654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.107+0000 7f4c64aec640 1 -- 172.21.15.71:0/1364893016 shutdown_connections 2023-08-25T10:05:02.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.107+0000 7f4c64aec640 1 -- 172.21.15.71:0/1364893016 wait complete. 2023-08-25T10:05:02.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.108+0000 7f4c64aec640 1 Processor -- start 2023-08-25T10:05:02.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 -- start start 2023-08-25T10:05:02.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c6006e240 0x7f4c601514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:02.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6006ed30 0x7f4c60157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:02.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c601ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:02.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6014fde0 con 0x7f4c600c6de0 2023-08-25T10:05:02.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6014a420 con 0x7f4c6006ed30 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.109+0000 7f4c64aec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60150920 con 0x7f4c6006e240 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c601ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c601ee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39864/0 (socket says 172.21.15.71:39864) 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/2826051420 learned_addr learned my addr 172.21.15.71:0/2826051420 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5e59c640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c6006e240 0x7f4c601514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c6006e240 msgr2=0x7f4c601514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:02.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c6006e240 0x7f4c601514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c57fff640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6006ed30 0x7f4c60157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:02.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6006ed30 msgr2=0x7f4c60157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:02.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6006ed30 0x7f4c60157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.110+0000 7f4c5ed9d640 1 -- 172.21.15.71:0/2826051420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58009b30 con 0x7f4c600c6de0 2023-08-25T10:05:02.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.111+0000 7f4c5ed9d640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c601ee9a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009f40 tx=0x7f4c58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:02.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.111+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58009510 con 0x7f4c600c6de0 2023-08-25T10:05:02.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.111+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5803e450 con 0x7f4c600c6de0 2023-08-25T10:05:02.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.111+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5803e7e0 con 0x7f4c600c6de0 2023-08-25T10:05:02.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.111+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c60148ea0 con 0x7f4c600c6de0 2023-08-25T10:05:02.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.111+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6014fff0 con 0x7f4c600c6de0 2023-08-25T10:05:02.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.112+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c60069160 con 0x7f4c600c6de0 2023-08-25T10:05:02.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.113+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4c5803e940 con 0x7f4c600c6de0 2023-08-25T10:05:02.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.114+0000 7f4c56ffd640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4c4006a080 0x7f4c4006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:02.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.114+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4c580ab240 con 0x7f4c600c6de0 2023-08-25T10:05:02.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.114+0000 7f4c5e59c640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4c4006a080 0x7f4c4006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:02.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.115+0000 7f4c5e59c640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4c4006a080 0x7f4c4006c540 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000a750 tx=0x7f4c50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:02.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.118+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4c58076340 con 0x7f4c600c6de0 2023-08-25T10:05:02.347 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.345+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c60150a60 con 0x7f4c600c6de0 2023-08-25T10:05:02.348 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.346+0000 7f4c56ffd640 1 -- 172.21.15.71:0/2826051420 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f4c58033e80 con 0x7f4c600c6de0 2023-08-25T10:05:02.349 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:02.349 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:02.349 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:02.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4c4006a080 msgr2=0x7f4c4006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:02.351 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4c4006a080 0x7f4c4006c540 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000a750 tx=0x7f4c50008040 comp rx=0 tx=0).stop 2023-08-25T10:05:02.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 msgr2=0x7f4c601ee9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:02.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c601ee9a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009f40 tx=0x7f4c58004a40 comp rx=0 tx=0).stop 2023-08-25T10:05:02.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 shutdown_connections 2023-08-25T10:05:02.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c6006ed30 0x7f4c60157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4c4006a080 0x7f4c4006c540 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.352 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c600c6de0 0x7f4c601ee9a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 --2- 172.21.15.71:0/2826051420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c6006e240 0x7f4c601514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:02.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 >> 172.21.15.71:0/2826051420 conn(0x7f4c60069e80 msgr2=0x7f4c60158510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:02.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.349+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 shutdown_connections 2023-08-25T10:05:02.353 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:02.350+0000 7f4c64aec640 1 -- 172.21.15.71:0/2826051420 wait complete. 2023-08-25T10:05:02.370 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:04.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.615+0000 7f55bafab640 1 Processor -- start 2023-08-25T10:05:04.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.615+0000 7f55bafab640 1 -- start start 2023-08-25T10:05:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.617+0000 7f55bafab640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f55b40af420 0x7f55b4069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.617+0000 7f55bafab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b406b350 0x7f55b40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.617+0000 7f55bafab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b40c0690 con 0x7f55b40c75c0 2023-08-25T10:05:04.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.617+0000 7f55bafab640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40bf830 con 0x7f55b40af420 2023-08-25T10:05:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.617+0000 7f55bafab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40bfb10 con 0x7f55b406b350 2023-08-25T10:05:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.618+0000 7f55ba7aa640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f55b40c75c0 0x7f55b4160ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:40672/0 (socket says 172.21.15.71:40672) 2023-08-25T10:05:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.618+0000 7f55ba7aa640 1 -- 172.21.15.71:0/1010169240 learned_addr learned my addr 172.21.15.71:0/1010169240 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.618+0000 7f55b97a8640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b406b350 0x7f55b40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.618+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/1010169240 >> v2:172.21.15.164:3300/0 conn(0x7f55b40af420 0x7f55b4069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.622 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.618+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3928500574 0 0) 0x7f55ac001490 con 0x7f55b40c75c0 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.618+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a0003520 con 0x7f55b40c75c0 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3760567468 0 0) 0x7f55a0003520 con 0x7f55b40c75c0 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b406b350 msgr2=0x7f55b40c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b406b350 0x7f55b40c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 >> v2:172.21.15.164:3300/0 conn(0x7f55b40af420 msgr2=0x7f55b4069850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 --2- 172.21.15.71:0/1010169240 >> v2:172.21.15.164:3300/0 conn(0x7f55b40af420 0x7f55b4069850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.623 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40bfdf0 con 0x7f55b40c75c0 2023-08-25T10:05:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f55ac001b80 con 0x7f55b40c75c0 2023-08-25T10:05:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 >> v1:172.21.15.71:6789/0 conn(0x7f55b40c75c0 legacy=0x7f55b4160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55a00039e0 0x7f55a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004380 0x7f55a0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55a0004ed0 0x7f55a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40bf830 con 0x7f55a0004ed0 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40bfb10 con 0x7f55a0004380 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a000bec0 con 0x7f55a00039e0 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.619+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55a0004ed0 0x7f55a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55b97a8640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004380 0x7f55a0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55a00039e0 0x7f55a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55ba7aa640 1 -- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55a00039e0 msgr2=0x7f55a0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:04.625 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55a00039e0 0x7f55a0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55ba7aa640 1 -- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004380 msgr2=0x7f55a0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:04.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004380 0x7f55a0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55ba7aa640 1 -- 172.21.15.71:0/1010169240 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40bfdf0 con 0x7f55a0004ed0 2023-08-25T10:05:04.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.620+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55a00039e0 0x7f55a0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:04.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.621+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55a0004ed0 0x7f55a000b980 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0013d0 tx=0x7f55a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1411302a989e0205 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:04.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.621+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0072d0 con 0x7f55a0004ed0 2023-08-25T10:05:04.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.621+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac02f070 con 0x7f55a0004ed0 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.621+0000 7f55b8fa7640 1 -- 172.21.15.71:0/1010169240 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac007670 con 0x7f55a0004ed0 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55bafab640 1 -- 172.21.15.71:0/1010169240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55a0004ed0 msgr2=0x7f55a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55bafab640 1 --2- 172.21.15.71:0/1010169240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55a0004ed0 0x7f55a000b980 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0013d0 tx=0x7f55a00057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55ba7aa640 1 -- 172.21.15.71:0/1010169240 reap_dead start 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55bafab640 1 -- 172.21.15.71:0/1010169240 shutdown_connections 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55bafab640 1 -- 172.21.15.71:0/1010169240 >> 172.21.15.71:0/1010169240 conn(0x7f55b406c280 msgr2=0x7f55b406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:04.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55bafab640 1 -- 172.21.15.71:0/1010169240 shutdown_connections 2023-08-25T10:05:04.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.622+0000 7f55bafab640 1 -- 172.21.15.71:0/1010169240 wait complete. 2023-08-25T10:05:04.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.625+0000 7f55bafab640 1 Processor -- start 2023-08-25T10:05:04.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.625+0000 7f55bafab640 1 -- start start 2023-08-25T10:05:04.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55bafab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004ed0 0x7f55b40a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55bafab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b40abd10 0x7f55b41f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55bafab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 0x7f55b415d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55bafab640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41f5c40 con 0x7f55b40ca780 2023-08-25T10:05:04.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55bafab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41f5ee0 con 0x7f55a0004ed0 2023-08-25T10:05:04.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55bafab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40ac5e0 con 0x7f55b40abd10 2023-08-25T10:05:04.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55ba7aa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 0x7f55b415d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55ba7aa640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 0x7f55b415d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39872/0 (socket says 172.21.15.71:39872) 2023-08-25T10:05:04.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55ba7aa640 1 -- 172.21.15.71:0/2660394354 learned_addr learned my addr 172.21.15.71:0/2660394354 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:04.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55b97a8640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b40abd10 0x7f55b41f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.627+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004ed0 0x7f55b40a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.633 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.628+0000 7f55ba7aa640 1 -- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b40abd10 msgr2=0x7f55b41f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.628+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b40abd10 0x7f55b41f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.628+0000 7f55ba7aa640 1 -- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004ed0 msgr2=0x7f55b40a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.628+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004ed0 0x7f55b40a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.628+0000 7f55ba7aa640 1 -- 172.21.15.71:0/2660394354 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b41f61a0 con 0x7f55b40ca780 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.628+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004ed0 0x7f55b40a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.629+0000 7f55ba7aa640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 0x7f55b415d2a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0013d0 tx=0x7f55ac001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:04.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.629+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac040070 con 0x7f55b40ca780 2023-08-25T10:05:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.629+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac001490 con 0x7f55b40ca780 2023-08-25T10:05:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.629+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac004d70 con 0x7f55b40ca780 2023-08-25T10:05:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.629+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b41f6480 con 0x7f55b40ca780 2023-08-25T10:05:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.629+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b41f6a40 con 0x7f55b40ca780 2023-08-25T10:05:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.631+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b406b1c0 con 0x7f55b40ca780 2023-08-25T10:05:04.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.632+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f55ac031030 con 0x7f55b40ca780 2023-08-25T10:05:04.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.633+0000 7f55a67fc640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f559806a080 0x7f559806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:04.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.633+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f55ac0a9c10 con 0x7f55b40ca780 2023-08-25T10:05:04.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.634+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f559806a080 0x7f559806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:04.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.634+0000 7f55b9fa9640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f559806a080 0x7f559806c540 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009bc0 tx=0x7f55b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:04.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.639+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f55ac074db0 con 0x7f55b40ca780 2023-08-25T10:05:04.866 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.864+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b40c0690 con 0x7f55b40ca780 2023-08-25T10:05:04.867 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.865+0000 7f55a67fc640 1 -- 172.21.15.71:0/2660394354 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f55b40c0690 con 0x7f55b40ca780 2023-08-25T10:05:04.868 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:04.868 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:04.869 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:04.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f559806a080 msgr2=0x7f559806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:04.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f559806a080 0x7f559806c540 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009bc0 tx=0x7f55b0008040 comp rx=0 tx=0).stop 2023-08-25T10:05:04.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 msgr2=0x7f55b415d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:04.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 0x7f55b415d2a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f55ac0013d0 tx=0x7f55ac001360 comp rx=0 tx=0).stop 2023-08-25T10:05:04.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 shutdown_connections 2023-08-25T10:05:04.870 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f559806a080 0x7f559806c540 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f55b40ca780 0x7f55b415d2a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f55b40abd10 0x7f55b41f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 --2- 172.21.15.71:0/2660394354 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f55a0004ed0 0x7f55b40a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:04.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 >> 172.21.15.71:0/2660394354 conn(0x7f55b406c280 msgr2=0x7f55b40c8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:04.871 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 shutdown_connections 2023-08-25T10:05:04.872 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:04.868+0000 7f55bafab640 1 -- 172.21.15.71:0/2660394354 wait complete. 2023-08-25T10:05:04.887 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:05:04.888 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:05:06.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:07.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.107+0000 7eff156a7640 1 Processor -- start 2023-08-25T10:05:07.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.108+0000 7eff156a7640 1 -- start start 2023-08-25T10:05:07.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.109+0000 7eff156a7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7eff100c3a40 0x7eff100c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:07.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.109+0000 7eff156a7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff100c1dc0 con 0x7eff101509d0 2023-08-25T10:05:07.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.109+0000 7eff156a7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff10156e00 con 0x7eff100c3660 2023-08-25T10:05:07.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.109+0000 7eff156a7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff10157a80 con 0x7eff100c3a40 2023-08-25T10:05:07.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.109+0000 7eff0f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7eff101509d0 0x7eff10150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:40674/0 (socket says 172.21.15.71:40674) 2023-08-25T10:05:07.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.109+0000 7eff07fff640 1 --2- 172.21.15.71:0/2797001806 >> v2:172.21.15.71:3300/0 conn(0x7eff100c3a40 0x7eff100c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:07.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.110+0000 7eff0f7fe640 1 -- 172.21.15.71:0/2797001806 learned_addr learned my addr 172.21.15.71:0/2797001806 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:07.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.110+0000 7eff07fff640 1 -- 172.21.15.71:0/2797001806 >> v1:172.21.15.164:6789/0 conn(0x7eff100c3660 legacy=0x7eff1006ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:07.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.110+0000 7eff07fff640 1 -- 172.21.15.71:0/2797001806 >> v1:172.21.15.71:6789/0 conn(0x7eff101509d0 legacy=0x7eff10150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:07.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.110+0000 7eff07fff640 1 -- 172.21.15.71:0/2797001806 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff101570e0 con 0x7eff100c3a40 2023-08-25T10:05:07.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.110+0000 7eff0e7fc640 1 -- 172.21.15.71:0/2797001806 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591543593 0 0) 0x7eff100c1dc0 con 0x7eff101509d0 2023-08-25T10:05:07.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.111+0000 7eff07fff640 1 --2- 172.21.15.71:0/2797001806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff100c1710 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efef80097a0 tx=0x7efef802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1698c5dab740b6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:07.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.111+0000 7eff0e7fc640 1 -- 172.21.15.71:0/2797001806 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8030610 con 0x7eff100c3a40 2023-08-25T10:05:07.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.111+0000 7eff0e7fc640 1 -- 172.21.15.71:0/2797001806 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef8030770 con 0x7eff100c3a40 2023-08-25T10:05:07.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.111+0000 7eff0e7fc640 1 -- 172.21.15.71:0/2797001806 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef8030b00 con 0x7eff100c3a40 2023-08-25T10:05:07.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 -- 172.21.15.71:0/2797001806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 msgr2=0x7eff100c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:07.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 --2- 172.21.15.71:0/2797001806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff100c1710 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efef80097a0 tx=0x7efef802b960 comp rx=0 tx=0).stop 2023-08-25T10:05:07.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 -- 172.21.15.71:0/2797001806 shutdown_connections 2023-08-25T10:05:07.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 --2- 172.21.15.71:0/2797001806 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff100c1710 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 -- 172.21.15.71:0/2797001806 >> 172.21.15.71:0/2797001806 conn(0x7eff100654f0 msgr2=0x7eff10065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:07.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 -- 172.21.15.71:0/2797001806 shutdown_connections 2023-08-25T10:05:07.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.112+0000 7eff156a7640 1 -- 172.21.15.71:0/2797001806 wait complete. 2023-08-25T10:05:07.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.114+0000 7eff156a7640 1 Processor -- start 2023-08-25T10:05:07.117 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.114+0000 7eff156a7640 1 -- start start 2023-08-25T10:05:07.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff156a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff100c3660 0x7eff10158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff156a7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff1006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff156a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff101509d0 0x7eff1006f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff156a7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1015a830 con 0x7eff100c3a40 2023-08-25T10:05:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff156a7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff10159700 con 0x7eff100c3660 2023-08-25T10:05:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff156a7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff100c5c50 con 0x7eff101509d0 2023-08-25T10:05:07.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff07fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff1006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:07.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff07fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff1006ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39890/0 (socket says 172.21.15.71:39890) 2023-08-25T10:05:07.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff07fff640 1 -- 172.21.15.71:0/2677196150 learned_addr learned my addr 172.21.15.71:0/2677196150 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:07.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff0effd640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff100c3660 0x7eff10158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:07.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.115+0000 7eff07fff640 1 -- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff101509d0 msgr2=0x7eff1006f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:07.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff07fff640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff101509d0 0x7eff1006f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff07fff640 1 -- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff100c3660 msgr2=0x7eff10158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:07.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff07fff640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff100c3660 0x7eff10158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff07fff640 1 -- 172.21.15.71:0/2677196150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef8009400 con 0x7eff100c3a40 2023-08-25T10:05:07.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff0effd640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff100c3660 0x7eff10158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:07.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff07fff640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff1006ed10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efef80098c0 tx=0x7efef8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:07.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef803e040 con 0x7eff100c3a40 2023-08-25T10:05:07.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef8039c90 con 0x7eff100c3a40 2023-08-25T10:05:07.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80443f0 con 0x7eff100c3a40 2023-08-25T10:05:07.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1015aa90 con 0x7eff100c3a40 2023-08-25T10:05:07.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.116+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff101ee8c0 con 0x7eff100c3a40 2023-08-25T10:05:07.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.117+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1015b0f0 con 0x7eff100c3a40 2023-08-25T10:05:07.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.118+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7efef8043390 con 0x7eff100c3a40 2023-08-25T10:05:07.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.119+0000 7eff077fe640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efef006a080 0x7efef006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:07.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.119+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7efef80aaee0 con 0x7eff100c3a40 2023-08-25T10:05:07.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.119+0000 7eff0effd640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efef006a080 0x7efef006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:07.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.120+0000 7eff0effd640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efef006a080 0x7efef006c540 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7eff0000a830 tx=0x7eff00008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:07.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.124+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7efef8075fe0 con 0x7eff100c3a40 2023-08-25T10:05:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.354+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff100c4e80 con 0x7eff100c3a40 2023-08-25T10:05:07.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.356+0000 7eff077fe640 1 -- 172.21.15.71:0/2677196150 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7efef8034090 con 0x7eff100c3a40 2023-08-25T10:05:07.358 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:07.358 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:07.358 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.358+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efef006a080 msgr2=0x7efef006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.358+0000 7eff156a7640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efef006a080 0x7efef006c540 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7eff0000a830 tx=0x7eff00008040 comp rx=0 tx=0).stop 2023-08-25T10:05:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.358+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 msgr2=0x7eff1006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.358+0000 7eff156a7640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff1006ed10 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efef80098c0 tx=0x7efef8004730 comp rx=0 tx=0).stop 2023-08-25T10:05:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 shutdown_connections 2023-08-25T10:05:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7efef006a080 0x7efef006c540 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff101509d0 0x7eff1006f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7eff100c3a40 0x7eff1006ed10 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 --2- 172.21.15.71:0/2677196150 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff100c3660 0x7eff10158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 >> 172.21.15.71:0/2677196150 conn(0x7eff100654f0 msgr2=0x7eff100c8470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 shutdown_connections 2023-08-25T10:05:07.362 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:07.359+0000 7eff156a7640 1 -- 172.21.15.71:0/2677196150 wait complete. 2023-08-25T10:05:07.378 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:09.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:09.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.622+0000 7fa68300d640 1 Processor -- start 2023-08-25T10:05:09.624 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.622+0000 7fa68300d640 1 -- start start 2023-08-25T10:05:09.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa68300d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa67c152d80 0x7fa67c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.626 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa68300d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa67c0c4560 0x7fa67c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa68300d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c070cf0 0x7fa67c0c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa68300d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c06eb60 con 0x7fa67c0c4560 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa68300d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67c06ca30 con 0x7fa67c152d80 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa68300d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c06cf90 con 0x7fa67c070cf0 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa67bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa67c0c4560 0x7fa67c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa67bfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fa67c0c4560 0x7fa67c0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39906/0 (socket says 172.21.15.71:39906) 2023-08-25T10:05:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.624+0000 7fa67bfff640 1 -- 172.21.15.71:0/1356154295 learned_addr learned my addr 172.21.15.71:0/1356154295 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:09.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa6815aa640 1 --2- 172.21.15.71:0/1356154295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c070cf0 0x7fa67c0c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.628 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa680da9640 1 --2- 172.21.15.71:0/1356154295 >> v2:172.21.15.164:3300/0 conn(0x7fa67c152d80 0x7fa67c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa67bfff640 1 -- 172.21.15.71:0/1356154295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c070cf0 msgr2=0x7fa67c0c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:09.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa67bfff640 1 --2- 172.21.15.71:0/1356154295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c070cf0 0x7fa67c0c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa67bfff640 1 -- 172.21.15.71:0/1356154295 >> v2:172.21.15.164:3300/0 conn(0x7fa67c152d80 msgr2=0x7fa67c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:09.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa67bfff640 1 --2- 172.21.15.71:0/1356154295 >> v2:172.21.15.164:3300/0 conn(0x7fa67c152d80 0x7fa67c06e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.625+0000 7fa67bfff640 1 -- 172.21.15.71:0/1356154295 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c158fd0 con 0x7fa67c0c4560 2023-08-25T10:05:09.629 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.627+0000 7fa67bfff640 1 --2- 172.21.15.71:0/1356154295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c0c6080 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa67c06dee0 tx=0x7fa66802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ceaf61e77f4654 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:09.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.627+0000 7fa67b7fe640 1 -- 172.21.15.71:0/1356154295 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6680317e0 con 0x7fa67c0c4560 2023-08-25T10:05:09.630 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.627+0000 7fa67b7fe640 1 -- 172.21.15.71:0/1356154295 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa668031940 con 0x7fa67c0c4560 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.627+0000 7fa67b7fe640 1 -- 172.21.15.71:0/1356154295 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668031cd0 con 0x7fa67c0c4560 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 -- 172.21.15.71:0/1356154295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 msgr2=0x7fa67c0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 --2- 172.21.15.71:0/1356154295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c0c6080 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa67c06dee0 tx=0x7fa66802b880 comp rx=0 tx=0).stop 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 -- 172.21.15.71:0/1356154295 shutdown_connections 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 --2- 172.21.15.71:0/1356154295 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c070cf0 0x7fa67c0c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 --2- 172.21.15.71:0/1356154295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c0c6080 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.631 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 --2- 172.21.15.71:0/1356154295 >> v2:172.21.15.164:3300/0 conn(0x7fa67c152d80 0x7fa67c06e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 -- 172.21.15.71:0/1356154295 >> 172.21.15.71:0/1356154295 conn(0x7fa67c0654f0 msgr2=0x7fa67c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 -- 172.21.15.71:0/1356154295 shutdown_connections 2023-08-25T10:05:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.628+0000 7fa68300d640 1 -- 172.21.15.71:0/1356154295 wait complete. 2023-08-25T10:05:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.630+0000 7fa68300d640 1 Processor -- start 2023-08-25T10:05:09.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.631+0000 7fa68300d640 1 -- start start 2023-08-25T10:05:09.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa68300d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa67c070cf0 0x7fa67c1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa68300d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa68300d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c152d80 0x7fa67c14d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa68300d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c158c90 con 0x7fa67c0c4560 2023-08-25T10:05:09.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa68300d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c1ee260 con 0x7fa67c070cf0 2023-08-25T10:05:09.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa68300d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c1ee540 con 0x7fa67c152d80 2023-08-25T10:05:09.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.632+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.635 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:39918/0 (socket says 172.21.15.71:39918) 2023-08-25T10:05:09.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 -- 172.21.15.71:0/2570073966 learned_addr learned my addr 172.21.15.71:0/2570073966 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:09.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa680da9640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa67c070cf0 0x7fa67c1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa6815aa640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c152d80 0x7fa67c14d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 -- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c152d80 msgr2=0x7fa67c14d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:09.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c152d80 0x7fa67c14d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 -- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa67c070cf0 msgr2=0x7fa67c1461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:09.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa67c070cf0 0x7fa67c1461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.633+0000 7fa67bfff640 1 -- 172.21.15.71:0/2570073966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6680094d0 con 0x7fa67c0c4560 2023-08-25T10:05:09.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.634+0000 7fa67bfff640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c158200 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa6680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:09.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.634+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668032880 con 0x7fa67c0c4560 2023-08-25T10:05:09.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.634+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa668032a20 con 0x7fa67c0c4560 2023-08-25T10:05:09.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.634+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668032dd0 con 0x7fa67c0c4560 2023-08-25T10:05:09.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.634+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c1ee820 con 0x7fa67c0c4560 2023-08-25T10:05:09.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.635+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c0c9ff0 con 0x7fa67c0c4560 2023-08-25T10:05:09.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.636+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c159250 con 0x7fa67c0c4560 2023-08-25T10:05:09.639 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.637+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa66804e450 con 0x7fa67c0c4560 2023-08-25T10:05:09.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.638+0000 7fa6797fa640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa66006a0d0 0x7fa66006c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:09.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.639+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fa6680aab40 con 0x7fa67c0c4560 2023-08-25T10:05:09.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.639+0000 7fa680da9640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa66006a0d0 0x7fa66006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:09.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.640+0000 7fa680da9640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa66006a0d0 0x7fa66006c590 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa670006fd0 tx=0x7fa670008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:09.646 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.644+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa67c159250 con 0x7fa67c0c4560 2023-08-25T10:05:09.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.931+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa67c159250 con 0x7fa67c0c4560 2023-08-25T10:05:09.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.932+0000 7fa6797fa640 1 -- 172.21.15.71:0/2570073966 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fa67c159250 con 0x7fa67c0c4560 2023-08-25T10:05:09.935 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:09.935 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:09.935 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:09.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa66006a0d0 msgr2=0x7fa66006c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:09.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa66006a0d0 0x7fa66006c590 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa670006fd0 tx=0x7fa670008040 comp rx=0 tx=0).stop 2023-08-25T10:05:09.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 msgr2=0x7fa67c158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c158200 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa668000c00 tx=0x7fa6680046c0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 shutdown_connections 2023-08-25T10:05:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa66006a0d0 0x7fa66006c590 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa67c152d80 0x7fa67c14d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa67c0c4560 0x7fa67c158200 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 --2- 172.21.15.71:0/2570073966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa67c070cf0 0x7fa67c1461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:09.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 >> 172.21.15.71:0/2570073966 conn(0x7fa67c0654f0 msgr2=0x7fa67c0c9700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:09.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.935+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 shutdown_connections 2023-08-25T10:05:09.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:09.936+0000 7fa68300d640 1 -- 172.21.15.71:0/2570073966 wait complete. 2023-08-25T10:05:09.959 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:11.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.184+0000 7fafc3fff640 1 Processor -- start 2023-08-25T10:05:12.186 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.184+0000 7fafc3fff640 1 -- start start 2023-08-25T10:05:12.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafc4158d60 0x7fafc40cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc40cad40 0x7fafc40c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc3fff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc415c6c0 con 0x7fafc4152fc0 2023-08-25T10:05:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc3fff640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc415d260 con 0x7fafc4158d60 2023-08-25T10:05:12.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc3fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc41592c0 con 0x7fafc40cad40 2023-08-25T10:05:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc27fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafc4158d60 0x7fafc40cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.186+0000 7fafc37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc40cad40 0x7fafc40c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc40cad40 0x7fafc40c97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:57140/0 (socket says 172.21.15.71:57140) 2023-08-25T10:05:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc27fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafc4158d60 0x7fafc40cbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:43144/0 (socket says 172.21.15.71:43144) 2023-08-25T10:05:12.189 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc37fe640 1 -- 172.21.15.71:0/1092770340 learned_addr learned my addr 172.21.15.71:0/1092770340 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc2ffd640 1 --1- 172.21.15.71:0/1092770340 >> v1:172.21.15.164:6789/0 conn(0x7fafc4152fc0 0x7fafc415cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:42428/0 (socket says 172.21.15.71:42428) 2023-08-25T10:05:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc27fc640 1 -- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc40cad40 msgr2=0x7fafc40c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc27fc640 1 --2- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc40cad40 0x7fafc40c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc27fc640 1 -- 172.21.15.71:0/1092770340 >> v1:172.21.15.164:6789/0 conn(0x7fafc4152fc0 legacy=0x7fafc415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:12.190 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.187+0000 7fafc27fc640 1 -- 172.21.15.71:0/1092770340 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc41595a0 con 0x7fafc4158d60 2023-08-25T10:05:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.189+0000 7fafc27fc640 1 --2- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4158d60 0x7fafc40cbc60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097a0 tx=0x7fafb802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d406bb58bda6d5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.189+0000 7fafc1ffb640 1 -- 172.21.15.71:0/1092770340 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8030800 con 0x7fafc4158d60 2023-08-25T10:05:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.189+0000 7fafc1ffb640 1 -- 172.21.15.71:0/1092770340 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb8030960 con 0x7fafc4158d60 2023-08-25T10:05:12.191 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.189+0000 7fafc1ffb640 1 -- 172.21.15.71:0/1092770340 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8030cf0 con 0x7fafc4158d60 2023-08-25T10:05:12.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 -- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4158d60 msgr2=0x7fafc40cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:12.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 --2- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4158d60 0x7fafc40cbc60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097a0 tx=0x7fafb802b940 comp rx=0 tx=0).stop 2023-08-25T10:05:12.192 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 -- 172.21.15.71:0/1092770340 shutdown_connections 2023-08-25T10:05:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 --2- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc40cad40 0x7fafc40c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 --2- 172.21.15.71:0/1092770340 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4158d60 0x7fafc40cbc60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 -- 172.21.15.71:0/1092770340 >> 172.21.15.71:0/1092770340 conn(0x7fafc406a3a0 msgr2=0x7fafc4070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 -- 172.21.15.71:0/1092770340 shutdown_connections 2023-08-25T10:05:12.193 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.190+0000 7fafc3fff640 1 -- 172.21.15.71:0/1092770340 wait complete. 2023-08-25T10:05:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.192+0000 7fafc3fff640 1 Processor -- start 2023-08-25T10:05:12.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.193+0000 7fafc3fff640 1 -- start start 2023-08-25T10:05:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 0x7fafc414c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4152fc0 0x7fafc40c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc4158d60 0x7fafc41ee850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc3fff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc41eed90 con 0x7fafc40cad40 2023-08-25T10:05:12.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc3fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c6ea0 con 0x7fafc4152fc0 2023-08-25T10:05:12.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc3fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40c7180 con 0x7fafc4158d60 2023-08-25T10:05:12.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 0x7fafc414c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 0x7fafc414c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:57466/0 (socket says 172.21.15.71:57466) 2023-08-25T10:05:12.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.194+0000 7fafc2ffd640 1 -- 172.21.15.71:0/707950987 learned_addr learned my addr 172.21.15.71:0/707950987 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc27fc640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4152fc0 0x7fafc40c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc37fe640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc4158d60 0x7fafc41ee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:12.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc2ffd640 1 -- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc4158d60 msgr2=0x7fafc41ee850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc2ffd640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc4158d60 0x7fafc41ee850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc2ffd640 1 -- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4152fc0 msgr2=0x7fafc40c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc2ffd640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4152fc0 0x7fafc40c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc2ffd640 1 -- 172.21.15.71:0/707950987 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8009400 con 0x7fafc40cad40 2023-08-25T10:05:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc37fe640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc4158d60 0x7fafc41ee850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc2ffd640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 0x7fafc414c740 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c4d0 tx=0x7fafb400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4007480 con 0x7fafc40cad40 2023-08-25T10:05:12.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb40075e0 con 0x7fafc40cad40 2023-08-25T10:05:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc40c7460 con 0x7fafc40cad40 2023-08-25T10:05:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.195+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb40049e0 con 0x7fafc40cad40 2023-08-25T10:05:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.196+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc40c7a20 con 0x7fafc40cad40 2023-08-25T10:05:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.197+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc40c7c10 con 0x7fafc40cad40 2023-08-25T10:05:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.197+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fafb4004b40 con 0x7fafc40cad40 2023-08-25T10:05:12.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.198+0000 7fafab7fe640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fafb006a050 0x7fafb006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.198+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fafb40880e0 con 0x7fafc40cad40 2023-08-25T10:05:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.199+0000 7fafc27fc640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fafb006a050 0x7fafb006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.199+0000 7fafc27fc640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fafb006a050 0x7fafb006c510 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fafb8036000 tx=0x7fafb803b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:12.208 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.203+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fafb40531e0 con 0x7fafc40cad40 2023-08-25T10:05:12.450 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.448+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafc4159850 con 0x7fafc40cad40 2023-08-25T10:05:12.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.449+0000 7fafab7fe640 1 -- 172.21.15.71:0/707950987 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fafb4018090 con 0x7fafc40cad40 2023-08-25T10:05:12.451 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:12.452 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:12.452 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:12.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fafb006a050 msgr2=0x7fafb006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:12.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fafb006a050 0x7fafb006c510 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fafb8036000 tx=0x7fafb803b040 comp rx=0 tx=0).stop 2023-08-25T10:05:12.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 msgr2=0x7fafc414c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:12.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 0x7fafc414c740 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fafb400c4d0 tx=0x7fafb400c990 comp rx=0 tx=0).stop 2023-08-25T10:05:12.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 shutdown_connections 2023-08-25T10:05:12.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc4152fc0 0x7fafc40c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fafb006a050 0x7fafb006c510 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafc4158d60 0x7fafc41ee850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 --2- 172.21.15.71:0/707950987 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafc40cad40 0x7fafc414c740 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:12.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 >> 172.21.15.71:0/707950987 conn(0x7fafc406a3a0 msgr2=0x7fafc40c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:12.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 shutdown_connections 2023-08-25T10:05:12.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:12.452+0000 7fafc3fff640 1 -- 172.21.15.71:0/707950987 wait complete. 2023-08-25T10:05:12.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:14.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:14.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.734+0000 7f9a70ee0640 1 Processor -- start 2023-08-25T10:05:14.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.734+0000 7f9a70ee0640 1 -- start start 2023-08-25T10:05:14.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a70ee0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a6c1594d0 0x7f9a6c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a70ee0640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9a6c06cc70 0x7f9a6c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a70ee0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0c3940 0x7f9a6c14fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a70ee0640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c06a960 con 0x7f9a6c06cc70 2023-08-25T10:05:14.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a70ee0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c06ac20 con 0x7f9a6c1594d0 2023-08-25T10:05:14.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a70ee0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c06af00 con 0x7f9a6c0c3940 2023-08-25T10:05:14.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a69d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9a6c06cc70 0x7f9a6c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:14.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a69d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f9a6c06cc70 0x7f9a6c14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:57472/0 (socket says 172.21.15.71:57472) 2023-08-25T10:05:14.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.736+0000 7f9a69d9b640 1 -- 172.21.15.71:0/2213719563 learned_addr learned my addr 172.21.15.71:0/2213719563 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:14.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.737+0000 7f9a69d9b640 1 -- 172.21.15.71:0/2213719563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0c3940 msgr2=0x7f9a6c14fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:14.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.737+0000 7f9a69d9b640 1 --2- 172.21.15.71:0/2213719563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0c3940 0x7f9a6c14fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.737+0000 7f9a69d9b640 1 -- 172.21.15.71:0/2213719563 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c1594d0 msgr2=0x7f9a6c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:14.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.737+0000 7f9a69d9b640 1 --2- 172.21.15.71:0/2213719563 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c1594d0 0x7f9a6c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.737+0000 7f9a69d9b640 1 -- 172.21.15.71:0/2213719563 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0c2e60 con 0x7f9a6c06cc70 2023-08-25T10:05:14.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.738+0000 7f9a69d9b640 1 --2- 172.21.15.71:0/2213719563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c14f850 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c071d10 tx=0x7f9a6002e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58844e3cca493607 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.738+0000 7f9a6959a640 1 -- 172.21.15.71:0/2213719563 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6002ee80 con 0x7f9a6c06cc70 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.738+0000 7f9a6959a640 1 -- 172.21.15.71:0/2213719563 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60036440 con 0x7f9a6c06cc70 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a6959a640 1 -- 172.21.15.71:0/2213719563 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a600367b0 con 0x7f9a6c06cc70 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 -- 172.21.15.71:0/2213719563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c06cc70 msgr2=0x7f9a6c14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/2213719563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c14f850 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c071d10 tx=0x7f9a6002e0d0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 -- 172.21.15.71:0/2213719563 shutdown_connections 2023-08-25T10:05:14.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/2213719563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c0c3940 0x7f9a6c14fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/2213719563 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c14f850 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/2213719563 >> v2:172.21.15.164:3300/0 conn(0x7f9a6c1594d0 0x7f9a6c0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.739+0000 7f9a70ee0640 1 -- 172.21.15.71:0/2213719563 >> 172.21.15.71:0/2213719563 conn(0x7f9a6c0654f0 msgr2=0x7f9a6c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.740+0000 7f9a70ee0640 1 -- 172.21.15.71:0/2213719563 shutdown_connections 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.740+0000 7f9a70ee0640 1 -- 172.21.15.71:0/2213719563 wait complete. 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.742+0000 7f9a70ee0640 1 Processor -- start 2023-08-25T10:05:14.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.742+0000 7f9a70ee0640 1 -- start start 2023-08-25T10:05:14.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a70ee0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c1df920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a70ee0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 0x7f9a6c140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a70ee0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c1594d0 0x7f9a6c141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a70ee0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c1ef170 con 0x7f9a6c0c3940 2023-08-25T10:05:14.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a70ee0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c156f60 con 0x7f9a6c06cc70 2023-08-25T10:05:14.750 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a70ee0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c157240 con 0x7f9a6c1594d0 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.743+0000 7f9a69d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 0x7f9a6c140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 0x7f9a6c140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:57488/0 (socket says 172.21.15.71:57488) 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 -- 172.21.15.71:0/153735452 learned_addr learned my addr 172.21.15.71:0/153735452 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a6ad9d640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c1594d0 0x7f9a6c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a6a59c640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c1df920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 -- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c1594d0 msgr2=0x7f9a6c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c1594d0 0x7f9a6c141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 -- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c06cc70 msgr2=0x7f9a6c1df920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c1df920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a69d9b640 1 -- 172.21.15.71:0/153735452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6000b770 con 0x7f9a6c0c3940 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.744+0000 7f9a6a59c640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c1df920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.745+0000 7f9a69d9b640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 0x7f9a6c140c10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9a6002e110 tx=0x7f9a60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.745+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60034950 con 0x7f9a6c0c3940 2023-08-25T10:05:14.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.745+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60034ab0 con 0x7f9a6c0c3940 2023-08-25T10:05:14.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.745+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60034e60 con 0x7f9a6c0c3940 2023-08-25T10:05:14.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.745+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c1e39c0 con 0x7f9a6c0c3940 2023-08-25T10:05:14.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.745+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c0c58b0 con 0x7f9a6c0c3940 2023-08-25T10:05:14.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.746+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c1e3b50 con 0x7f9a6c0c3940 2023-08-25T10:05:14.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.748+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9a60041480 con 0x7f9a6c0c3940 2023-08-25T10:05:14.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.748+0000 7f9a5affd640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:14.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.749+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9a600abcc0 con 0x7f9a6c0c3940 2023-08-25T10:05:14.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.749+0000 7f9a6a59c640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:14.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.750+0000 7f9a6a59c640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006fd0 tx=0x7f9a5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:14.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.753+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9a600b2080 con 0x7f9a6c0c3940 2023-08-25T10:05:14.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.979+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6c069160 con 0x7f9a6c0c3940 2023-08-25T10:05:14.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.981+0000 7f9a5affd640 1 -- 172.21.15.71:0/153735452 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f9a60076ee0 con 0x7f9a6c0c3940 2023-08-25T10:05:14.984 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:14.984 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:14.985 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:14.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 msgr2=0x7f9a4c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:14.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006fd0 tx=0x7f9a5c008040 comp rx=0 tx=0).stop 2023-08-25T10:05:14.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 msgr2=0x7f9a6c140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:14.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 0x7f9a6c140c10 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9a6002e110 tx=0x7f9a60004730 comp rx=0 tx=0).stop 2023-08-25T10:05:14.986 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 shutdown_connections 2023-08-25T10:05:14.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a6c1594d0 0x7f9a6c141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a6c0c3940 0x7f9a6c140c10 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 --2- 172.21.15.71:0/153735452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a6c06cc70 0x7f9a6c1df920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:14.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 >> 172.21.15.71:0/153735452 conn(0x7f9a6c0654f0 msgr2=0x7f9a6c0c2340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:14.987 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 shutdown_connections 2023-08-25T10:05:14.988 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:14.984+0000 7f9a70ee0640 1 -- 172.21.15.71:0/153735452 wait complete. 2023-08-25T10:05:15.005 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:17.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:17.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.249+0000 7f36dcda4640 1 Processor -- start 2023-08-25T10:05:17.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.249+0000 7f36dcda4640 1 -- start start 2023-08-25T10:05:17.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36dcda4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f36d80c4560 0x7f36d806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:17.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36dcda4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d806eb60 con 0x7f36d8152d80 2023-08-25T10:05:17.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36dcda4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d806ca30 con 0x7f36d8070cf0 2023-08-25T10:05:17.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36dcda4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d806cf90 con 0x7f36d80c4560 2023-08-25T10:05:17.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36d5d9b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f36d80c4560 0x7f36d806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:17.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36d659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f36d8152d80 0x7f36d80c6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37062/0 (socket says 172.21.15.71:37062) 2023-08-25T10:05:17.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.251+0000 7f36d659c640 1 -- 172.21.15.71:0/2192889406 learned_addr learned my addr 172.21.15.71:0/2192889406 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:17.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.252+0000 7f36d5d9b640 1 -- 172.21.15.71:0/2192889406 >> v1:172.21.15.164:6789/0 conn(0x7f36d8070cf0 legacy=0x7f36d81637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.252+0000 7f36d5d9b640 1 -- 172.21.15.71:0/2192889406 >> v1:172.21.15.71:6789/0 conn(0x7f36d8152d80 legacy=0x7f36d80c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.252+0000 7f36d5d9b640 1 -- 172.21.15.71:0/2192889406 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8158fd0 con 0x7f36d80c4560 2023-08-25T10:05:17.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.252+0000 7f36d559a640 1 -- 172.21.15.71:0/2192889406 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265031759 0 0) 0x7f36d806eb60 con 0x7f36d8152d80 2023-08-25T10:05:17.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.253+0000 7f36d5d9b640 1 --2- 172.21.15.71:0/2192889406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d80c4560 0x7f36d806e620 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0097a0 tx=0x7f36cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf39c705d7a80ff server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:17.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.254+0000 7f36d559a640 1 -- 172.21.15.71:0/2192889406 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc031830 con 0x7f36d80c4560 2023-08-25T10:05:17.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.254+0000 7f36d559a640 1 -- 172.21.15.71:0/2192889406 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc031990 con 0x7f36d80c4560 2023-08-25T10:05:17.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.254+0000 7f36d559a640 1 -- 172.21.15.71:0/2192889406 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc031d20 con 0x7f36d80c4560 2023-08-25T10:05:17.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 -- 172.21.15.71:0/2192889406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d80c4560 msgr2=0x7f36d806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 --2- 172.21.15.71:0/2192889406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d80c4560 0x7f36d806e620 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0097a0 tx=0x7f36cc02b8d0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 -- 172.21.15.71:0/2192889406 shutdown_connections 2023-08-25T10:05:17.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 --2- 172.21.15.71:0/2192889406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d80c4560 0x7f36d806e620 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 -- 172.21.15.71:0/2192889406 >> 172.21.15.71:0/2192889406 conn(0x7f36d80654f0 msgr2=0x7f36d8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:17.261 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 -- 172.21.15.71:0/2192889406 shutdown_connections 2023-08-25T10:05:17.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.255+0000 7f36dcda4640 1 -- 172.21.15.71:0/2192889406 wait complete. 2023-08-25T10:05:17.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.257+0000 7f36dcda4640 1 Processor -- start 2023-08-25T10:05:17.262 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.257+0000 7f36dcda4640 1 -- start start 2023-08-25T10:05:17.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36dcda4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 0x7f36d8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:17.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36dcda4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d80c4560 0x7f36d80c20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:17.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36dcda4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d8152d80 0x7f36d8156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36dcda4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d814c880 con 0x7f36d8070cf0 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36dcda4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8146ca0 con 0x7f36d80c4560 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36dcda4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8145f40 con 0x7f36d8152d80 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36d659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 0x7f36d8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36d659c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 0x7f36d8158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:57504/0 (socket says 172.21.15.71:57504) 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36d659c640 1 -- 172.21.15.71:0/3393519209 learned_addr learned my addr 172.21.15.71:0/3393519209 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:17.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36d5d9b640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d80c4560 0x7f36d80c20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36d6d9d640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d8152d80 0x7f36d8156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.258+0000 7f36d659c640 1 -- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d8152d80 msgr2=0x7f36d8156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36d659c640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d8152d80 0x7f36d8156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36d659c640 1 -- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d80c4560 msgr2=0x7f36d80c20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36d659c640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d80c4560 0x7f36d80c20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36d659c640 1 -- 172.21.15.71:0/3393519209 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc009400 con 0x7f36d8070cf0 2023-08-25T10:05:17.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36d6d9d640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d8152d80 0x7f36d8156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36d659c640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 0x7f36d8158440 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c4a0 tx=0x7f36c800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c80074e0 con 0x7f36d8070cf0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c8007640 con 0x7f36d8070cf0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d814b5c0 con 0x7f36d8070cf0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8004a00 con 0x7f36d8070cf0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.259+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d814b2e0 con 0x7f36d8070cf0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.261+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d8145da0 con 0x7f36d8070cf0 2023-08-25T10:05:17.267 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.262+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f36c8004b60 con 0x7f36d8070cf0 2023-08-25T10:05:17.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.263+0000 7f36c6ffd640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36b806a050 0x7f36b806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:17.268 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.263+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f36c8087c00 con 0x7f36d8070cf0 2023-08-25T10:05:17.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.263+0000 7f36d5d9b640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36b806a050 0x7f36b806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:17.269 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.264+0000 7f36d5d9b640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36b806a050 0x7f36b806c510 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f36cc036000 tx=0x7f36cc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:17.271 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.269+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f36d8145da0 con 0x7f36d8070cf0 2023-08-25T10:05:17.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.545+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36d8145da0 con 0x7f36d8070cf0 2023-08-25T10:05:17.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.546+0000 7f36c6ffd640 1 -- 172.21.15.71:0/3393519209 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f36d8145da0 con 0x7f36d8070cf0 2023-08-25T10:05:17.549 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:17.549 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:17.549 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:17.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.549+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36b806a050 msgr2=0x7f36b806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.549+0000 7f36dcda4640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36b806a050 0x7f36b806c510 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f36cc036000 tx=0x7f36cc03c040 comp rx=0 tx=0).stop 2023-08-25T10:05:17.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.549+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 msgr2=0x7f36d8158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:17.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.549+0000 7f36dcda4640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 0x7f36d8158440 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c4a0 tx=0x7f36c800c960 comp rx=0 tx=0).stop 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 shutdown_connections 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36d80c4560 0x7f36d80c20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36b806a050 0x7f36b806c510 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36d8152d80 0x7f36d8156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 --2- 172.21.15.71:0/3393519209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f36d8070cf0 0x7f36d8158440 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 >> 172.21.15.71:0/3393519209 conn(0x7f36d80654f0 msgr2=0x7f36d80658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:17.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 shutdown_connections 2023-08-25T10:05:17.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:17.550+0000 7f36dcda4640 1 -- 172.21.15.71:0/3393519209 wait complete. 2023-08-25T10:05:17.576 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:05:17.577 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:05:19.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:19.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.821+0000 7fc042f55640 1 Processor -- start 2023-08-25T10:05:19.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.821+0000 7fc042f55640 1 -- start start 2023-08-25T10:05:19.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.823+0000 7fc042f55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c1594d0 0x7fc03c0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:19.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.823+0000 7fc042f55640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc03c06cc70 0x7fc03c14f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:19.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.823+0000 7fc042f55640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c06a960 con 0x7fc03c0c3940 2023-08-25T10:05:19.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.823+0000 7fc042f55640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03c06ac20 con 0x7fc03c06cc70 2023-08-25T10:05:19.826 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.823+0000 7fc042f55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c06af00 con 0x7fc03c1594d0 2023-08-25T10:05:19.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.823+0000 7fc033fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc03c06cc70 0x7fc03c14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:19.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc040cf1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c1594d0 0x7fc03c0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:19.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc033fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc03c06cc70 0x7fc03c14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:43200/0 (socket says 172.21.15.71:43200) 2023-08-25T10:05:19.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc033fff640 1 -- 172.21.15.71:0/4040743595 learned_addr learned my addr 172.21.15.71:0/4040743595 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc033fff640 1 -- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c1594d0 msgr2=0x7fc03c0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc033fff640 1 --2- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c1594d0 0x7fc03c0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc033fff640 1 -- 172.21.15.71:0/4040743595 >> v1:172.21.15.164:6789/0 conn(0x7fc03c0c3940 legacy=0x7fc03c1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:19.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.824+0000 7fc033fff640 1 -- 172.21.15.71:0/4040743595 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0c2e60 con 0x7fc03c06cc70 2023-08-25T10:05:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.826+0000 7fc033fff640 1 --2- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c14f850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00bb10 tx=0x7fc02c02e150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d24a703ae0fb2b4 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.826+0000 7fc0337fe640 1 -- 172.21.15.71:0/4040743595 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c036440 con 0x7fc03c06cc70 2023-08-25T10:05:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc0337fe640 1 -- 172.21.15.71:0/4040743595 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c0365a0 con 0x7fc03c06cc70 2023-08-25T10:05:19.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc0337fe640 1 -- 172.21.15.71:0/4040743595 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c036930 con 0x7fc03c06cc70 2023-08-25T10:05:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 -- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 msgr2=0x7fc03c14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 --2- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c14f850 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00bb10 tx=0x7fc02c02e150 comp rx=0 tx=0).stop 2023-08-25T10:05:19.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 -- 172.21.15.71:0/4040743595 shutdown_connections 2023-08-25T10:05:19.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 --2- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c14f850 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:19.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 --2- 172.21.15.71:0/4040743595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c1594d0 0x7fc03c0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:19.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 -- 172.21.15.71:0/4040743595 >> 172.21.15.71:0/4040743595 conn(0x7fc03c0654f0 msgr2=0x7fc03c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:19.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.827+0000 7fc042f55640 1 -- 172.21.15.71:0/4040743595 shutdown_connections 2023-08-25T10:05:19.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.828+0000 7fc042f55640 1 -- 172.21.15.71:0/4040743595 wait complete. 2023-08-25T10:05:19.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.829+0000 7fc042f55640 1 Processor -- start 2023-08-25T10:05:19.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.830+0000 7fc042f55640 1 -- start start 2023-08-25T10:05:19.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.830+0000 7fc042f55640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c1e32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc042f55640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c0c3940 0x7fc03c1e37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc042f55640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 0x7fc03c1e1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc042f55640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0c8450 con 0x7fc03c1594d0 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc042f55640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c0c9760 con 0x7fc03c06cc70 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc042f55640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c1eee40 con 0x7fc03c0c3940 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 0x7fc03c1e1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:19.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 0x7fc03c1e1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:57508/0 (socket says 172.21.15.71:57508) 2023-08-25T10:05:19.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 -- 172.21.15.71:0/1590589360 learned_addr learned my addr 172.21.15.71:0/1590589360 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:19.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc040cf1640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c1e32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc033fff640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c0c3940 0x7fc03c1e37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 -- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c0c3940 msgr2=0x7fc03c1e37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c0c3940 0x7fc03c1e37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 -- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 msgr2=0x7fc03c1e32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c1e32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.831+0000 7fc0414f2640 1 -- 172.21.15.71:0/1590589360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c00b770 con 0x7fc03c1594d0 2023-08-25T10:05:19.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.832+0000 7fc0414f2640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 0x7fc03c1e1cd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c6b0 tx=0x7fc03400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:19.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.832+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034003b70 con 0x7fc03c1594d0 2023-08-25T10:05:19.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.832+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc034003cf0 con 0x7fc03c1594d0 2023-08-25T10:05:19.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.832+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034003ea0 con 0x7fc03c1594d0 2023-08-25T10:05:19.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.832+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c0c9c40 con 0x7fc03c1594d0 2023-08-25T10:05:19.837 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.832+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c157e80 con 0x7fc03c1594d0 2023-08-25T10:05:19.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.834+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c070d60 con 0x7fc03c1594d0 2023-08-25T10:05:19.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.834+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc034005310 con 0x7fc03c1594d0 2023-08-25T10:05:19.838 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.835+0000 7fc0317fa640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc02806a050 0x7fc02806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:19.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.836+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc0340888a0 con 0x7fc03c1594d0 2023-08-25T10:05:19.839 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.836+0000 7fc040cf1640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc02806a050 0x7fc02806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:19.840 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.837+0000 7fc040cf1640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc02806a050 0x7fc02806c510 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc038009ad0 tx=0x7fc038008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:19.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:19.841+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc034053a30 con 0x7fc03c1594d0 2023-08-25T10:05:20.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.140+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc03c069160 con 0x7fc03c1594d0 2023-08-25T10:05:20.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.141+0000 7fc0317fa640 1 -- 172.21.15.71:0/1590589360 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fc034057050 con 0x7fc03c1594d0 2023-08-25T10:05:20.144 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:20.145 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:20.145 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:20.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.143+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc02806a050 msgr2=0x7fc02806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:20.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc02806a050 0x7fc02806c510 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc038009ad0 tx=0x7fc038008040 comp rx=0 tx=0).stop 2023-08-25T10:05:20.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 msgr2=0x7fc03c1e1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:20.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 0x7fc03c1e1cd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c6b0 tx=0x7fc03400cb70 comp rx=0 tx=0).stop 2023-08-25T10:05:20.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 shutdown_connections 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc02806a050 0x7fc02806c510 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03c1594d0 0x7fc03c1e1cd0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc03c0c3940 0x7fc03c1e37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 --2- 172.21.15.71:0/1590589360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc03c06cc70 0x7fc03c1e32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 >> 172.21.15.71:0/1590589360 conn(0x7fc03c0654f0 msgr2=0x7fc03c14d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 shutdown_connections 2023-08-25T10:05:20.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:20.144+0000 7fc042f55640 1 -- 172.21.15.71:0/1590589360 wait complete. 2023-08-25T10:05:20.168 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:22.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:22.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.386+0000 7ff7b7fff640 1 Processor -- start 2023-08-25T10:05:22.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.386+0000 7ff7b7fff640 1 -- start start 2023-08-25T10:05:22.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.387+0000 7ff7b7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 0x7ff7b81576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.387+0000 7ff7b7fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b8157bf0 con 0x7ff7b81542d0 2023-08-25T10:05:22.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.387+0000 7ff7b7fff640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b806a130 con 0x7ff7b8069850 2023-08-25T10:05:22.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.387+0000 7ff7b7fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b80c4b60 con 0x7ff7b80c56c0 2023-08-25T10:05:22.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b6ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff7b81542d0 0x7ff7b80c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52716/0 (socket says 172.21.15.71:52716) 2023-08-25T10:05:22.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b6ffd640 1 -- 172.21.15.71:0/48051523 learned_addr learned my addr 172.21.15.71:0/48051523 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:22.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b77fe640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 0x7ff7b81576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629799087 0 0) 0x7ff7a8001110 con 0x7ff7b81542d0 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a4003520 con 0x7ff7b81542d0 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 528941899 0 0) 0x7ff7a4003520 con 0x7ff7b81542d0 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 msgr2=0x7ff7b81576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b67fc640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 0x7ff7b81576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.388+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 >> v1:172.21.15.164:6789/0 conn(0x7ff7b8069850 legacy=0x7ff7b80c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b81546b0 con 0x7ff7b81542d0 2023-08-25T10:05:22.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7ff7a8001800 con 0x7ff7b81542d0 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 >> v1:172.21.15.71:6789/0 conn(0x7ff7b81542d0 legacy=0x7ff7b80c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7a4003a10 0x7ff7a4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7a40047c0 0x7ff7a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 0x7ff7a4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b80c4b60 con 0x7ff7a4003a10 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a400bb70 con 0x7ff7a40047c0 2023-08-25T10:05:22.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a400bd30 con 0x7ff7b80c56c0 2023-08-25T10:05:22.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7a4003a10 0x7ff7a4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7affff640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7a40047c0 0x7ff7a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b6ffd640 1 -- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 msgr2=0x7ff7a4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:22.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b77fe640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 0x7ff7a4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7b80c56c0 0x7ff7a4005230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b6ffd640 1 -- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7a40047c0 msgr2=0x7ff7a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:22.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7a40047c0 0x7ff7a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.397 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7b6ffd640 1 -- 172.21.15.71:0/48051523 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a4003520 con 0x7ff7a4003a10 2023-08-25T10:05:22.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.389+0000 7ff7affff640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7a40047c0 0x7ff7a4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:22.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.390+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7a4003a10 0x7ff7a4003e30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800c190 tx=0x7ff7a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d20d8a1da5a366b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:22.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8005ad0 con 0x7ff7a4003a10 2023-08-25T10:05:22.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a8005c50 con 0x7ff7a4003a10 2023-08-25T10:05:22.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b67fc640 1 -- 172.21.15.71:0/48051523 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a802c7e0 con 0x7ff7a4003a10 2023-08-25T10:05:22.402 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b7fff640 1 -- 172.21.15.71:0/48051523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7a4003a10 msgr2=0x7ff7a4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/48051523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7a4003a10 0x7ff7a4003e30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800c190 tx=0x7ff7a40057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b77fe640 1 -- 172.21.15.71:0/48051523 reap_dead start 2023-08-25T10:05:22.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b7fff640 1 -- 172.21.15.71:0/48051523 shutdown_connections 2023-08-25T10:05:22.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b7fff640 1 -- 172.21.15.71:0/48051523 >> 172.21.15.71:0/48051523 conn(0x7ff7b806c280 msgr2=0x7ff7b806c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:22.403 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b7fff640 1 -- 172.21.15.71:0/48051523 shutdown_connections 2023-08-25T10:05:22.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.391+0000 7ff7b7fff640 1 -- 172.21.15.71:0/48051523 wait complete. 2023-08-25T10:05:22.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.393+0000 7ff7b7fff640 1 Processor -- start 2023-08-25T10:05:22.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.393+0000 7ff7b7fff640 1 -- start start 2023-08-25T10:05:22.404 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.394+0000 7ff7b7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7a4003a10 0x7ff7b814b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.394+0000 7ff7b7fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 0x7ff7b80c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.394+0000 7ff7b7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b8154aa0 0x7ff7b80c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.394+0000 7ff7b7fff640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b8157220 con 0x7ff7b80c3610 2023-08-25T10:05:22.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.394+0000 7ff7b7fff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b8157d90 con 0x7ff7b8154aa0 2023-08-25T10:05:22.405 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.394+0000 7ff7b7fff640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b8157fc0 con 0x7ff7a4003a10 2023-08-25T10:05:22.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 0x7ff7b80c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 0x7ff7b80c1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60600/0 (socket says 172.21.15.71:60600) 2023-08-25T10:05:22.406 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 -- 172.21.15.71:0/2654683260 learned_addr learned my addr 172.21.15.71:0/2654683260 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:22.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7a4003a10 0x7ff7b814b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 -- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7a4003a10 msgr2=0x7ff7b814b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7a4003a10 0x7ff7b814b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.407 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 -- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b8154aa0 msgr2=0x7ff7b80c2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7b77fe640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b8154aa0 0x7ff7b80c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b8154aa0 0x7ff7b80c2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 -- 172.21.15.71:0/2654683260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b80c1870 con 0x7ff7b80c3610 2023-08-25T10:05:22.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7affff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 0x7ff7b80c1d00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000d7a0 tx=0x7ff7a000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:22.408 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.395+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0017070 con 0x7ff7b80c3610 2023-08-25T10:05:22.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.396+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a0005070 con 0x7ff7b80c3610 2023-08-25T10:05:22.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.396+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0005420 con 0x7ff7b80c3610 2023-08-25T10:05:22.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.396+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b8071650 con 0x7ff7b80c3610 2023-08-25T10:05:22.409 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.396+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b8156ba0 con 0x7ff7b80c3610 2023-08-25T10:05:22.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.397+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b8156720 con 0x7ff7b80c3610 2023-08-25T10:05:22.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.398+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff7a0007030 con 0x7ff7b80c3610 2023-08-25T10:05:22.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.399+0000 7ff7af7fe640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79806a050 0x7ff79806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:22.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.399+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff7a0088310 con 0x7ff7b80c3610 2023-08-25T10:05:22.410 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.399+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79806a050 0x7ff79806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:22.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.400+0000 7ff7b6ffd640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79806a050 0x7ff79806c510 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8034000 tx=0x7ff7a8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:22.411 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.403+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff7b8156720 con 0x7ff7b80c3610 2023-08-25T10:05:22.632 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.629+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7b8156720 con 0x7ff7b80c3610 2023-08-25T10:05:22.634 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.630+0000 7ff7af7fe640 1 -- 172.21.15.71:0/2654683260 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7ff7b8156720 con 0x7ff7b80c3610 2023-08-25T10:05:22.635 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:22.636 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:22.636 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79806a050 msgr2=0x7ff79806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79806a050 0x7ff79806c510 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8034000 tx=0x7ff7a8009030 comp rx=0 tx=0).stop 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 msgr2=0x7ff7b80c1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 0x7ff7b80c1d00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000d7a0 tx=0x7ff7a000dc60 comp rx=0 tx=0).stop 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 shutdown_connections 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7b8154aa0 0x7ff7b80c2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff79806a050 0x7ff79806c510 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.637 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff7b80c3610 0x7ff7b80c1d00 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 --2- 172.21.15.71:0/2654683260 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7a4003a10 0x7ff7b814b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 >> 172.21.15.71:0/2654683260 conn(0x7ff7b806c280 msgr2=0x7ff7b8151eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 shutdown_connections 2023-08-25T10:05:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:22.633+0000 7ff7b7fff640 1 -- 172.21.15.71:0/2654683260 wait complete. 2023-08-25T10:05:22.653 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:24.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:24.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.873+0000 7f3356fab640 1 Processor -- start 2023-08-25T10:05:24.876 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.873+0000 7f3356fab640 1 -- start start 2023-08-25T10:05:24.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.876+0000 7f3356fab640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f33500c4510 0x7f33500cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.876+0000 7f3356fab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33501590c0 con 0x7f33500cec00 2023-08-25T10:05:24.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.876+0000 7f3356fab640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3350159380 con 0x7f3350153310 2023-08-25T10:05:24.879 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.876+0000 7f3356fab640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3350159660 con 0x7f33500c4510 2023-08-25T10:05:24.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334ffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f33500cec00 0x7f33500c3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52730/0 (socket says 172.21.15.71:52730) 2023-08-25T10:05:24.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334ffff640 1 -- 172.21.15.71:0/968419438 learned_addr learned my addr 172.21.15.71:0/968419438 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:24.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f3355548640 1 --2- 172.21.15.71:0/968419438 >> v2:172.21.15.164:3300/0 conn(0x7f33500c4510 0x7f33500cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354588278 0 0) 0x7f33501590c0 con 0x7f33500cec00 2023-08-25T10:05:24.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3340003520 con 0x7f33500cec00 2023-08-25T10:05:24.882 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577327563 0 0) 0x7f3340003520 con 0x7f33500cec00 2023-08-25T10:05:24.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 >> v2:172.21.15.164:3300/0 conn(0x7f33500c4510 msgr2=0x7f33500cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:24.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 --2- 172.21.15.71:0/968419438 >> v2:172.21.15.164:3300/0 conn(0x7f33500c4510 0x7f33500cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 >> v1:172.21.15.164:6789/0 conn(0x7f3350153310 legacy=0x7f335006a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:24.883 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.877+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3350159940 con 0x7f33500cec00 2023-08-25T10:05:24.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f333c001670 con 0x7f33500cec00 2023-08-25T10:05:24.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 >> v1:172.21.15.71:6789/0 conn(0x7f33500cec00 legacy=0x7f33500c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:24.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039e0 0x7f3340003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33400047c0 0x7f3340004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33400051d0 0x7f334000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3350159660 con 0x7f33400051d0 2023-08-25T10:05:24.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334000c110 con 0x7f33400039e0 2023-08-25T10:05:24.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334000c2d0 con 0x7f33400047c0 2023-08-25T10:05:24.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3355548640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33400051d0 0x7f334000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f334ffff640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33400047c0 0x7f3340004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3354d47640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039e0 0x7f3340003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3355548640 1 -- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33400047c0 msgr2=0x7f3340004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:24.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3355548640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33400047c0 0x7f3340004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3355548640 1 -- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039e0 msgr2=0x7f3340003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:24.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3355548640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400039e0 0x7f3340003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.888 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.878+0000 7f3355548640 1 -- 172.21.15.71:0/968419438 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340003520 con 0x7f33400051d0 2023-08-25T10:05:24.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.879+0000 7f3355548640 1 -- 172.21.15.71:0/968419438 reap_dead start 2023-08-25T10:05:24.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.880+0000 7f3355548640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33400051d0 0x7f334000b980 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3348007360 tx=0x7f33400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eee6e423359b12b server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:24.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.880+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334802ccf0 con 0x7f33400051d0 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.880+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334802ce90 con 0x7f33400051d0 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.880+0000 7f334f7fe640 1 -- 172.21.15.71:0/968419438 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348003bd0 con 0x7f33400051d0 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 -- 172.21.15.71:0/968419438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33400051d0 msgr2=0x7f334000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33400051d0 0x7f334000b980 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3348007360 tx=0x7f33400057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 -- 172.21.15.71:0/968419438 shutdown_connections 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 --2- 172.21.15.71:0/968419438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f33400051d0 0x7f334000b980 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 -- 172.21.15.71:0/968419438 >> 172.21.15.71:0/968419438 conn(0x7f3350065910 msgr2=0x7f3350065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:24.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 -- 172.21.15.71:0/968419438 shutdown_connections 2023-08-25T10:05:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.881+0000 7f3356fab640 1 -- 172.21.15.71:0/968419438 wait complete. 2023-08-25T10:05:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.883+0000 7f3356fab640 1 Processor -- start 2023-08-25T10:05:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.883+0000 7f3356fab640 1 -- start start 2023-08-25T10:05:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3356fab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400051d0 0x7f33500c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3356fab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f335014d7d0 0x7f33500c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3356fab640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 0x7f33500c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3356fab640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33500c8950 con 0x7f335014dc10 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3356fab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3350152ac0 con 0x7f33400051d0 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3356fab640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33500c80f0 con 0x7f335014d7d0 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3355548640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 0x7f33500c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3355548640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 0x7f33500c7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60620/0 (socket says 172.21.15.71:60620) 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3355548640 1 -- 172.21.15.71:0/1914371072 learned_addr learned my addr 172.21.15.71:0/1914371072 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f3354d47640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400051d0 0x7f33500c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.893 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.884+0000 7f334ffff640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f335014d7d0 0x7f33500c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f3355548640 1 -- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f335014d7d0 msgr2=0x7f33500c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f3355548640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f335014d7d0 0x7f33500c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f3355548640 1 -- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400051d0 msgr2=0x7f33500c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f3355548640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400051d0 0x7f33500c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f3355548640 1 -- 172.21.15.71:0/1914371072 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33500c8390 con 0x7f335014dc10 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f3355548640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 0x7f33500c7560 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f334802c760 tx=0x7f3348004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:24.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.885+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348038070 con 0x7f335014dc10 2023-08-25T10:05:24.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.886+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348039040 con 0x7f335014dc10 2023-08-25T10:05:24.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.886+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348004730 con 0x7f335014dc10 2023-08-25T10:05:24.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.886+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33500c8670 con 0x7f335014dc10 2023-08-25T10:05:24.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.886+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33500c69b0 con 0x7f335014dc10 2023-08-25T10:05:24.895 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.887+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335014e3d0 con 0x7f335014dc10 2023-08-25T10:05:24.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.888+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3348034440 con 0x7f335014dc10 2023-08-25T10:05:24.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.889+0000 7f334d7fa640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f333406a050 0x7f333406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:24.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.889+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f33480aa610 con 0x7f335014dc10 2023-08-25T10:05:24.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.890+0000 7f3354d47640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f333406a050 0x7f333406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:24.896 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.891+0000 7f3354d47640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f333406a050 0x7f333406c510 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3344003db0 tx=0x7f3344009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:24.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:24.897+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f335014e3d0 con 0x7f335014dc10 2023-08-25T10:05:25.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.190+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f335014e7b0 con 0x7f335014dc10 2023-08-25T10:05:25.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.192+0000 7f334d7fa640 1 -- 172.21.15.71:0/1914371072 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f335014e7b0 con 0x7f335014dc10 2023-08-25T10:05:25.196 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:25.196 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:25.197 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:25.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f333406a050 msgr2=0x7f333406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:25.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f333406a050 0x7f333406c510 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3344003db0 tx=0x7f3344009040 comp rx=0 tx=0).stop 2023-08-25T10:05:25.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 msgr2=0x7f33500c7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:25.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 0x7f33500c7560 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f334802c760 tx=0x7f3348004ab0 comp rx=0 tx=0).stop 2023-08-25T10:05:25.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 shutdown_connections 2023-08-25T10:05:25.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f333406a050 0x7f333406c510 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:25.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f335014dc10 0x7f33500c7560 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:25.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f335014d7d0 0x7f33500c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:25.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 --2- 172.21.15.71:0/1914371072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33400051d0 0x7f33500c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:25.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 >> 172.21.15.71:0/1914371072 conn(0x7f3350065910 msgr2=0x7f3350157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:25.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 shutdown_connections 2023-08-25T10:05:25.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:25.195+0000 7f3356fab640 1 -- 172.21.15.71:0/1914371072 wait complete. 2023-08-25T10:05:25.219 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:27.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:05:27.210 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:05:27.223 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:27.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.486+0000 7f9a6e5cb640 1 Processor -- start 2023-08-25T10:05:27.489 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.486+0000 7f9a6e5cb640 1 -- start start 2023-08-25T10:05:27.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.487+0000 7f9a6e5cb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 0x7f9a680cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.487+0000 7f9a6e5cb640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6815c6c0 con 0x7f9a68158d60 2023-08-25T10:05:27.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.487+0000 7f9a6e5cb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6815d260 con 0x7f9a68152fc0 2023-08-25T10:05:27.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.487+0000 7f9a6e5cb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681592c0 con 0x7f9a680cad40 2023-08-25T10:05:27.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a6cdc8640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f9a68158d60 0x7f9a680c97a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52746/0 (socket says 172.21.15.71:52746) 2023-08-25T10:05:27.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a6cdc8640 1 -- 172.21.15.71:0/922953874 learned_addr learned my addr 172.21.15.71:0/922953874 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:27.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a6ddca640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 0x7f9a680cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274217253 0 0) 0x7f9a6815c6c0 con 0x7f9a68158d60 2023-08-25T10:05:27.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a54003520 con 0x7f9a68158d60 2023-08-25T10:05:27.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554927445 0 0) 0x7f9a54003520 con 0x7f9a68158d60 2023-08-25T10:05:27.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.488+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 msgr2=0x7f9a680cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 0x7f9a680cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 >> v1:172.21.15.164:6789/0 conn(0x7f9a68152fc0 legacy=0x7f9a6815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a681595a0 con 0x7f9a68158d60 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f9a58001670 con 0x7f9a68158d60 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 >> v1:172.21.15.71:6789/0 conn(0x7f9a68158d60 legacy=0x7f9a680c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a54003a10 0x7f9a54003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a540047c0 0x7f9a54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 0x7f9a54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681592c0 con 0x7f9a540047c0 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5400bb10 con 0x7f9a54003a10 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5400bcd0 con 0x7f9a680cad40 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6cdc8640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a540047c0 0x7f9a54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6d5c9640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a54003a10 0x7f9a54003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6ddca640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 0x7f9a54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6cdc8640 1 -- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 msgr2=0x7f9a54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:27.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6cdc8640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a680cad40 0x7f9a54005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6cdc8640 1 -- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a54003a10 msgr2=0x7f9a54003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:27.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6cdc8640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a54003a10 0x7f9a54003e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.489+0000 7f9a6cdc8640 1 -- 172.21.15.71:0/922953874 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a54003520 con 0x7f9a540047c0 2023-08-25T10:05:27.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.490+0000 7f9a6cdc8640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a540047c0 0x7f9a54004c40 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c190 tx=0x7f9a540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d320e0ed5a9df635 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:27.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58005ad0 con 0x7f9a540047c0 2023-08-25T10:05:27.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58005c30 con 0x7f9a540047c0 2023-08-25T10:05:27.498 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a5ffff640 1 -- 172.21.15.71:0/922953874 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5802c8a0 con 0x7f9a540047c0 2023-08-25T10:05:27.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/922953874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a540047c0 msgr2=0x7f9a54004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/922953874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a540047c0 0x7f9a54004c40 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800c190 tx=0x7f9a540057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a6ddca640 1 -- 172.21.15.71:0/922953874 reap_dead start 2023-08-25T10:05:27.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/922953874 shutdown_connections 2023-08-25T10:05:27.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/922953874 >> 172.21.15.71:0/922953874 conn(0x7f9a6806a3a0 msgr2=0x7f9a68070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:27.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.491+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/922953874 shutdown_connections 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.492+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/922953874 wait complete. 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.493+0000 7f9a6e5cb640 1 Processor -- start 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.493+0000 7f9a6e5cb640 1 -- start start 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.494+0000 7f9a6e5cb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a540047c0 0x7f9a681eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.494+0000 7f9a6e5cb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a680cc590 0x7f9a681ef140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.494+0000 7f9a6e5cb640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 0x7f9a68155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.494+0000 7f9a6e5cb640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681ee700 con 0x7f9a680c8040 2023-08-25T10:05:27.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.494+0000 7f9a6e5cb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681f2680 con 0x7f9a680cc590 2023-08-25T10:05:27.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.494+0000 7f9a6e5cb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681f2960 con 0x7f9a540047c0 2023-08-25T10:05:27.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 0x7f9a68155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 0x7f9a68155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60638/0 (socket says 172.21.15.71:60638) 2023-08-25T10:05:27.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 -- 172.21.15.71:0/3916394831 learned_addr learned my addr 172.21.15.71:0/3916394831 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:27.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6d5c9640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a540047c0 0x7f9a681eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6cdc8640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a680cc590 0x7f9a681ef140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 -- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a540047c0 msgr2=0x7f9a681eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a540047c0 0x7f9a681eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 -- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a680cc590 msgr2=0x7f9a681ef140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a680cc590 0x7f9a681ef140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.505 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 -- 172.21.15.71:0/3916394831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6815bcc0 con 0x7f9a680c8040 2023-08-25T10:05:27.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6cdc8640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a680cc590 0x7f9a681ef140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:27.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.495+0000 7f9a6ddca640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 0x7f9a68155b90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9a680cbaf0 tx=0x7f9a6000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:27.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.496+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60017400 con 0x7f9a680c8040 2023-08-25T10:05:27.507 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.496+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60011040 con 0x7f9a680c8040 2023-08-25T10:05:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.496+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6815bfa0 con 0x7f9a680c8040 2023-08-25T10:05:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.496+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a600177c0 con 0x7f9a680c8040 2023-08-25T10:05:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.496+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a681560d0 con 0x7f9a680c8040 2023-08-25T10:05:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.497+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6804e420 con 0x7f9a680c8040 2023-08-25T10:05:27.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.498+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f9a60017920 con 0x7f9a680c8040 2023-08-25T10:05:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.499+0000 7f9a5dffb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.499+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f9a60089070 con 0x7f9a680c8040 2023-08-25T10:05:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.499+0000 7f9a6d5c9640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.500+0000 7f9a6d5c9640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9a64001690 tx=0x7f9a64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.504+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f9a60054200 con 0x7f9a680c8040 2023-08-25T10:05:27.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.737+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a680c9580 con 0x7f9a680c8040 2023-08-25T10:05:27.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.738+0000 7f9a5dffb640 1 -- 172.21.15.71:0/3916394831 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f9a60057820 con 0x7f9a680c8040 2023-08-25T10:05:27.741 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:27.742 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:27.742 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:27.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 msgr2=0x7f9a4c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9a64001690 tx=0x7f9a64009040 comp rx=0 tx=0).stop 2023-08-25T10:05:27.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 msgr2=0x7f9a68155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:27.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 0x7f9a68155b90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9a680cbaf0 tx=0x7f9a6000f4a0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 shutdown_connections 2023-08-25T10:05:27.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a680cc590 0x7f9a681ef140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f9a4c06a050 0x7f9a4c06c510 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a680c8040 0x7f9a68155b90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 --2- 172.21.15.71:0/3916394831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a540047c0 0x7f9a681eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:27.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 >> 172.21.15.71:0/3916394831 conn(0x7f9a6806a3a0 msgr2=0x7f9a680c35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:27.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 shutdown_connections 2023-08-25T10:05:27.746 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:27.741+0000 7f9a6e5cb640 1 -- 172.21.15.71:0/3916394831 wait complete. 2023-08-25T10:05:27.762 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-08-25T10:05:29.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:30.000 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.997+0000 7f7c2d5fc640 1 Processor -- start 2023-08-25T10:05:30.001 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.997+0000 7f7c2d5fc640 1 -- start start 2023-08-25T10:05:30.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.998+0000 7f7c2d5fc640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f7c280cb570 0x7f7c280ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:30.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.998+0000 7f7c2d5fc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280b1af0 con 0x7f7c2815cd00 2023-08-25T10:05:30.002 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.998+0000 7f7c2d5fc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280b0fb0 con 0x7f7c280cb190 2023-08-25T10:05:30.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.998+0000 7f7c2d5fc640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c280c98f0 con 0x7f7c280cb570 2023-08-25T10:05:30.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c2cdfb640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7c2815cd00 0x7f7c28158d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52756/0 (socket says 172.21.15.71:52756) 2023-08-25T10:05:30.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c2cdfb640 1 -- 172.21.15.71:0/2028075061 learned_addr learned my addr 172.21.15.71:0/2028075061 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:30.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c277fe640 1 --2- 172.21.15.71:0/2028075061 >> v2:172.21.15.71:3300/0 conn(0x7f7c280cb570 0x7f7c280ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:30.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c277fe640 1 -- 172.21.15.71:0/2028075061 >> v1:172.21.15.164:6789/0 conn(0x7f7c280cb190 legacy=0x7f7c280c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.003 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c277fe640 1 -- 172.21.15.71:0/2028075061 >> v1:172.21.15.71:6789/0 conn(0x7f7c2815cd00 legacy=0x7f7c28158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c26ffd640 1 -- 172.21.15.71:0/2028075061 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017187210 0 0) 0x7f7c280b1af0 con 0x7f7c2815cd00 2023-08-25T10:05:30.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:29.999+0000 7f7c277fe640 1 -- 172.21.15.71:0/2028075061 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2815d130 con 0x7f7c280cb570 2023-08-25T10:05:30.004 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.001+0000 7f7c277fe640 1 --2- 172.21.15.71:0/2028075061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c280ca910 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0097a0 tx=0x7f7c1c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf128b9a38cf3b58 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:30.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.002+0000 7f7c26ffd640 1 -- 172.21.15.71:0/2028075061 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c02f810 con 0x7f7c280cb570 2023-08-25T10:05:30.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.002+0000 7f7c26ffd640 1 -- 172.21.15.71:0/2028075061 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c02f970 con 0x7f7c280cb570 2023-08-25T10:05:30.005 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.002+0000 7f7c26ffd640 1 -- 172.21.15.71:0/2028075061 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c02fd00 con 0x7f7c280cb570 2023-08-25T10:05:30.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.003+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/2028075061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 msgr2=0x7f7c280ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.006 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.003+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/2028075061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c280ca910 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0097a0 tx=0x7f7c1c02b9f0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.004+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/2028075061 shutdown_connections 2023-08-25T10:05:30.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.004+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/2028075061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c280ca910 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.004+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/2028075061 >> 172.21.15.71:0/2028075061 conn(0x7f7c2806ae20 msgr2=0x7f7c2806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:30.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.004+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/2028075061 shutdown_connections 2023-08-25T10:05:30.007 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.004+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/2028075061 wait complete. 2023-08-25T10:05:30.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.006+0000 7f7c2d5fc640 1 Processor -- start 2023-08-25T10:05:30.009 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.006+0000 7f7c2d5fc640 1 -- start start 2023-08-25T10:05:30.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.007+0000 7f7c2d5fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c280cb190 0x7f7c2814dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:30.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.007+0000 7f7c2d5fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c2814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:30.010 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.007+0000 7f7c2d5fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2815cd00 0x7f7c28155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:30.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.007+0000 7f7c2d5fc640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2814b670 con 0x7f7c280cb570 2023-08-25T10:05:30.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.007+0000 7f7c2d5fc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2814c230 con 0x7f7c2815cd00 2023-08-25T10:05:30.011 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.007+0000 7f7c2d5fc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2814b3a0 con 0x7f7c280cb190 2023-08-25T10:05:30.012 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c2814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c2814dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35154/0 (socket says 172.21.15.71:35154) 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 -- 172.21.15.71:0/1269072905 learned_addr learned my addr 172.21.15.71:0/1269072905 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c2cdfb640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2815cd00 0x7f7c28155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c27fff640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c280cb190 0x7f7c2814dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 -- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c280cb190 msgr2=0x7f7c2814dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c280cb190 0x7f7c2814dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 -- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2815cd00 msgr2=0x7f7c28155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2815cd00 0x7f7c28155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.008+0000 7f7c277fe640 1 -- 172.21.15.71:0/1269072905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c009400 con 0x7f7c280cb570 2023-08-25T10:05:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.009+0000 7f7c277fe640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c2814dff0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:30.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.009+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c03b050 con 0x7f7c280cb570 2023-08-25T10:05:30.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.009+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c03c040 con 0x7f7c280cb570 2023-08-25T10:05:30.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.009+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c004490 con 0x7f7c280cb570 2023-08-25T10:05:30.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.009+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c28153890 con 0x7f7c280cb570 2023-08-25T10:05:30.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.009+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2814c760 con 0x7f7c280cb570 2023-08-25T10:05:30.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.010+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2804e420 con 0x7f7c280cb570 2023-08-25T10:05:30.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.012+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f7c1c037750 con 0x7f7c280cb570 2023-08-25T10:05:30.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.013+0000 7f7c24ff9640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c1406a050 0x7f7c1406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:30.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.014+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f7c1c0ab2d0 con 0x7f7c280cb570 2023-08-25T10:05:30.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.014+0000 7f7c27fff640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c1406a050 0x7f7c1406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:30.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.015+0000 7f7c27fff640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c1406a050 0x7f7c1406c510 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c18000f60 tx=0x7f7c18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:30.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.020+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f7c1c076630 con 0x7f7c280cb570 2023-08-25T10:05:30.354 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.351+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c280c9df0 con 0x7f7c280cb570 2023-08-25T10:05:30.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.352+0000 7f7c24ff9640 1 -- 172.21.15.71:0/1269072905 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7f7c1c02ee60 con 0x7f7c280cb570 2023-08-25T10:05:30.355 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:30.356 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:30.356 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:30.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c1406a050 msgr2=0x7f7c1406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c1406a050 0x7f7c1406c510 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c18000f60 tx=0x7f7c18009040 comp rx=0 tx=0).stop 2023-08-25T10:05:30.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 msgr2=0x7f7c2814dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:30.358 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c2814dff0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000c00 tx=0x7f7c1c0046c0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 shutdown_connections 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c2815cd00 0x7f7c28155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f7c1406a050 0x7f7c1406c510 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7c280cb570 0x7f7c2814dff0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 --2- 172.21.15.71:0/1269072905 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c280cb190 0x7f7c2814dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 >> 172.21.15.71:0/1269072905 conn(0x7f7c2806ae20 msgr2=0x7f7c280c6190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:30.359 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 shutdown_connections 2023-08-25T10:05:30.360 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:30.355+0000 7f7c2d5fc640 1 -- 172.21.15.71:0/1269072905 wait complete. 2023-08-25T10:05:30.376 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-08-25T10:05:30.377 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-08-25T09:49:34.063814+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4293": { "addr": "172.21.15.71:6839/3369232025", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6838", "nonce": 3369232025, "type": "v2" }, { "addr": "172.21.15.71:6839", "nonce": 3369232025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4293, "incarnation": 8, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 11 }, "gid_4306": { "addr": "172.21.15.164:6837/881271092", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6836", "nonce": 881271092, "type": "v2" }, { "addr": "172.21.15.164:6837", "nonce": 881271092, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4306, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4810": { "addr": "172.21.15.164:6835/882191558", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6834", "nonce": 882191558, "type": "v2" }, { "addr": "172.21.15.164:6835", "nonce": 882191558, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4810, "incarnation": 34, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:stopping", "state_seq": 40 }, "gid_4897": { "addr": "172.21.15.164:6833/3042271229", "addrs": { "addrvec": [ { "addr": "172.21.15.164:6832", "nonce": 3042271229, "type": "v2" }, { "addr": "172.21.15.164:6833", "nonce": 3042271229, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 3, "state": "up:active", "state_seq": 20 }, "gid_5547": { "addr": "172.21.15.71:6836/3952960380", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6834", "nonce": 3952960380, "type": "v2" }, { "addr": "172.21.15.71:6836", "nonce": 3952960380, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5547, "incarnation": 63, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 3, "modified": "2023-08-25T10:02:14.873638+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4306, "mds_1": 4293, "mds_2": 5547, "mds_3": 4897, "mds_4": 4810 } } } ], "standbys": [ { "addr": "172.21.15.71:6837/3908401002", "addrs": { "addrvec": [ { "addr": "172.21.15.71:6835", "nonce": 3908401002, "type": "v2" }, { "addr": "172.21.15.71:6837", "nonce": 3908401002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 69, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5703, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-08-25T10:05:32.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-08-25T10:05:32.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.600+0000 7fa418bd2640 1 Processor -- start 2023-08-25T10:05:32.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.600+0000 7fa418bd2640 1 -- start start 2023-08-25T10:05:32.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.602+0000 7fa418bd2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa4141542d0 0x7fa4141576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.602+0000 7fa418bd2640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa414157bf0 con 0x7fa4140c4fb0 2023-08-25T10:05:32.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.602+0000 7fa418bd2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41406a130 con 0x7fa414069850 2023-08-25T10:05:32.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.602+0000 7fa418bd2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4140c4b60 con 0x7fa4141542d0 2023-08-25T10:05:32.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.602+0000 7fa413fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa4140c4fb0 0x7fa4140c5440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34630/0 (socket says 172.21.15.71:34630) 2023-08-25T10:05:32.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.602+0000 7fa413fff640 1 -- 172.21.15.71:0/1744574050 learned_addr learned my addr 172.21.15.71:0/1744574050 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:32.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4137fe640 1 --2- 172.21.15.71:0/1744574050 >> v2:172.21.15.164:3300/0 conn(0x7fa4141542d0 0x7fa4141576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865895888 0 0) 0x7fa40c001490 con 0x7fa4140c4fb0 2023-08-25T10:05:32.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa400003520 con 0x7fa4140c4fb0 2023-08-25T10:05:32.608 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4148627373 0 0) 0x7fa400003520 con 0x7fa4140c4fb0 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 >> v2:172.21.15.164:3300/0 conn(0x7fa4141542d0 msgr2=0x7fa4141576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 --2- 172.21.15.71:0/1744574050 >> v2:172.21.15.164:3300/0 conn(0x7fa4141542d0 0x7fa4141576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 >> v1:172.21.15.164:6789/0 conn(0x7fa414069850 legacy=0x7fa4140c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4141546b0 con 0x7fa4140c4fb0 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.603+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fa40c001b80 con 0x7fa4140c4fb0 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 >> v1:172.21.15.71:6789/0 conn(0x7fa4140c4fb0 legacy=0x7fa4140c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000039e0 0x7fa400003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000047c0 0x7fa400004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa40000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4140c4b60 con 0x7fa400005180 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40000c0b0 con 0x7fa4000039e0 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40000c270 con 0x7fa4000047c0 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa413fff640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa40000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa4137fe640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000039e0 0x7fa400003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa412ffd640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000047c0 0x7fa400004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa413fff640 1 -- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000047c0 msgr2=0x7fa400004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa413fff640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4000047c0 0x7fa400004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa413fff640 1 -- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000039e0 msgr2=0x7fa400003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa413fff640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4000039e0 0x7fa400003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.604+0000 7fa413fff640 1 -- 172.21.15.71:0/1744574050 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400003520 con 0x7fa400005180 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.605+0000 7fa413fff640 1 -- 172.21.15.71:0/1744574050 reap_dead start 2023-08-25T10:05:32.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.605+0000 7fa413fff640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa40000b980 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00a800 tx=0x7fa4000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb22828440c75b8a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:32.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c034630 con 0x7fa400005180 2023-08-25T10:05:32.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c02f070 con 0x7fa400005180 2023-08-25T10:05:32.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa4127fc640 1 -- 172.21.15.71:0/1744574050 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0349f0 con 0x7fa400005180 2023-08-25T10:05:32.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa418bd2640 1 -- 172.21.15.71:0/1744574050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 msgr2=0x7fa40000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa40000b980 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00a800 tx=0x7fa4000057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa418bd2640 1 -- 172.21.15.71:0/1744574050 shutdown_connections 2023-08-25T10:05:32.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1744574050 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa40000b980 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.606+0000 7fa418bd2640 1 -- 172.21.15.71:0/1744574050 >> 172.21.15.71:0/1744574050 conn(0x7fa41406c280 msgr2=0x7fa41406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:32.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.607+0000 7fa418bd2640 1 -- 172.21.15.71:0/1744574050 shutdown_connections 2023-08-25T10:05:32.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.607+0000 7fa418bd2640 1 -- 172.21.15.71:0/1744574050 wait complete. 2023-08-25T10:05:32.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.608+0000 7fa418bd2640 1 Processor -- start 2023-08-25T10:05:32.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.608+0000 7fa418bd2640 1 -- start start 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa418bd2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa4141ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa418bd2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4140c1830 0x7fa41414cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa418bd2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa41414d530 0x7fa4140c3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa418bd2640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41414c770 con 0x7fa400005180 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa418bd2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4141ecc30 con 0x7fa4140c1830 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa418bd2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41414b810 con 0x7fa41414d530 2023-08-25T10:05:32.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.609+0000 7fa4137fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa4141ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa4141ec6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35172/0 (socket says 172.21.15.71:35172) 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 -- 172.21.15.71:0/1071310629 learned_addr learned my addr 172.21.15.71:0/1071310629 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa412ffd640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4140c1830 0x7fa41414cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa413fff640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa41414d530 0x7fa4140c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 -- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa41414d530 msgr2=0x7fa4140c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa41414d530 0x7fa4140c3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.617 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 -- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4140c1830 msgr2=0x7fa41414cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4140c1830 0x7fa41414cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 -- 172.21.15.71:0/1071310629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41414baf0 con 0x7fa400005180 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa4137fe640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa4141ec6f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa404011540 tx=0x7fa404011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.610+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404015930 con 0x7fa400005180 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.611+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa404015a90 con 0x7fa400005180 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.611+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa404015e20 con 0x7fa400005180 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.611+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41414bdd0 con 0x7fa400005180 2023-08-25T10:05:32.618 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.611+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41414c390 con 0x7fa400005180 2023-08-25T10:05:32.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.612+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41414c550 con 0x7fa400005180 2023-08-25T10:05:32.619 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.613+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fa40400f460 con 0x7fa400005180 2023-08-25T10:05:32.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.614+0000 7fa40bfff640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3f406a080 0x7fa3f406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:32.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.614+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fa404089350 con 0x7fa400005180 2023-08-25T10:05:32.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.614+0000 7fa412ffd640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3f406a080 0x7fa3f406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:32.620 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.615+0000 7fa412ffd640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3f406a080 0x7fa3f406c540 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa414071d40 tx=0x7fa3fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:32.621 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.618+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fa404057ee0 con 0x7fa400005180 2023-08-25T10:05:32.880 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.877+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa41404e420 con 0x7fa400005180 2023-08-25T10:05:32.881 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.878+0000 7fa40bfff640 1 -- 172.21.15.71:0/1071310629 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+6524 (secure 0 0 0) 0x7fa404057a70 con 0x7fa400005180 2023-08-25T10:05:32.881 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:32.882 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5703,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/3908401002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6835","nonce":3908401002},{"type":"v1","addr":"172.21.15.71:6837","nonce":3908401002}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-08-25T09:49:34.063814+0000","modified":"2023-08-25T10:02:14.873638+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1,2,3,4],"up":{"mds_0":4306,"mds_1":4293,"mds_2":5547,"mds_3":4897,"mds_4":4810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"e","rank":1,"incarnation":8,"state":"up:active","state_seq":11,"addr":"172.21.15.71:6839/3369232025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3369232025},{"type":"v1","addr":"172.21.15.71:6839","nonce":3369232025}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4306":{"gid":4306,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.164:6837/881271092","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":881271092},{"type":"v1","addr":"172.21.15.164:6837","nonce":881271092}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4810":{"gid":4810,"name":"d","rank":4,"incarnation":34,"state":"up:stopping","state_seq":40,"addr":"172.21.15.164:6835/882191558","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":882191558},{"type":"v1","addr":"172.21.15.164:6835","nonce":882191558}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4897":{"gid":4897,"name":"b","rank":3,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.164:6833/3042271229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3042271229},{"type":"v1","addr":"172.21.15.164:6833","nonce":3042271229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5547":{"gid":5547,"name":"a","rank":2,"incarnation":63,"state":"up:active","state_seq":23,"addr":"172.21.15.71:6836/3952960380","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":3952960380},{"type":"v1","addr":"172.21.15.71:6836","nonce":3952960380}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-08-25T10:05:32.882 INFO:teuthology.orchestra.run.smithi071.stderr:dumped fsmap epoch 94 2023-08-25T10:05:32.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3f406a080 msgr2=0x7fa3f406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3f406a080 0x7fa3f406c540 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa414071d40 tx=0x7fa3fc009040 comp rx=0 tx=0).stop 2023-08-25T10:05:32.884 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 msgr2=0x7fa4141ec6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:32.885 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa4141ec6f0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa404011540 tx=0x7fa404011a00 comp rx=0 tx=0).stop 2023-08-25T10:05:32.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 shutdown_connections 2023-08-25T10:05:32.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4140c1830 0x7fa41414cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fa3f406a080 0x7fa3f406c540 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa41414d530 0x7fa4140c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 --2- 172.21.15.71:0/1071310629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa400005180 0x7fa4141ec6f0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:32.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.881+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 >> 172.21.15.71:0/1071310629 conn(0x7fa41406c280 msgr2=0x7fa414150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:32.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.882+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 shutdown_connections 2023-08-25T10:05:32.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:32.882+0000 7fa418bd2640 1 -- 172.21.15.71:0/1071310629 wait complete. 2023-08-25T10:05:32.905 ERROR:tasks.mds_thrash.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 124, in _run self.do_thrash() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 257, in do_thrash self.wait_for_stable() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 217, in wait_for_stable raise RuntimeError('timeout waiting for cluster to stabilize') RuntimeError: timeout waiting for cluster to stabilize 2023-08-25T10:05:37.334 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.fs.[cephfs] failed 2023-08-25T10:05:37.334 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-08-25T10:05:37.335 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T10:05:37.335 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T10:05:37.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T10:05:37.483 INFO:tasks.workunit.client.0.smithi071.stderr:warning: die() called many times. Recursion error or racy threaded death! 2023-08-25T10:05:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:fatal: cannot pread pack file: Transport endpoint is not connected 2023-08-25T10:05:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:fatal: cannot pread pack file: Transport endpoint is not connected 2023-08-25T10:05:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:fatal: cannot pread pack file: Software caused connection abort 2023-08-25T10:05:37.484 INFO:tasks.workunit.client.0.smithi071.stderr:fatal: cannot pread pack file: Transport endpoint is not connected 2023-08-25T10:05:37.525 INFO:tasks.workunit.client.0.smithi071.stderr:fatal: fetch-pack: invalid index-pack output 2023-08-25T10:05:37.526 INFO:tasks.workunit.client.0.smithi071.stderr:warning: unable to unlink '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/shallow_MOOSXq': Transport endpoint is not connected 2023-08-25T10:05:37.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.591+0000 7f0c8e804640 1 Processor -- start 2023-08-25T10:05:37.593 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.591+0000 7f0c8e804640 1 -- start start 2023-08-25T10:05:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c8e804640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c880c3a40 0x7f0c880c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c8e804640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0c881509d0 0x7f0c88150e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c8e804640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c880c1dc0 con 0x7f0c880c3660 2023-08-25T10:05:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c8e804640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c88157a80 con 0x7f0c881509d0 2023-08-25T10:05:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c8e804640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c881513a0 con 0x7f0c880c3a40 2023-08-25T10:05:37.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c87fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0c880c3660 0x7f0c8806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34644/0 (socket says 172.21.15.71:34644) 2023-08-25T10:05:37.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c87fff640 1 -- 172.21.15.71:0/154561755 learned_addr learned my addr 172.21.15.71:0/154561755 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.593+0000 7f0c8cda1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f0c881509d0 0x7f0c88150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.594+0000 7f0c877fe640 1 --2- 172.21.15.71:0/154561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c880c3a40 0x7f0c880c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.594+0000 7f0c8cda1640 1 -- 172.21.15.71:0/154561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c880c3a40 msgr2=0x7f0c880c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.594+0000 7f0c8cda1640 1 --2- 172.21.15.71:0/154561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c880c3a40 0x7f0c880c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.594+0000 7f0c8cda1640 1 -- 172.21.15.71:0/154561755 >> v1:172.21.15.71:6789/0 conn(0x7f0c880c3660 legacy=0x7f0c8806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.594+0000 7f0c8cda1640 1 -- 172.21.15.71:0/154561755 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c88156e00 con 0x7f0c881509d0 2023-08-25T10:05:37.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.594+0000 7f0c86ffd640 1 -- 172.21.15.71:0/154561755 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230429241 0 0) 0x7f0c880c1dc0 con 0x7f0c880c3660 2023-08-25T10:05:37.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.595+0000 7f0c8cda1640 1 --2- 172.21.15.71:0/154561755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c881509d0 0x7f0c88150e60 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009d90 tx=0x7f0c8002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4fc83d968a09db server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.595+0000 7f0c86ffd640 1 -- 172.21.15.71:0/154561755 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8002cce0 con 0x7f0c881509d0 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.595+0000 7f0c86ffd640 1 -- 172.21.15.71:0/154561755 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8002ce40 con 0x7f0c881509d0 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.595+0000 7f0c86ffd640 1 -- 172.21.15.71:0/154561755 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c80034660 con 0x7f0c881509d0 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 -- 172.21.15.71:0/154561755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c881509d0 msgr2=0x7f0c88150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 --2- 172.21.15.71:0/154561755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c881509d0 0x7f0c88150e60 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009d90 tx=0x7f0c8002bf30 comp rx=0 tx=0).stop 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 -- 172.21.15.71:0/154561755 shutdown_connections 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 --2- 172.21.15.71:0/154561755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c881509d0 0x7f0c88150e60 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 --2- 172.21.15.71:0/154561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c880c3a40 0x7f0c880c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 -- 172.21.15.71:0/154561755 >> 172.21.15.71:0/154561755 conn(0x7f0c880654f0 msgr2=0x7f0c88065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 -- 172.21.15.71:0/154561755 shutdown_connections 2023-08-25T10:05:37.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.596+0000 7f0c8e804640 1 -- 172.21.15.71:0/154561755 wait complete. 2023-08-25T10:05:37.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.598+0000 7f0c8e804640 1 Processor -- start 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.598+0000 7f0c8e804640 1 -- start start 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c8e804640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c880c3660 0x7f0c880702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c8e804640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 0x7f0c8806aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c8e804640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c881509d0 0x7f0c88158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c8e804640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8814ad40 con 0x7f0c880c3a40 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c8e804640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8814a730 con 0x7f0c880c3660 2023-08-25T10:05:37.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c8e804640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8814bca0 con 0x7f0c881509d0 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c877fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 0x7f0c8806aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c877fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 0x7f0c8806aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35194/0 (socket says 172.21.15.71:35194) 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.599+0000 7f0c877fe640 1 -- 172.21.15.71:0/4048773090 learned_addr learned my addr 172.21.15.71:0/4048773090 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c8cda1640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c881509d0 0x7f0c88158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c87fff640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c880c3660 0x7f0c880702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c877fe640 1 -- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c881509d0 msgr2=0x7f0c88158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:37.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c877fe640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c881509d0 0x7f0c88158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c877fe640 1 -- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c880c3660 msgr2=0x7f0c880702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c877fe640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c880c3660 0x7f0c880702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c877fe640 1 -- 172.21.15.71:0/4048773090 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c80009a70 con 0x7f0c880c3a40 2023-08-25T10:05:37.604 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c87fff640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c880c3660 0x7f0c880702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c877fe640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 0x7f0c8806aa10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00d5f0 tx=0x7f0c7c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c0125f0 con 0x7f0c880c3a40 2023-08-25T10:05:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.600+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c012750 con 0x7f0c880c3a40 2023-08-25T10:05:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.601+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c012b00 con 0x7f0c880c3a40 2023-08-25T10:05:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.601+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8814bf60 con 0x7f0c880c3a40 2023-08-25T10:05:37.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.601+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8814c240 con 0x7f0c880c3a40 2023-08-25T10:05:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.602+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c880c29c0 con 0x7f0c880c3a40 2023-08-25T10:05:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.603+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0c7c012c60 con 0x7f0c880c3a40 2023-08-25T10:05:37.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.604+0000 7f0c84ff9640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0c7406a050 0x7f0c7406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:37.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.604+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f0c7c088fe0 con 0x7f0c880c3a40 2023-08-25T10:05:37.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.604+0000 7f0c87fff640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0c7406a050 0x7f0c7406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:37.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.604+0000 7f0c87fff640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0c7406a050 0x7f0c7406c510 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0c78000f60 tx=0x7f0c78009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:37.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.612+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0c7c054140 con 0x7f0c880c3a40 2023-08-25T10:05:37.673 INFO:tasks.workunit.client.0.smithi071.stderr:+ retry 2023-08-25T10:05:37.673 INFO:tasks.workunit.client.0.smithi071.stderr:+ rm -rf ceph 2023-08-25T10:05:37.675 INFO:tasks.workunit.client.0.smithi071.stderr:rm: cannot remove 'ceph': Transport endpoint is not connected 2023-08-25T10:05:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T10:05:37.677 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-08-25T10:05:37.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-08-25T10:05:37.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.806+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c88069160 con 0x7f0c880c3a40 2023-08-25T10:05:37.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.809+0000 7f0c84ff9640 1 -- 172.21.15.71:0/4048773090 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20877 (secure 0 0 0) 0x7f0c7c057760 con 0x7f0c880c3a40 2023-08-25T10:05:37.811 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:37.811 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":69,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:59:40.756440+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6837/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.71:6834/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6832/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.71:6836/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6833/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.164:6835/2707317155":"2023-08-26T09:52:30.557077+0000","172.21.15.71:6835/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.164:6834/2707317155":"2023-08-26T09:52:30.557077+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":504,"length":2}]},{"pool":5,"snaps":[{"begin":504,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T10:05:37.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0c7406a050 msgr2=0x7f0c7406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:37.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0c7406a050 0x7f0c7406c510 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0c78000f60 tx=0x7f0c78009040 comp rx=0 tx=0).stop 2023-08-25T10:05:37.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 msgr2=0x7f0c8806aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 0x7f0c8806aa10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00d5f0 tx=0x7f0c7c00dab0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 shutdown_connections 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0c7406a050 0x7f0c7406c510 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c881509d0 0x7f0c88158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0c880c3a40 0x7f0c8806aa10 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 --2- 172.21.15.71:0/4048773090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c880c3660 0x7f0c880702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:37.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 >> 172.21.15.71:0/4048773090 conn(0x7f0c880654f0 msgr2=0x7f0c880c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:37.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.811+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 shutdown_connections 2023-08-25T10:05:37.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:37.812+0000 7f0c8e804640 1 -- 172.21.15.71:0/4048773090 wait complete. 2023-08-25T10:05:37.831 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T10:05:37.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T10:05:38.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.041+0000 7f2959ce7640 1 Processor -- start 2023-08-25T10:05:38.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.041+0000 7f2959ce7640 1 -- start start 2023-08-25T10:05:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2959ce7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2954150f90 0x7f295415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2959ce7640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f29540713c0 0x7f29540c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2959ce7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2954156d20 con 0x7f2954070f70 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2959ce7640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2954156b30 con 0x7f29540713c0 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2959ce7640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2954157000 con 0x7f2954150f90 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2953fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f29540713c0 0x7f29540c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2953fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f29540713c0 0x7f29540c78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35196/0 (socket says 172.21.15.71:35196) 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2953fff640 1 -- 172.21.15.71:0/2439876447 learned_addr learned my addr 172.21.15.71:0/2439876447 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f2953fff640 1 -- 172.21.15.71:0/2439876447 >> v2:172.21.15.164:3300/0 conn(0x7f2954150f90 msgr2=0x7f295415ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:38.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.043+0000 7f29537fe640 1 --2- 172.21.15.71:0/2439876447 >> v2:172.21.15.164:3300/0 conn(0x7f2954150f90 0x7f295415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.044+0000 7f2953fff640 1 --2- 172.21.15.71:0/2439876447 >> v2:172.21.15.164:3300/0 conn(0x7f2954150f90 0x7f295415ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.044+0000 7f2953fff640 1 -- 172.21.15.71:0/2439876447 >> v1:172.21.15.71:6789/0 conn(0x7f2954070f70 legacy=0x7f29540c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.044+0000 7f2953fff640 1 -- 172.21.15.71:0/2439876447 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29541572e0 con 0x7f29540713c0 2023-08-25T10:05:38.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.044+0000 7f29537fe640 1 --2- 172.21.15.71:0/2439876447 >> v2:172.21.15.164:3300/0 conn(0x7f2954150f90 0x7f295415ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.045+0000 7f2953fff640 1 --2- 172.21.15.71:0/2439876447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29540713c0 0x7f29540c78b0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2954070c90 tx=0x7f294c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956ae86d65d99cf6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.045+0000 7f29527fc640 1 -- 172.21.15.71:0/2439876447 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c02cba0 con 0x7f29540713c0 2023-08-25T10:05:38.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.045+0000 7f29527fc640 1 -- 172.21.15.71:0/2439876447 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294c02cd00 con 0x7f29540713c0 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.045+0000 7f29527fc640 1 -- 172.21.15.71:0/2439876447 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294c034610 con 0x7f29540713c0 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 -- 172.21.15.71:0/2439876447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29540713c0 msgr2=0x7f29540c78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 --2- 172.21.15.71:0/2439876447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29540713c0 0x7f29540c78b0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2954070c90 tx=0x7f294c02c000 comp rx=0 tx=0).stop 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 -- 172.21.15.71:0/2439876447 shutdown_connections 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 --2- 172.21.15.71:0/2439876447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29540713c0 0x7f29540c78b0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 --2- 172.21.15.71:0/2439876447 >> v2:172.21.15.164:3300/0 conn(0x7f2954150f90 0x7f295415ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 -- 172.21.15.71:0/2439876447 >> 172.21.15.71:0/2439876447 conn(0x7f2954069e80 msgr2=0x7f29540654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:38.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 -- 172.21.15.71:0/2439876447 shutdown_connections 2023-08-25T10:05:38.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.046+0000 7f2959ce7640 1 -- 172.21.15.71:0/2439876447 wait complete. 2023-08-25T10:05:38.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.047+0000 7f2959ce7640 1 Processor -- start 2023-08-25T10:05:38.051 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 -- start start 2023-08-25T10:05:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 0x7f29541532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.052 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29540713c0 0x7f29540c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2954150f90 0x7f29540c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29541f0370 con 0x7f2954070f70 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29541ecba0 con 0x7f29540713c0 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.048+0000 7f2959ce7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29541f0860 con 0x7f2954150f90 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 0x7f29541532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 0x7f29541532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35206/0 (socket says 172.21.15.71:35206) 2023-08-25T10:05:38.053 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 -- 172.21.15.71:0/3258651991 learned_addr learned my addr 172.21.15.71:0/3258651991 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 -- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2954150f90 msgr2=0x7f29540c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2954150f90 0x7f29540c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 -- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29540713c0 msgr2=0x7f29540c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f2952ffd640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29540713c0 0x7f29540c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29540713c0 0x7f29540c5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 -- 172.21.15.71:0/3258651991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c009b30 con 0x7f2954070f70 2023-08-25T10:05:38.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.049+0000 7f29537fe640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 0x7f29541532d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f294400c7f0 tx=0x7f294400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.050+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944007820 con 0x7f2954070f70 2023-08-25T10:05:38.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.050+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2944007980 con 0x7f2954070f70 2023-08-25T10:05:38.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.050+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944007cf0 con 0x7f2954070f70 2023-08-25T10:05:38.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.050+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29541ec590 con 0x7f2954070f70 2023-08-25T10:05:38.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.050+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29541ec830 con 0x7f2954070f70 2023-08-25T10:05:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.051+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29541ec3f0 con 0x7f2954070f70 2023-08-25T10:05:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.052+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f29440113f0 con 0x7f2954070f70 2023-08-25T10:05:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.053+0000 7f293bfff640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f294006a050 0x7f294006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.053+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f2944088d50 con 0x7f2954070f70 2023-08-25T10:05:38.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.053+0000 7f2952ffd640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f294006a050 0x7f294006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.054+0000 7f2952ffd640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f294006a050 0x7f294006c510 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2954159ff0 tx=0x7f2948009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.058+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f29541ec3f0 con 0x7f2954070f70 2023-08-25T10:05:38.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.257+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29541ec3f0 con 0x7f2954070f70 2023-08-25T10:05:38.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.258+0000 7f293bfff640 1 -- 172.21.15.71:0/3258651991 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29541ec3f0 con 0x7f2954070f70 2023-08-25T10:05:38.260 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T10:05:38.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f294006a050 msgr2=0x7f294006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.263 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f294006a050 0x7f294006c510 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2954159ff0 tx=0x7f2948009040 comp rx=0 tx=0).stop 2023-08-25T10:05:38.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 msgr2=0x7f29541532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.264 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 0x7f29541532d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f294400c7f0 tx=0x7f294400ccb0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 shutdown_connections 2023-08-25T10:05:38.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f29540713c0 0x7f29540c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f294006a050 0x7f294006c510 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.265 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2954150f90 0x7f29540c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 --2- 172.21.15.71:0/3258651991 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2954070f70 0x7f29541532d0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 >> 172.21.15.71:0/3258651991 conn(0x7f2954069e80 msgr2=0x7f29540654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:38.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 shutdown_connections 2023-08-25T10:05:38.266 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.261+0000 7f2959ce7640 1 -- 172.21.15.71:0/3258651991 wait complete. 2023-08-25T10:05:38.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T10:05:38.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.488+0000 7ff1de21a640 1 Processor -- start 2023-08-25T10:05:38.490 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.488+0000 7ff1de21a640 1 -- start start 2023-08-25T10:05:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.489+0000 7ff1de21a640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1d80cb190 0x7ff1d80ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.489+0000 7ff1de21a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff1d80cb570 0x7ff1d815cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.491 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.489+0000 7ff1de21a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d80b1af0 con 0x7ff1d80c7f10 2023-08-25T10:05:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.489+0000 7ff1de21a640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80c98f0 con 0x7ff1d80cb190 2023-08-25T10:05:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.489+0000 7ff1de21a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d80b0fb0 con 0x7ff1d80cb570 2023-08-25T10:05:38.492 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1d80cb190 0x7ff1d80ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7ff1d80cb190 0x7ff1d80ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35218/0 (socket says 172.21.15.71:35218) 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 -- 172.21.15.71:0/2864418367 learned_addr learned my addr 172.21.15.71:0/2864418367 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dca17640 1 --2- 172.21.15.71:0/2864418367 >> v2:172.21.15.164:3300/0 conn(0x7ff1d80cb570 0x7ff1d815cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 -- 172.21.15.71:0/2864418367 >> v2:172.21.15.164:3300/0 conn(0x7ff1d80cb570 msgr2=0x7ff1d815cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 --2- 172.21.15.71:0/2864418367 >> v2:172.21.15.164:3300/0 conn(0x7ff1d80cb570 0x7ff1d815cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 -- 172.21.15.71:0/2864418367 >> v1:172.21.15.71:6789/0 conn(0x7ff1d80c7f10 legacy=0x7ff1d8158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.493 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.490+0000 7ff1dd218640 1 -- 172.21.15.71:0/2864418367 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d80b1530 con 0x7ff1d80cb190 2023-08-25T10:05:38.494 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.492+0000 7ff1dd218640 1 --2- 172.21.15.71:0/2864418367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d80ca910 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff1d81536c0 tx=0x7ff1d402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276e3e7a4fd72902 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.492+0000 7ff1cffff640 1 -- 172.21.15.71:0/2864418367 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d402cbb0 con 0x7ff1d80cb190 2023-08-25T10:05:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.492+0000 7ff1cffff640 1 -- 172.21.15.71:0/2864418367 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d402cd10 con 0x7ff1d80cb190 2023-08-25T10:05:38.495 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.492+0000 7ff1cffff640 1 -- 172.21.15.71:0/2864418367 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4034610 con 0x7ff1d80cb190 2023-08-25T10:05:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 -- 172.21.15.71:0/2864418367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 msgr2=0x7ff1d80ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 --2- 172.21.15.71:0/2864418367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d80ca910 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff1d81536c0 tx=0x7ff1d402c010 comp rx=0 tx=0).stop 2023-08-25T10:05:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 -- 172.21.15.71:0/2864418367 shutdown_connections 2023-08-25T10:05:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 --2- 172.21.15.71:0/2864418367 >> v2:172.21.15.164:3300/0 conn(0x7ff1d80cb570 0x7ff1d815cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 --2- 172.21.15.71:0/2864418367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d80ca910 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 -- 172.21.15.71:0/2864418367 >> 172.21.15.71:0/2864418367 conn(0x7ff1d806ae20 msgr2=0x7ff1d806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 -- 172.21.15.71:0/2864418367 shutdown_connections 2023-08-25T10:05:38.497 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.493+0000 7ff1de21a640 1 -- 172.21.15.71:0/2864418367 wait complete. 2023-08-25T10:05:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.495+0000 7ff1de21a640 1 Processor -- start 2023-08-25T10:05:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.495+0000 7ff1de21a640 1 -- start start 2023-08-25T10:05:38.499 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1de21a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d80c7f10 0x7ff1d814d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1de21a640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d81f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1de21a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d80cb570 0x7ff1d81f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1de21a640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d814d190 con 0x7ff1d80cb190 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1de21a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81f25d0 con 0x7ff1d80c7f10 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1de21a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d814c550 con 0x7ff1d80cb570 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1dca17640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d81f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.500 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1dca17640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d81f1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35232/0 (socket says 172.21.15.71:35232) 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.496+0000 7ff1dca17640 1 -- 172.21.15.71:0/3933309510 learned_addr learned my addr 172.21.15.71:0/3933309510 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dda19640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d80cb570 0x7ff1d81f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dd218640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d80c7f10 0x7ff1d814d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dca17640 1 -- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d80cb570 msgr2=0x7ff1d81f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dca17640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d80cb570 0x7ff1d81f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dca17640 1 -- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d80c7f10 msgr2=0x7ff1d814d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dca17640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d80c7f10 0x7ff1d814d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.501 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dca17640 1 -- 172.21.15.71:0/3933309510 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d4009b30 con 0x7ff1d80cb190 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dda19640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d80cb570 0x7ff1d81f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1dca17640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d81f1a00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800d5c0 tx=0x7ff1c800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80125f0 con 0x7ff1d80cb190 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c8012750 con 0x7ff1d80cb190 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8012b00 con 0x7ff1d80cb190 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d81ef690 con 0x7ff1d80cb190 2023-08-25T10:05:38.502 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.497+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d814cad0 con 0x7ff1d80cb190 2023-08-25T10:05:38.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.499+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7ff1c8012c60 con 0x7ff1d80cb190 2023-08-25T10:05:38.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.499+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d80b3cc0 con 0x7ff1d80cb190 2023-08-25T10:05:38.503 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.500+0000 7ff1cdffb640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff1bc06a050 0x7ff1bc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.500+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7ff1c8088f20 con 0x7ff1d80cb190 2023-08-25T10:05:38.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.501+0000 7ff1dd218640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff1bc06a050 0x7ff1bc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.504 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.501+0000 7ff1dd218640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff1bc06a050 0x7ff1bc06c510 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4035000 tx=0x7ff1d4030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.506 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.505+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7ff1c8054140 con 0x7ff1d80cb190 2023-08-25T10:05:38.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.692+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff1d804e420 con 0x7ff1d80cb190 2023-08-25T10:05:38.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.693+0000 7ff1cdffb640 1 -- 172.21.15.71:0/3933309510 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff1c8057760 con 0x7ff1d80cb190 2023-08-25T10:05:38.695 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T10:05:38.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.696+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff1bc06a050 msgr2=0x7ff1bc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.696+0000 7ff1de21a640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff1bc06a050 0x7ff1bc06c510 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4035000 tx=0x7ff1d4030000 comp rx=0 tx=0).stop 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.696+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 msgr2=0x7ff1d81f1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.696+0000 7ff1de21a640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d81f1a00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800d5c0 tx=0x7ff1c800da80 comp rx=0 tx=0).stop 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 shutdown_connections 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7ff1bc06a050 0x7ff1bc06c510 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1d80cb570 0x7ff1d81f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1d80cb190 0x7ff1d81f1a00 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 --2- 172.21.15.71:0/3933309510 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1d80c7f10 0x7ff1d814d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 >> 172.21.15.71:0/3933309510 conn(0x7ff1d806ae20 msgr2=0x7ff1d80c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:38.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 shutdown_connections 2023-08-25T10:05:38.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.697+0000 7ff1de21a640 1 -- 172.21.15.71:0/3933309510 wait complete. 2023-08-25T10:05:38.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T10:05:38.773 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-08-25T10:05:38.841 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b118dbef594b45b7b0dd722658514201 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-08-25T10:05:38.844 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-08-25T10:05:38.854 INFO:tasks.mds_thrash:joining mds_thrasher 2023-08-25T10:05:38.855 ERROR:teuthology.run_tasks:Manager failed: mds_thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-08-25T10:05:38.861 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-08-25T10:05:38.881 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-08-25T10:05:38.882 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T10:05:38.882 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T10:05:38.935 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T10:05:38.936 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T10:05:38.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.938+0000 7f8e54471640 1 Processor -- start 2023-08-25T10:05:38.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.938+0000 7f8e54471640 1 -- start start 2023-08-25T10:05:38.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.940+0000 7f8e54471640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 0x7f8e4c0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.940+0000 7f8e54471640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c1590c0 con 0x7f8e4c153310 2023-08-25T10:05:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.940+0000 7f8e54471640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c159380 con 0x7f8e4c0cec00 2023-08-25T10:05:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.940+0000 7f8e54471640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c159660 con 0x7f8e4c0c4510 2023-08-25T10:05:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.940+0000 7f8e5220d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8e4c153310 0x7f8e4c06a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34668/0 (socket says 172.21.15.71:34668) 2023-08-25T10:05:38.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.940+0000 7f8e5220d640 1 -- 172.21.15.71:0/3952836917 learned_addr learned my addr 172.21.15.71:0/3952836917 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:38.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e52a0e640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 0x7f8e4c0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753851875 0 0) 0x7f8e4c1590c0 con 0x7f8e4c153310 2023-08-25T10:05:38.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38003520 con 0x7f8e4c153310 2023-08-25T10:05:38.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1742972338 0 0) 0x7f8e38003520 con 0x7f8e4c153310 2023-08-25T10:05:38.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 msgr2=0x7f8e4c0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 0x7f8e4c0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 >> v1:172.21.15.164:6789/0 conn(0x7f8e4c0cec00 legacy=0x7f8e4c0c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c159940 con 0x7f8e4c153310 2023-08-25T10:05:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f8e48001670 con 0x7f8e4c153310 2023-08-25T10:05:38.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 >> v1:172.21.15.71:6789/0 conn(0x7f8e4c153310 legacy=0x7f8e4c06a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 0x7f8e38003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e38003f90 0x7f8e380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38004d00 0x7f8e38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c159660 con 0x7f8e38003f90 2023-08-25T10:05:38.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800bb70 con 0x7f8e38004d00 2023-08-25T10:05:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.941+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800bd30 con 0x7f8e4c0c4510 2023-08-25T10:05:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e38003f90 0x7f8e380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e5220d640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 0x7f8e38003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e52a0e640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38004d00 0x7f8e38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e51a0c640 1 -- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 msgr2=0x7f8e38003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0c4510 0x7f8e38003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e51a0c640 1 -- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38004d00 msgr2=0x7f8e38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38004d00 0x7f8e38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.942+0000 7f8e51a0c640 1 -- 172.21.15.71:0/3952836917 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38003520 con 0x7f8e38003f90 2023-08-25T10:05:38.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.943+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e38003f90 0x7f8e380047c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001750 tx=0x7f8e380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f120b0b96e71d0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.953 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0052d0 con 0x7f8e38003f90 2023-08-25T10:05:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c02e050 con 0x7f8e38003f90 2023-08-25T10:05:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e5120b640 1 -- 172.21.15.71:0/3952836917 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c003830 con 0x7f8e38003f90 2023-08-25T10:05:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e54471640 1 -- 172.21.15.71:0/3952836917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e38003f90 msgr2=0x7f8e380047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e54471640 1 --2- 172.21.15.71:0/3952836917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e38003f90 0x7f8e380047c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001750 tx=0x7f8e380057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e52a0e640 1 -- 172.21.15.71:0/3952836917 reap_dead start 2023-08-25T10:05:38.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e54471640 1 -- 172.21.15.71:0/3952836917 shutdown_connections 2023-08-25T10:05:38.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e54471640 1 -- 172.21.15.71:0/3952836917 >> 172.21.15.71:0/3952836917 conn(0x7f8e4c065910 msgr2=0x7f8e4c065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:38.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e54471640 1 -- 172.21.15.71:0/3952836917 shutdown_connections 2023-08-25T10:05:38.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.944+0000 7f8e54471640 1 -- 172.21.15.71:0/3952836917 wait complete. 2023-08-25T10:05:38.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.946+0000 7f8e54471640 1 Processor -- start 2023-08-25T10:05:38.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 -- start start 2023-08-25T10:05:38.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38003f90 0x7f8e4c154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 0x7f8e4c0b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0cdd10 0x7f8e4c0b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.956 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c152120 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c154ac0 con 0x7f8e38003f90 2023-08-25T10:05:38.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.947+0000 7f8e54471640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c151e40 con 0x7f8e4c0cdd10 2023-08-25T10:05:38.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 0x7f8e4c0b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 0x7f8e4c0b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35254/0 (socket says 172.21.15.71:35254) 2023-08-25T10:05:38.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 -- 172.21.15.71:0/2548940582 learned_addr learned my addr 172.21.15.71:0/2548940582 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:38.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e5220d640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38003f90 0x7f8e4c154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e52a0e640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0cdd10 0x7f8e4c0b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 -- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0cdd10 msgr2=0x7f8e4c0b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0cdd10 0x7f8e4c0b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 -- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38003f90 msgr2=0x7f8e4c154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:38.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38003f90 0x7f8e4c154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:38.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.948+0000 7f8e51a0c640 1 -- 172.21.15.71:0/2548940582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0b1bf0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.949+0000 7f8e51a0c640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 0x7f8e4c0b26d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c008fd0 tx=0x7f8e3c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.949+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c007a30 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.949+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c0069e0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.949+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c0b1ed0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.949+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c006d50 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.949+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c150b30 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.950+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c069160 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.951+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8e3c0035f0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.952+0000 7f8e42ffd640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e3006a050 0x7f8e3006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:38.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.952+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f8e3c0aaca0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.952+0000 7f8e5220d640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e3006a050 0x7f8e3006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:38.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.953+0000 7f8e5220d640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e3006a050 0x7f8e3006c510 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8e48001050 tx=0x7f8e48001130 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:38.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:38.956+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8e3c075ec0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:38.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T10:05:39.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.144+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e4c158780 con 0x7f8e4c0cd8b0 2023-08-25T10:05:39.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.145+0000 7f8e42ffd640 1 -- 172.21.15.71:0/2548940582 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e3c0794e0 con 0x7f8e4c0cd8b0 2023-08-25T10:05:39.147 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:39.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e3006a050 msgr2=0x7f8e3006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e3006a050 0x7f8e3006c510 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8e48001050 tx=0x7f8e48001130 comp rx=0 tx=0).stop 2023-08-25T10:05:39.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 msgr2=0x7f8e4c0b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 0x7f8e4c0b26d0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c008fd0 tx=0x7f8e3c001120 comp rx=0 tx=0).stop 2023-08-25T10:05:39.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 shutdown_connections 2023-08-25T10:05:39.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e3006a050 0x7f8e3006c510 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e4c0cdd10 0x7f8e4c0b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8e4c0cd8b0 0x7f8e4c0b26d0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 --2- 172.21.15.71:0/2548940582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38003f90 0x7f8e4c154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 >> 172.21.15.71:0/2548940582 conn(0x7f8e4c065910 msgr2=0x7f8e4c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 shutdown_connections 2023-08-25T10:05:39.153 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.148+0000 7f8e54471640 1 -- 172.21.15.71:0/2548940582 wait complete. 2023-08-25T10:05:39.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T10:05:39.228 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.225+0000 7fc67ef2f640 1 Processor -- start 2023-08-25T10:05:39.229 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.226+0000 7fc67ef2f640 1 -- start start 2023-08-25T10:05:39.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc67ef2f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc678159fc0 0x7fc6780c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc67ef2f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67806a960 con 0x7fc67806cc70 2023-08-25T10:05:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc67ef2f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc67806ac20 con 0x7fc6781594d0 2023-08-25T10:05:39.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc67ef2f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc67806af00 con 0x7fc678159fc0 2023-08-25T10:05:39.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc677fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc67806cc70 0x7fc6781598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34674/0 (socket says 172.21.15.71:34674) 2023-08-25T10:05:39.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc677fff640 1 -- 172.21.15.71:0/2656719756 learned_addr learned my addr 172.21.15.71:0/2656719756 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:39.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc678159fc0 0x7fc6780c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333792511 0 0) 0x7fc67806a960 con 0x7fc67806cc70 2023-08-25T10:05:39.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.229+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc668003520 con 0x7fc67806cc70 2023-08-25T10:05:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 732123898 0 0) 0x7fc668003520 con 0x7fc67806cc70 2023-08-25T10:05:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc678159fc0 msgr2=0x7fc6780c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc678159fc0 0x7fc6780c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 >> v1:172.21.15.164:6789/0 conn(0x7fc6781594d0 legacy=0x7fc67814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780c2e60 con 0x7fc67806cc70 2023-08-25T10:05:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fc664001670 con 0x7fc67806cc70 2023-08-25T10:05:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 >> v1:172.21.15.71:6789/0 conn(0x7fc67806cc70 legacy=0x7fc6781598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc668003a10 0x7fc668003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6680047c0 0x7fc668004c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc668005140 0x7fc66800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67806af00 con 0x7fc668005140 2023-08-25T10:05:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66800bf50 con 0x7fc668003a10 2023-08-25T10:05:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66800e2f0 con 0x7fc6680047c0 2023-08-25T10:05:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc668005140 0x7fc66800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc677fff640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6680047c0 0x7fc668004c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.230+0000 7fc67cccb640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc668003a10 0x7fc668003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.231+0000 7fc67d4cc640 1 -- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6680047c0 msgr2=0x7fc668004c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.231+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6680047c0 0x7fc668004c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.231+0000 7fc67d4cc640 1 -- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc668003a10 msgr2=0x7fc668003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.231+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc668003a10 0x7fc668003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.231+0000 7fc67d4cc640 1 -- 172.21.15.71:0/2656719756 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668003520 con 0x7fc668005140 2023-08-25T10:05:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc668005140 0x7fc66800b980 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc670009f90 tx=0x7fc6680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bffec645df1df45d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67002cbd0 con 0x7fc668005140 2023-08-25T10:05:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67000f040 con 0x7fc668005140 2023-08-25T10:05:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc6777fe640 1 -- 172.21.15.71:0/2656719756 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670034440 con 0x7fc668005140 2023-08-25T10:05:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc67ef2f640 1 -- 172.21.15.71:0/2656719756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc668005140 msgr2=0x7fc66800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/2656719756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc668005140 0x7fc66800b980 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc670009f90 tx=0x7fc6680057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.232+0000 7fc67d4cc640 1 -- 172.21.15.71:0/2656719756 reap_dead start 2023-08-25T10:05:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.233+0000 7fc67ef2f640 1 -- 172.21.15.71:0/2656719756 shutdown_connections 2023-08-25T10:05:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.233+0000 7fc67ef2f640 1 -- 172.21.15.71:0/2656719756 >> 172.21.15.71:0/2656719756 conn(0x7fc6780654f0 msgr2=0x7fc678065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.233+0000 7fc67ef2f640 1 -- 172.21.15.71:0/2656719756 shutdown_connections 2023-08-25T10:05:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.233+0000 7fc67ef2f640 1 -- 172.21.15.71:0/2656719756 wait complete. 2023-08-25T10:05:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.234+0000 7fc67ef2f640 1 Processor -- start 2023-08-25T10:05:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.234+0000 7fc67ef2f640 1 -- start start 2023-08-25T10:05:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.235+0000 7fc67ef2f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668005140 0x7fc678148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.235+0000 7fc67ef2f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc678145cf0 0x7fc6781444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.235+0000 7fc67ef2f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 0x7fc678144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.235+0000 7fc67ef2f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678157d60 con 0x7fc67806a0e0 2023-08-25T10:05:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.235+0000 7fc67ef2f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678146e60 con 0x7fc678145cf0 2023-08-25T10:05:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.235+0000 7fc67ef2f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc678144030 con 0x7fc668005140 2023-08-25T10:05:39.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 0x7fc678144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 0x7fc678144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35262/0 (socket says 172.21.15.71:35262) 2023-08-25T10:05:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 -- 172.21.15.71:0/4085870078 learned_addr learned my addr 172.21.15.71:0/4085870078 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc677fff640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc678145cf0 0x7fc6781444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67cccb640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668005140 0x7fc678148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 -- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668005140 msgr2=0x7fc678148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668005140 0x7fc678148840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 -- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc678145cf0 msgr2=0x7fc6781444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc678145cf0 0x7fc6781444c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 -- 172.21.15.71:0/4085870078 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6781ee970 con 0x7fc67806a0e0 2023-08-25T10:05:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc67d4cc640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 0x7fc678144a00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc670037040 tx=0x7fc670009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.236+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670003b90 con 0x7fc67806a0e0 2023-08-25T10:05:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.237+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc670003d10 con 0x7fc67806a0e0 2023-08-25T10:05:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.237+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670034b80 con 0x7fc67806a0e0 2023-08-25T10:05:39.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.237+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6781eec50 con 0x7fc67806a0e0 2023-08-25T10:05:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.237+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc678158ca0 con 0x7fc67806a0e0 2023-08-25T10:05:39.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.238+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6781441a0 con 0x7fc67806a0e0 2023-08-25T10:05:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.239+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc670034ce0 con 0x7fc67806a0e0 2023-08-25T10:05:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.239+0000 7fc6757fa640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc65c06a050 0x7fc65c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.240+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc6700aa150 con 0x7fc67806a0e0 2023-08-25T10:05:39.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.240+0000 7fc67cccb640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc65c06a050 0x7fc65c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.241+0000 7fc67cccb640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc65c06a050 0x7fc65c06c510 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc66c007fd0 tx=0x7fc66c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.244+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc670075390 con 0x7fc67806a0e0 2023-08-25T10:05:39.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.366+0000 7f132c81f640 1 Processor -- start 2023-08-25T10:05:39.368 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.366+0000 7f132c81f640 1 -- start start 2023-08-25T10:05:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.368+0000 7f132c81f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13280c7740 0x7f132806a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.368+0000 7f132c81f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 0x7f13281592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.368+0000 7f132c81f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1328158e40 con 0x7f13281530c0 2023-08-25T10:05:39.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.368+0000 7f132c81f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132806bc80 con 0x7f132806aea0 2023-08-25T10:05:39.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.368+0000 7f132c81f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13281534a0 con 0x7f13280c7740 2023-08-25T10:05:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.369+0000 7f1326d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 0x7f13281592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.369+0000 7f1326d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 0x7f13281592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:51546/0 (socket says 172.21.15.71:51546) 2023-08-25T10:05:39.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.369+0000 7f1326d9d640 1 -- 172.21.15.71:0/2827817209 learned_addr learned my addr 172.21.15.71:0/2827817209 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:39.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.369+0000 7f131ffff640 1 --2- 172.21.15.71:0/2827817209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13280c7740 0x7f132806a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.370+0000 7f131ffff640 1 -- 172.21.15.71:0/2827817209 >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 msgr2=0x7f13281592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.370+0000 7f131ffff640 1 --2- 172.21.15.71:0/2827817209 >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 0x7f13281592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.370+0000 7f131ffff640 1 -- 172.21.15.71:0/2827817209 >> v1:172.21.15.164:6789/0 conn(0x7f13281530c0 legacy=0x7f13280c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.370+0000 7f131ffff640 1 -- 172.21.15.71:0/2827817209 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132815c330 con 0x7f13280c7740 2023-08-25T10:05:39.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.370+0000 7f1326d9d640 1 --2- 172.21.15.71:0/2827817209 >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 0x7f13281592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T10:05:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.371+0000 7f131ffff640 1 --2- 172.21.15.71:0/2827817209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13280c7740 0x7f132806a960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f13100097a0 tx=0x7f131002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a9fd81f7dedc7a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.371+0000 7f1325d9b640 1 -- 172.21.15.71:0/2827817209 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310030800 con 0x7f13280c7740 2023-08-25T10:05:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.371+0000 7f1325d9b640 1 -- 172.21.15.71:0/2827817209 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310030960 con 0x7f13280c7740 2023-08-25T10:05:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.371+0000 7f1325d9b640 1 -- 172.21.15.71:0/2827817209 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310030cf0 con 0x7f13280c7740 2023-08-25T10:05:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 -- 172.21.15.71:0/2827817209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13280c7740 msgr2=0x7f132806a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 --2- 172.21.15.71:0/2827817209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13280c7740 0x7f132806a960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f13100097a0 tx=0x7f131002b940 comp rx=0 tx=0).stop 2023-08-25T10:05:39.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 -- 172.21.15.71:0/2827817209 shutdown_connections 2023-08-25T10:05:39.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 --2- 172.21.15.71:0/2827817209 >> v2:172.21.15.164:3300/0 conn(0x7f132806aea0 0x7f13281592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 --2- 172.21.15.71:0/2827817209 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13280c7740 0x7f132806a960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 -- 172.21.15.71:0/2827817209 >> 172.21.15.71:0/2827817209 conn(0x7f13280654f0 msgr2=0x7f1328065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 -- 172.21.15.71:0/2827817209 shutdown_connections 2023-08-25T10:05:39.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.372+0000 7f132c81f640 1 -- 172.21.15.71:0/2827817209 wait complete. 2023-08-25T10:05:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.375+0000 7f132c81f640 1 Processor -- start 2023-08-25T10:05:39.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.375+0000 7f132c81f640 1 -- start start 2023-08-25T10:05:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f132c81f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f132806aea0 0x7f132815bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f132c81f640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 0x7f13281549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f132c81f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13281530c0 0x7f13281538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f132c81f640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1328155380 con 0x7f13280c7740 2023-08-25T10:05:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f132c81f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132815c140 con 0x7f132806aea0 2023-08-25T10:05:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f132c81f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1328156710 con 0x7f13281530c0 2023-08-25T10:05:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f131ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 0x7f13281549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f131ffff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 0x7f13281549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35278/0 (socket says 172.21.15.71:35278) 2023-08-25T10:05:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.377+0000 7f131ffff640 1 -- 172.21.15.71:0/103699086 learned_addr learned my addr 172.21.15.71:0/103699086 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f132659c640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f132806aea0 0x7f132815bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f1326d9d640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13281530c0 0x7f13281538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131ffff640 1 -- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13281530c0 msgr2=0x7f13281538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131ffff640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13281530c0 0x7f13281538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131ffff640 1 -- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f132806aea0 msgr2=0x7f132815bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131ffff640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f132806aea0 0x7f132815bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131ffff640 1 -- 172.21.15.71:0/103699086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1320009b30 con 0x7f13280c7740 2023-08-25T10:05:39.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f1326d9d640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13281530c0 0x7f13281538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:39.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131ffff640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 0x7f13281549c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f131002bf50 tx=0x7f1310004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131003e040 con 0x7f13280c7740 2023-08-25T10:05:39.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310039cb0 con 0x7f13280c7740 2023-08-25T10:05:39.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1310044630 con 0x7f13280c7740 2023-08-25T10:05:39.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.378+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1310009400 con 0x7f13280c7740 2023-08-25T10:05:39.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.379+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13281569b0 con 0x7f13280c7740 2023-08-25T10:05:39.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.380+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1328069160 con 0x7f13280c7740 2023-08-25T10:05:39.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.381+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1310043390 con 0x7f13280c7740 2023-08-25T10:05:39.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.382+0000 7f131effd640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f130806a050 0x7f130806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.382+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f13100aada0 con 0x7f13280c7740 2023-08-25T10:05:39.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.382+0000 7f132659c640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f130806a050 0x7f130806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.382+0000 7f132659c640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f130806a050 0x7f130806c510 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1318000f60 tx=0x7f1318009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.390+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1310075fc0 con 0x7f13280c7740 2023-08-25T10:05:39.447 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.445+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc678069160 con 0x7fc67806a0e0 2023-08-25T10:05:39.449 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.447+0000 7fc6757fa640 1 -- 172.21.15.71:0/4085870078 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20877 (secure 0 0 0) 0x7fc6700789b0 con 0x7fc67806a0e0 2023-08-25T10:05:39.449 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:39.450 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":69,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:59:40.756440+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6837/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.71:6834/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6832/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.71:6836/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6833/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.164:6835/2707317155":"2023-08-26T09:52:30.557077+0000","172.21.15.71:6835/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.164:6834/2707317155":"2023-08-26T09:52:30.557077+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":504,"length":2}]},{"pool":5,"snaps":[{"begin":504,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T10:05:39.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.449+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc65c06a050 msgr2=0x7fc65c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.449+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc65c06a050 0x7fc65c06c510 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc66c007fd0 tx=0x7fc66c009040 comp rx=0 tx=0).stop 2023-08-25T10:05:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.449+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 msgr2=0x7fc678144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.449+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 0x7fc678144a00 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc670037040 tx=0x7fc670009750 comp rx=0 tx=0).stop 2023-08-25T10:05:39.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 shutdown_connections 2023-08-25T10:05:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc678145cf0 0x7fc6781444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc65c06a050 0x7fc65c06c510 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc67806a0e0 0x7fc678144a00 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 --2- 172.21.15.71:0/4085870078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc668005140 0x7fc678148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 >> 172.21.15.71:0/4085870078 conn(0x7fc6780654f0 msgr2=0x7fc67814dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 shutdown_connections 2023-08-25T10:05:39.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.450+0000 7fc67ef2f640 1 -- 172.21.15.71:0/4085870078 wait complete. 2023-08-25T10:05:39.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T10:05:39.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T10:05:39.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.584+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1328159ce0 con 0x7f13280c7740 2023-08-25T10:05:39.586 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.585+0000 7f131effd640 1 -- 172.21.15.71:0/103699086 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13100795e0 con 0x7f13280c7740 2023-08-25T10:05:39.587 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T10:05:39.589 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.587+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f130806a050 msgr2=0x7f130806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.587+0000 7f132c81f640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f130806a050 0x7f130806c510 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1318000f60 tx=0x7f1318009040 comp rx=0 tx=0).stop 2023-08-25T10:05:39.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.587+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 msgr2=0x7f13281549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.587+0000 7f132c81f640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 0x7f13281549c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f131002bf50 tx=0x7f1310004730 comp rx=0 tx=0).stop 2023-08-25T10:05:39.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.587+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 shutdown_connections 2023-08-25T10:05:39.590 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f130806a050 0x7f130806c510 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13281530c0 0x7f13281538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f13280c7740 0x7f13281549c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 --2- 172.21.15.71:0/103699086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f132806aea0 0x7f132815bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 >> 172.21.15.71:0/103699086 conn(0x7f13280654f0 msgr2=0x7f132814ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 shutdown_connections 2023-08-25T10:05:39.592 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.588+0000 7f132c81f640 1 -- 172.21.15.71:0/103699086 wait complete. 2023-08-25T10:05:39.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T10:05:39.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.680+0000 7fb9d8abf640 1 Processor -- start 2023-08-25T10:05:39.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.680+0000 7fb9d8abf640 1 -- start start 2023-08-25T10:05:39.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d8abf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb9d415ce50 0x7fb9d40cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d8abf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d41591c0 con 0x7fb9d4152d80 2023-08-25T10:05:39.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d8abf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d41594a0 con 0x7fb9d40cacb0 2023-08-25T10:05:39.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d8abf640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d4159780 con 0x7fb9d415ce50 2023-08-25T10:05:39.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d259c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb9d4152d80 0x7fb9d40c9360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34678/0 (socket says 172.21.15.71:34678) 2023-08-25T10:05:39.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d259c640 1 -- 172.21.15.71:0/3862331101 learned_addr learned my addr 172.21.15.71:0/3862331101 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:39.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/3862331101 >> v2:172.21.15.164:3300/0 conn(0x7fb9d415ce50 0x7fb9d40cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.682+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725913715 0 0) 0x7fb9d41591c0 con 0x7fb9d4152d80 2023-08-25T10:05:39.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc003520 con 0x7fb9d4152d80 2023-08-25T10:05:39.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998697377 0 0) 0x7fb9bc003520 con 0x7fb9d4152d80 2023-08-25T10:05:39.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 >> v2:172.21.15.164:3300/0 conn(0x7fb9d415ce50 msgr2=0x7fb9d40cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 --2- 172.21.15.71:0/3862331101 >> v2:172.21.15.164:3300/0 conn(0x7fb9d415ce50 0x7fb9d40cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 >> v1:172.21.15.164:6789/0 conn(0x7fb9d40cacb0 legacy=0x7fb9d415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4159a60 con 0x7fb9d4152d80 2023-08-25T10:05:39.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fb9c4001670 con 0x7fb9d4152d80 2023-08-25T10:05:39.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 >> v1:172.21.15.71:6789/0 conn(0x7fb9d4152d80 legacy=0x7fb9d40c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9bc0039e0 0x7fb9bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0047c0 0x7fb9bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc0051f0 0x7fb9bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4159780 con 0x7fb9bc0051f0 2023-08-25T10:05:39.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc00c140 con 0x7fb9bc0047c0 2023-08-25T10:05:39.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc00c300 con 0x7fb9bc0039e0 2023-08-25T10:05:39.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.683+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc0051f0 0x7fb9bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d1d9b640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0047c0 0x7fb9bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d259c640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9bc0039e0 0x7fb9bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9bc0039e0 msgr2=0x7fb9bc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9bc0039e0 0x7fb9bc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0047c0 msgr2=0x7fb9bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0047c0 0x7fb9bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.684+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/3862331101 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc003520 con 0x7fb9bc0051f0 2023-08-25T10:05:39.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.685+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/3862331101 reap_dead start 2023-08-25T10:05:39.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.685+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc0051f0 0x7fb9bc00b980 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc007360 tx=0x7fb9bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80493d0a8f9c96d server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.685+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc02ccf0 con 0x7fb9bc0051f0 2023-08-25T10:05:39.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.685+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc02ce50 con 0x7fb9bc0051f0 2023-08-25T10:05:39.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.685+0000 7fb9d159a640 1 -- 172.21.15.71:0/3862331101 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc003bd0 con 0x7fb9bc0051f0 2023-08-25T10:05:39.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 -- 172.21.15.71:0/3862331101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc0051f0 msgr2=0x7fb9bc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc0051f0 0x7fb9bc00b980 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc007360 tx=0x7fb9bc0057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 -- 172.21.15.71:0/3862331101 shutdown_connections 2023-08-25T10:05:39.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/3862331101 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9bc0051f0 0x7fb9bc00b980 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 -- 172.21.15.71:0/3862331101 >> 172.21.15.71:0/3862331101 conn(0x7fb9d406b8f0 msgr2=0x7fb9d40654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 -- 172.21.15.71:0/3862331101 shutdown_connections 2023-08-25T10:05:39.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.686+0000 7fb9d8abf640 1 -- 172.21.15.71:0/3862331101 wait complete. 2023-08-25T10:05:39.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.688+0000 7fb9d8abf640 1 Processor -- start 2023-08-25T10:05:39.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.688+0000 7fb9d8abf640 1 -- start start 2023-08-25T10:05:39.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d8abf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0051f0 0x7fb9d40b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d8abf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9d40b42a0 0x7fb9d40c6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d8abf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 0x7fb9d41f1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d8abf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d414e120 con 0x7fb9d415c110 2023-08-25T10:05:39.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d8abf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4147af0 con 0x7fb9bc0051f0 2023-08-25T10:05:39.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d8abf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d4147530 con 0x7fb9d40b42a0 2023-08-25T10:05:39.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d2d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 0x7fb9d41f1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d2d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 0x7fb9d41f1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35286/0 (socket says 172.21.15.71:35286) 2023-08-25T10:05:39.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.689+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/4236085731 learned_addr learned my addr 172.21.15.71:0/4236085731 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d259c640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0051f0 0x7fb9d40b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d1d9b640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9d40b42a0 0x7fb9d40c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9d40b42a0 msgr2=0x7fb9d40c6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9d40b42a0 0x7fb9d40c6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0051f0 msgr2=0x7fb9d40b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0051f0 0x7fb9d40b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d2d9d640 1 -- 172.21.15.71:0/4236085731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d4147810 con 0x7fb9d415c110 2023-08-25T10:05:39.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d259c640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0051f0 0x7fb9d40b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:39.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d1d9b640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9d40b42a0 0x7fb9d40c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:39.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d2d9d640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 0x7fb9d41f1660 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc02dfd0 tx=0x7fb9cc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc039070 con 0x7fb9d415c110 2023-08-25T10:05:39.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9cc003e30 con 0x7fb9d415c110 2023-08-25T10:05:39.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d414dad0 con 0x7fb9d415c110 2023-08-25T10:05:39.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.690+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc0044b0 con 0x7fb9d415c110 2023-08-25T10:05:39.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.691+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d4156d50 con 0x7fb9d415c110 2023-08-25T10:05:39.700 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.692+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d40b1a60 con 0x7fb9d415c110 2023-08-25T10:05:39.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.693+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb9cc03d070 con 0x7fb9d415c110 2023-08-25T10:05:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.694+0000 7fb9c2ffd640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb9b406a050 0x7fb9b406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.694+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb9cc0a9eb0 con 0x7fb9d415c110 2023-08-25T10:05:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.694+0000 7fb9d259c640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb9b406a050 0x7fb9b406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.702 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.695+0000 7fb9d259c640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb9b406a050 0x7fb9b406c510 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40018e0 tx=0x7fb9c4001850 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.703 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.698+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb9cc074fb0 con 0x7fb9d415c110 2023-08-25T10:05:39.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.801+0000 7f05d6a63640 1 Processor -- start 2023-08-25T10:05:39.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.802+0000 7f05d6a63640 1 -- start start 2023-08-25T10:05:39.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.803+0000 7f05d6a63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d01576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.803+0000 7f05d6a63640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05d0069850 0x7f05d00c7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.803+0000 7f05d6a63640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f05d00c75c0 0x7f05d00c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.803+0000 7f05d6a63640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d00c4b60 con 0x7f05d0069850 2023-08-25T10:05:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.803+0000 7f05d6a63640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d0157bf0 con 0x7f05d00c75c0 2023-08-25T10:05:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.803+0000 7f05d6a63640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d006a130 con 0x7f05d01542d0 2023-08-25T10:05:39.806 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05d0069850 0x7f05d00c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f05d0069850 0x7f05d00c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35292/0 (socket says 172.21.15.71:35292) 2023-08-25T10:05:39.807 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 -- 172.21.15.71:0/1053752639 learned_addr learned my addr 172.21.15.71:0/1053752639 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:39.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5a61640 1 --2- 172.21.15.71:0/1053752639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d01576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.808 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d6262640 1 --2- 172.21.15.71:0/1053752639 >> v2:172.21.15.164:3300/0 conn(0x7f05d00c75c0 0x7f05d00c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 -- 172.21.15.71:0/1053752639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 msgr2=0x7f05d01576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 --2- 172.21.15.71:0/1053752639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d01576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 -- 172.21.15.71:0/1053752639 >> v2:172.21.15.164:3300/0 conn(0x7f05d00c75c0 msgr2=0x7f05d00c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:39.809 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 --2- 172.21.15.71:0/1053752639 >> v2:172.21.15.164:3300/0 conn(0x7f05d00c75c0 0x7f05d00c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d5260640 1 -- 172.21.15.71:0/1053752639 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d01546b0 con 0x7f05d0069850 2023-08-25T10:05:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.804+0000 7f05d6262640 1 --2- 172.21.15.71:0/1053752639 >> v2:172.21.15.164:3300/0 conn(0x7f05d00c75c0 0x7f05d00c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.805+0000 7f05d5260640 1 --2- 172.21.15.71:0/1053752639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c7080 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f05d00719d0 tx=0x7f05c002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e9c0b5dd3253f5 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d4a5f640 1 -- 172.21.15.71:0/1053752639 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0030850 con 0x7f05d0069850 2023-08-25T10:05:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d4a5f640 1 -- 172.21.15.71:0/1053752639 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c00309b0 con 0x7f05d0069850 2023-08-25T10:05:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d4a5f640 1 -- 172.21.15.71:0/1053752639 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0030d40 con 0x7f05d0069850 2023-08-25T10:05:39.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 -- 172.21.15.71:0/1053752639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 msgr2=0x7f05d00c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 --2- 172.21.15.71:0/1053752639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c7080 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f05d00719d0 tx=0x7f05c002b990 comp rx=0 tx=0).stop 2023-08-25T10:05:39.812 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 -- 172.21.15.71:0/1053752639 shutdown_connections 2023-08-25T10:05:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 --2- 172.21.15.71:0/1053752639 >> v2:172.21.15.164:3300/0 conn(0x7f05d00c75c0 0x7f05d00c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 --2- 172.21.15.71:0/1053752639 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c7080 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 --2- 172.21.15.71:0/1053752639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d01576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.806+0000 7f05d6a63640 1 -- 172.21.15.71:0/1053752639 >> 172.21.15.71:0/1053752639 conn(0x7f05d006c280 msgr2=0x7f05d006c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.807+0000 7f05d6a63640 1 -- 172.21.15.71:0/1053752639 shutdown_connections 2023-08-25T10:05:39.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.807+0000 7f05d6a63640 1 -- 172.21.15.71:0/1053752639 wait complete. 2023-08-25T10:05:39.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.808+0000 7f05d6a63640 1 Processor -- start 2023-08-25T10:05:39.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.808+0000 7f05d6a63640 1 -- start start 2023-08-25T10:05:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d6a63640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d6a63640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d00c75c0 0x7f05d00c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d6a63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d00c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d6a63640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d01e35a0 con 0x7f05d0069850 2023-08-25T10:05:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d6a63640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d00c08d0 con 0x7f05d00c75c0 2023-08-25T10:05:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d6a63640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d00c98d0 con 0x7f05d01542d0 2023-08-25T10:05:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.809+0000 7f05d5a61640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:35298/0 (socket says 172.21.15.71:35298) 2023-08-25T10:05:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 -- 172.21.15.71:0/3231992180 learned_addr learned my addr 172.21.15.71:0/3231992180 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5260640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d00c75c0 0x7f05d00c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d6262640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d00c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 -- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 msgr2=0x7f05d00c2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d00c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 -- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d00c75c0 msgr2=0x7f05d00c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d00c75c0 0x7f05d00c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 -- 172.21.15.71:0/3231992180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c00094d0 con 0x7f05d0069850 2023-08-25T10:05:39.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5260640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d00c75c0 0x7f05d00c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:39.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d5a61640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c9390 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00c820 tx=0x7f05cc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc004220 con 0x7f05d0069850 2023-08-25T10:05:39.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc0043a0 con 0x7f05d0069850 2023-08-25T10:05:39.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc007a70 con 0x7f05d0069850 2023-08-25T10:05:39.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.810+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d01ee8e0 con 0x7f05d0069850 2023-08-25T10:05:39.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.811+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d01eee10 con 0x7f05d0069850 2023-08-25T10:05:39.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.812+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d004e420 con 0x7f05d0069850 2023-08-25T10:05:39.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.813+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f05cc007bd0 con 0x7f05d0069850 2023-08-25T10:05:39.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.814+0000 7f05c67fc640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f05b406a050 0x7f05b406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:39.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.814+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f05cc088960 con 0x7f05d0069850 2023-08-25T10:05:39.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.814+0000 7f05d5260640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f05b406a050 0x7f05b406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:39.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.815+0000 7f05d5260640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f05b406a050 0x7f05b406c510 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05c0036000 tx=0x7f05c003b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:39.825 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.818+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f05cc053a60 con 0x7f05d0069850 2023-08-25T10:05:39.886 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.884+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9d4069160 con 0x7fb9d415c110 2023-08-25T10:05:39.887 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.885+0000 7fb9c2ffd640 1 -- 172.21.15.71:0/4236085731 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9cc044020 con 0x7fb9d415c110 2023-08-25T10:05:39.887 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T10:05:39.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb9b406a050 msgr2=0x7fb9b406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb9b406a050 0x7fb9b406c510 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40018e0 tx=0x7fb9c4001850 comp rx=0 tx=0).stop 2023-08-25T10:05:39.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 msgr2=0x7fb9d41f1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:39.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 0x7fb9d41f1660 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc02dfd0 tx=0x7fb9cc004ab0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 shutdown_connections 2023-08-25T10:05:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb9b406a050 0x7fb9b406c510 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb9d415c110 0x7fb9d41f1660 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb9d40b42a0 0x7fb9d40c6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 --2- 172.21.15.71:0/4236085731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9bc0051f0 0x7fb9d40b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:39.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 >> 172.21.15.71:0/4236085731 conn(0x7fb9d406b8f0 msgr2=0x7fb9d40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:39.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 shutdown_connections 2023-08-25T10:05:39.892 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:39.888+0000 7fb9d8abf640 1 -- 172.21.15.71:0/4236085731 wait complete. 2023-08-25T10:05:39.908 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T10:05:40.013 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.011+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f05d0069f10 con 0x7f05d0069850 2023-08-25T10:05:40.014 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.012+0000 7f05c67fc640 1 -- 172.21.15.71:0/3231992180 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v69) v1 ==== 103+0+11 (secure 0 0 0) 0x7f05cc01e0a0 con 0x7f05d0069850 2023-08-25T10:05:40.014 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:40.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.014+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f05b406a050 msgr2=0x7f05b406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.014+0000 7f05d6a63640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f05b406a050 0x7f05b406c510 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05c0036000 tx=0x7f05c003b040 comp rx=0 tx=0).stop 2023-08-25T10:05:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.014+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 msgr2=0x7f05d00c9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.014+0000 7f05d6a63640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c9390 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00c820 tx=0x7f05cc00cce0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 shutdown_connections 2023-08-25T10:05:40.017 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d00c75c0 0x7f05d00c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f05b406a050 0x7f05b406c510 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d01542d0 0x7f05d00c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 --2- 172.21.15.71:0/3231992180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f05d0069850 0x7f05d00c9390 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 >> 172.21.15.71:0/3231992180 conn(0x7f05d006c280 msgr2=0x7f05d01dccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 shutdown_connections 2023-08-25T10:05:40.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.015+0000 7f05d6a63640 1 -- 172.21.15.71:0/3231992180 wait complete. 2023-08-25T10:05:40.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T10:05:40.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.106+0000 7f01490c5640 1 Processor -- start 2023-08-25T10:05:40.108 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.106+0000 7f01490c5640 1 -- start start 2023-08-25T10:05:40.109 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.107+0000 7f01490c5640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f01440c3a40 0x7f01440c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.107+0000 7f01490c5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01440c1dc0 con 0x7f01441509d0 2023-08-25T10:05:40.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.107+0000 7f01490c5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0144156e00 con 0x7f01440c3660 2023-08-25T10:05:40.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.107+0000 7f01490c5640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0144157a80 con 0x7f01440c3a40 2023-08-25T10:05:40.110 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f01440c3a40 0x7f01440c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014359e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f01441509d0 0x7f0144150de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44134/0 (socket says 172.21.15.71:44134) 2023-08-25T10:05:40.111 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014259c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f01440c3a40 0x7f01440c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59768/0 (socket says 172.21.15.71:59768) 2023-08-25T10:05:40.112 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014359e640 1 -- 172.21.15.71:0/2766760574 learned_addr learned my addr 172.21.15.71:0/2766760574 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:40.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014259c640 1 -- 172.21.15.71:0/2766760574 >> v1:172.21.15.164:6789/0 conn(0x7f01440c3660 legacy=0x7f014406ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014259c640 1 -- 172.21.15.71:0/2766760574 >> v1:172.21.15.71:6789/0 conn(0x7f01441509d0 legacy=0x7f0144150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.113 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f0141d9b640 1 -- 172.21.15.71:0/2766760574 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881979512 0 0) 0x7f01440c1dc0 con 0x7f01441509d0 2023-08-25T10:05:40.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.108+0000 7f014259c640 1 -- 172.21.15.71:0/2766760574 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01441570e0 con 0x7f01440c3a40 2023-08-25T10:05:40.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.109+0000 7f014259c640 1 --2- 172.21.15.71:0/2766760574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3a40 0x7f01440c1710 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f01380097a0 tx=0x7f013802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7c64c6e188b120 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.110+0000 7f0141d9b640 1 -- 172.21.15.71:0/2766760574 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138030610 con 0x7f01440c3a40 2023-08-25T10:05:40.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.110+0000 7f0141d9b640 1 -- 172.21.15.71:0/2766760574 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138030770 con 0x7f01440c3a40 2023-08-25T10:05:40.114 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.110+0000 7f0141d9b640 1 -- 172.21.15.71:0/2766760574 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138030b00 con 0x7f01440c3a40 2023-08-25T10:05:40.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.110+0000 7f01490c5640 1 -- 172.21.15.71:0/2766760574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3a40 msgr2=0x7f01440c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.110+0000 7f01490c5640 1 --2- 172.21.15.71:0/2766760574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3a40 0x7f01440c1710 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f01380097a0 tx=0x7f013802b960 comp rx=0 tx=0).stop 2023-08-25T10:05:40.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.110+0000 7f01490c5640 1 -- 172.21.15.71:0/2766760574 shutdown_connections 2023-08-25T10:05:40.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.111+0000 7f01490c5640 1 --2- 172.21.15.71:0/2766760574 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3a40 0x7f01440c1710 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.115 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.111+0000 7f01490c5640 1 -- 172.21.15.71:0/2766760574 >> 172.21.15.71:0/2766760574 conn(0x7f01440654f0 msgr2=0x7f0144065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.111+0000 7f01490c5640 1 -- 172.21.15.71:0/2766760574 shutdown_connections 2023-08-25T10:05:40.116 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.111+0000 7f01490c5640 1 -- 172.21.15.71:0/2766760574 wait complete. 2023-08-25T10:05:40.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.112+0000 7f01490c5640 1 Processor -- start 2023-08-25T10:05:40.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.112+0000 7f01490c5640 1 -- start start 2023-08-25T10:05:40.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f01490c5640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 0x7f014415a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.118 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f01490c5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01440c3a40 0x7f01441f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f01490c5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01441509d0 0x7f01441ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f01490c5640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01441f23b0 con 0x7f01440c3660 2023-08-25T10:05:40.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f01490c5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01441f1da0 con 0x7f01440c3a40 2023-08-25T10:05:40.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f01490c5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0144158160 con 0x7f01441509d0 2023-08-25T10:05:40.119 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.113+0000 7f0142d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 0x7f014415a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 0x7f014415a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59782/0 (socket says 172.21.15.71:59782) 2023-08-25T10:05:40.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 -- 172.21.15.71:0/3097904994 learned_addr learned my addr 172.21.15.71:0/3097904994 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:40.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 -- 172.21.15.71:0/3097904994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01441509d0 msgr2=0x7f01441ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01441509d0 0x7f01441ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 -- 172.21.15.71:0/3097904994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01440c3a40 msgr2=0x7f01441f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01440c3a40 0x7f01441f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 -- 172.21.15.71:0/3097904994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0138009400 con 0x7f01440c3660 2023-08-25T10:05:40.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f0142d9d640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 0x7f014415a740 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f013400b3f0 tx=0x7f013400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134011830 con 0x7f01440c3660 2023-08-25T10:05:40.121 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.114+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0134011990 con 0x7f01440c3660 2023-08-25T10:05:40.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.115+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0144158440 con 0x7f01440c3660 2023-08-25T10:05:40.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.115+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0134011d40 con 0x7f01440c3660 2023-08-25T10:05:40.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.115+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0144158720 con 0x7f01440c3660 2023-08-25T10:05:40.122 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.116+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01440c4e80 con 0x7f01440c3660 2023-08-25T10:05:40.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.117+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f013401a400 con 0x7f01440c3660 2023-08-25T10:05:40.123 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.118+0000 7f012b7fe640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f013006a080 0x7f013006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.124 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.118+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f01340888e0 con 0x7f01440c3660 2023-08-25T10:05:40.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.118+0000 7f014259c640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f013006a080 0x7f013006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.125 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.119+0000 7f014259c640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f013006a080 0x7f013006c540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0138036000 tx=0x7f0138030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.126 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.122+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0134053a70 con 0x7f01440c3660 2023-08-25T10:05:40.233 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.232+0000 7fe67536b640 1 Processor -- start 2023-08-25T10:05:40.234 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.232+0000 7fe67536b640 1 -- start start 2023-08-25T10:05:40.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.233+0000 7fe67536b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 0x7fe67015cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.233+0000 7fe67536b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670158fe0 con 0x7fe67006c510 2023-08-25T10:05:40.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.233+0000 7fe67536b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67006aa70 con 0x7fe67015c7c0 2023-08-25T10:05:40.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.233+0000 7fe67536b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67006ad50 con 0x7fe6700bfaa0 2023-08-25T10:05:40.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.233+0000 7fe66f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe67006c510 0x7fe670152e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44138/0 (socket says 172.21.15.71:44138) 2023-08-25T10:05:40.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.233+0000 7fe66f7fe640 1 -- 172.21.15.71:0/2290926289 learned_addr learned my addr 172.21.15.71:0/2290926289 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:40.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66e7fc640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 0x7fe67015cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357418058 0 0) 0x7fe670158fe0 con 0x7fe67006c510 2023-08-25T10:05:40.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe65c003520 con 0x7fe67006c510 2023-08-25T10:05:40.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2610905020 0 0) 0x7fe65c003520 con 0x7fe67006c510 2023-08-25T10:05:40.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 msgr2=0x7fe67015cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 0x7fe67015cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 >> v1:172.21.15.164:6789/0 conn(0x7fe67015c7c0 legacy=0x7fe67006e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6701592a0 con 0x7fe67006c510 2023-08-25T10:05:40.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fe668001670 con 0x7fe67006c510 2023-08-25T10:05:40.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 >> v1:172.21.15.71:6789/0 conn(0x7fe67006c510 legacy=0x7fe670152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe65c0039a0 0x7fe65c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 0x7fe65c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe65c004d00 0x7fe65c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67006ad50 con 0x7fe65c0039a0 2023-08-25T10:05:40.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c00ba30 con 0x7fe65c004d00 2023-08-25T10:05:40.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.234+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c00bbf0 con 0x7fe6700bfaa0 2023-08-25T10:05:40.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66effd640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe65c0039a0 0x7fe65c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66f7fe640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe65c004d00 0x7fe65c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66e7fc640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 0x7fe65c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66effd640 1 -- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 msgr2=0x7fe65c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66effd640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6700bfaa0 0x7fe65c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66effd640 1 -- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe65c004d00 msgr2=0x7fe65c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66effd640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe65c004d00 0x7fe65c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.235+0000 7fe66effd640 1 -- 172.21.15.71:0/2290926289 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c003520 con 0x7fe65c0039a0 2023-08-25T10:05:40.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.236+0000 7fe66effd640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe65c0039a0 0x7fe65c003dc0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe660001e20 tx=0x7fe65c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da98b09fd2885ef7 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.248 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.236+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660003b70 con 0x7fe65c0039a0 2023-08-25T10:05:40.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.236+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe660003cd0 con 0x7fe65c0039a0 2023-08-25T10:05:40.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.236+0000 7fe66dffb640 1 -- 172.21.15.71:0/2290926289 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660005310 con 0x7fe65c0039a0 2023-08-25T10:05:40.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe67536b640 1 -- 172.21.15.71:0/2290926289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe65c0039a0 msgr2=0x7fe65c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.249 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe67536b640 1 --2- 172.21.15.71:0/2290926289 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe65c0039a0 0x7fe65c003dc0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe660001e20 tx=0x7fe65c0057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe66f7fe640 1 -- 172.21.15.71:0/2290926289 reap_dead start 2023-08-25T10:05:40.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe67536b640 1 -- 172.21.15.71:0/2290926289 shutdown_connections 2023-08-25T10:05:40.250 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe67536b640 1 -- 172.21.15.71:0/2290926289 >> 172.21.15.71:0/2290926289 conn(0x7fe6700654f0 msgr2=0x7fe6700658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe67536b640 1 -- 172.21.15.71:0/2290926289 shutdown_connections 2023-08-25T10:05:40.251 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.237+0000 7fe67536b640 1 -- 172.21.15.71:0/2290926289 wait complete. 2023-08-25T10:05:40.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.239+0000 7fe67536b640 1 Processor -- start 2023-08-25T10:05:40.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.239+0000 7fe67536b640 1 -- start start 2023-08-25T10:05:40.252 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe67536b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c0039a0 0x7fe6700c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe67536b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6701575e0 0x7fe6701eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.253 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe67536b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 0x7fe670148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe67536b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6701f1e10 con 0x7fe6701f18d0 2023-08-25T10:05:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe67536b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6701f20f0 con 0x7fe6701575e0 2023-08-25T10:05:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe67536b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6701f23d0 con 0x7fe65c0039a0 2023-08-25T10:05:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe66f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 0x7fe670148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.254 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe66f7fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 0x7fe670148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59804/0 (socket says 172.21.15.71:59804) 2023-08-25T10:05:40.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.240+0000 7fe66f7fe640 1 -- 172.21.15.71:0/3034949011 learned_addr learned my addr 172.21.15.71:0/3034949011 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:40.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66effd640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c0039a0 0x7fe6700c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66e7fc640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6701575e0 0x7fe6701eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66f7fe640 1 -- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c0039a0 msgr2=0x7fe6700c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66f7fe640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c0039a0 0x7fe6700c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66f7fe640 1 -- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6701575e0 msgr2=0x7fe6701eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66f7fe640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6701575e0 0x7fe6701eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66f7fe640 1 -- 172.21.15.71:0/3034949011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6701f26b0 con 0x7fe6701f18d0 2023-08-25T10:05:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66e7fc640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6701575e0 0x7fe6701eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:40.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66effd640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c0039a0 0x7fe6700c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe66f7fe640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 0x7fe670148fb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe670071bd0 tx=0x7fe66800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66800eed0 con 0x7fe6701f18d0 2023-08-25T10:05:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66800c610 con 0x7fe6701f18d0 2023-08-25T10:05:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66800c9c0 con 0x7fe6701f18d0 2023-08-25T10:05:40.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.241+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6701ee6b0 con 0x7fe6701f18d0 2023-08-25T10:05:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.242+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6701ee360 con 0x7fe6701f18d0 2023-08-25T10:05:40.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.243+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe670069160 con 0x7fe6701f18d0 2023-08-25T10:05:40.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.244+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe66800cb20 con 0x7fe6701f18d0 2023-08-25T10:05:40.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.245+0000 7fe6677fe640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe65006a050 0x7fe65006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.245+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe6680887d0 con 0x7fe6701f18d0 2023-08-25T10:05:40.259 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.245+0000 7fe66effd640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe65006a050 0x7fe65006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.246+0000 7fe66effd640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe65006a050 0x7fe65006c510 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe660034000 tx=0x7fe660009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.260 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.249+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe668053990 con 0x7fe6701f18d0 2023-08-25T10:05:40.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.313+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0144069160 con 0x7f01440c3660 2023-08-25T10:05:40.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.314+0000 7f012b7fe640 1 -- 172.21.15.71:0/3097904994 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0134057090 con 0x7f01440c3660 2023-08-25T10:05:40.316 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T10:05:40.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.316+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f013006a080 msgr2=0x7f013006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f013006a080 0x7f013006c540 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0138036000 tx=0x7f0138030000 comp rx=0 tx=0).stop 2023-08-25T10:05:40.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 msgr2=0x7f014415a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.319 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 0x7f014415a740 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f013400b3f0 tx=0x7f013400b8b0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 shutdown_connections 2023-08-25T10:05:40.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01440c3a40 0x7f01441f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f013006a080 0x7f013006c540 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01441509d0 0x7f01441ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.320 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 --2- 172.21.15.71:0/3097904994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f01440c3660 0x7f014415a740 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 >> 172.21.15.71:0/3097904994 conn(0x7f01440654f0 msgr2=0x7f01440658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 shutdown_connections 2023-08-25T10:05:40.321 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.317+0000 7f01490c5640 1 -- 172.21.15.71:0/3097904994 wait complete. 2023-08-25T10:05:40.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T10:05:40.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.451+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe6700c8400 con 0x7fe6701f18d0 2023-08-25T10:05:40.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.451+0000 7fe6677fe640 1 -- 172.21.15.71:0/3034949011 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v69) v1 ==== 80+0+492 (secure 0 0 0) 0x7fe668056fb0 con 0x7fe6701f18d0 2023-08-25T10:05:40.455 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6837/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:40.456 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6834/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:40.456 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6832/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:40.456 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6836/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:40.456 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6833/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:40.457 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6835/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:40.457 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6835/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:40.457 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6834/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:40.458 INFO:teuthology.orchestra.run.smithi071.stderr:listed 8 entries 2023-08-25T10:05:40.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.454+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe65006a050 msgr2=0x7fe65006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.454+0000 7fe67536b640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe65006a050 0x7fe65006c510 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe660034000 tx=0x7fe660009030 comp rx=0 tx=0).stop 2023-08-25T10:05:40.459 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.454+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 msgr2=0x7fe670148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.454+0000 7fe67536b640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 0x7fe670148fb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe670071bd0 tx=0x7fe66800eb90 comp rx=0 tx=0).stop 2023-08-25T10:05:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 shutdown_connections 2023-08-25T10:05:40.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6701575e0 0x7fe6701eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe65006a050 0x7fe65006c510 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6701f18d0 0x7fe670148fb0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 --2- 172.21.15.71:0/3034949011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c0039a0 0x7fe6700c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 >> 172.21.15.71:0/3034949011 conn(0x7fe6700654f0 msgr2=0x7fe6700cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 shutdown_connections 2023-08-25T10:05:40.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.455+0000 7fe67536b640 1 -- 172.21.15.71:0/3034949011 wait complete. 2023-08-25T10:05:40.475 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T10:05:40.475 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T10:05:40.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T10:05:40.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.538+0000 7fec8f555640 1 Processor -- start 2023-08-25T10:05:40.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.538+0000 7fec8f555640 1 -- start start 2023-08-25T10:05:40.546 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8f555640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec8806f620 0x7fec8806bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8f555640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec88159a60 con 0x7fec88153300 2023-08-25T10:05:40.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8f555640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec88159d40 con 0x7fec8806a6b0 2023-08-25T10:05:40.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8f555640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8806b920 con 0x7fec8806f620 2023-08-25T10:05:40.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8d2f1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fec88153300 0x7fec880c5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44142/0 (socket says 172.21.15.71:44142) 2023-08-25T10:05:40.547 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8d2f1640 1 -- 172.21.15.71:0/3841800953 learned_addr learned my addr 172.21.15.71:0/3841800953 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:40.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec8daf2640 1 --2- 172.21.15.71:0/3841800953 >> v2:172.21.15.164:3300/0 conn(0x7fec8806f620 0x7fec8806bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526800715 0 0) 0x7fec88159a60 con 0x7fec88153300 2023-08-25T10:05:40.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.541+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec74003520 con 0x7fec88153300 2023-08-25T10:05:40.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114747127 0 0) 0x7fec74003520 con 0x7fec88153300 2023-08-25T10:05:40.548 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 >> v2:172.21.15.164:3300/0 conn(0x7fec8806f620 msgr2=0x7fec8806bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 --2- 172.21.15.71:0/3841800953 >> v2:172.21.15.164:3300/0 conn(0x7fec8806f620 0x7fec8806bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 >> v1:172.21.15.164:6789/0 conn(0x7fec8806a6b0 legacy=0x7fec88070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec880698f0 con 0x7fec88153300 2023-08-25T10:05:40.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fec84001670 con 0x7fec88153300 2023-08-25T10:05:40.549 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 >> v1:172.21.15.71:6789/0 conn(0x7fec88153300 legacy=0x7fec880c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec740039e0 0x7fec74003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec740047c0 0x7fec74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec74005180 0x7fec7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7400c060 con 0x7fec740047c0 2023-08-25T10:05:40.550 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7400c1d0 con 0x7fec740039e0 2023-08-25T10:05:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7400c390 con 0x7fec74005180 2023-08-25T10:05:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec8caf0640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec740047c0 0x7fec74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.542+0000 7fec8d2f1640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec740039e0 0x7fec74003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8daf2640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec74005180 0x7fec7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8caf0640 1 -- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec74005180 msgr2=0x7fec7400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8caf0640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec74005180 0x7fec7400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8caf0640 1 -- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec740039e0 msgr2=0x7fec74003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8caf0640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec740039e0 0x7fec74003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8caf0640 1 -- 172.21.15.71:0/3841800953 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec74003520 con 0x7fec740047c0 2023-08-25T10:05:40.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.543+0000 7fec8daf2640 1 -- 172.21.15.71:0/3841800953 reap_dead start 2023-08-25T10:05:40.552 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.544+0000 7fec8caf0640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec740047c0 0x7fec74004c40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fec78001750 tx=0x7fec740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e15c4c536285e server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.544+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780052d0 con 0x7fec740047c0 2023-08-25T10:05:40.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.544+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7802e050 con 0x7fec740047c0 2023-08-25T10:05:40.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.544+0000 7fec7ffff640 1 -- 172.21.15.71:0/3841800953 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78003830 con 0x7fec740047c0 2023-08-25T10:05:40.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.544+0000 7fec8f555640 1 -- 172.21.15.71:0/3841800953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec740047c0 msgr2=0x7fec74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.553 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.544+0000 7fec8f555640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec740047c0 0x7fec74004c40 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fec78001750 tx=0x7fec740057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.545+0000 7fec8f555640 1 -- 172.21.15.71:0/3841800953 shutdown_connections 2023-08-25T10:05:40.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.545+0000 7fec8f555640 1 --2- 172.21.15.71:0/3841800953 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec740047c0 0x7fec74004c40 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.545+0000 7fec8f555640 1 -- 172.21.15.71:0/3841800953 >> 172.21.15.71:0/3841800953 conn(0x7fec880654f0 msgr2=0x7fec88065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.545+0000 7fec8f555640 1 -- 172.21.15.71:0/3841800953 shutdown_connections 2023-08-25T10:05:40.554 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.545+0000 7fec8f555640 1 -- 172.21.15.71:0/3841800953 wait complete. 2023-08-25T10:05:40.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.546+0000 7fec8f555640 1 Processor -- start 2023-08-25T10:05:40.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.546+0000 7fec8f555640 1 -- start start 2023-08-25T10:05:40.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.547+0000 7fec8f555640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec740047c0 0x7fec88145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.547+0000 7fec8f555640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec88148350 0x7fec881487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.557 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.547+0000 7fec8f555640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 0x7fec881ef020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.547+0000 7fec8f555640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8815a020 con 0x7fec881eeba0 2023-08-25T10:05:40.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.547+0000 7fec8f555640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec881f2d50 con 0x7fec88148350 2023-08-25T10:05:40.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.547+0000 7fec8f555640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec88158f10 con 0x7fec740047c0 2023-08-25T10:05:40.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 0x7fec881ef020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.558 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 0x7fec881ef020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59822/0 (socket says 172.21.15.71:59822) 2023-08-25T10:05:40.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 -- 172.21.15.71:0/3508848132 learned_addr learned my addr 172.21.15.71:0/3508848132 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:40.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8d2f1640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec740047c0 0x7fec88145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8caf0640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec88148350 0x7fec881487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 -- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec740047c0 msgr2=0x7fec88145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.559 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec740047c0 0x7fec88145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 -- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec88148350 msgr2=0x7fec881487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec88148350 0x7fec881487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 -- 172.21.15.71:0/3508848132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec88148cf0 con 0x7fec881eeba0 2023-08-25T10:05:40.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8caf0640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec88148350 0x7fec881487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:40.560 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8d2f1640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec740047c0 0x7fec88145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:40.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.548+0000 7fec8daf2640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 0x7fec881ef020 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fec88074280 tx=0x7fec8000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.549+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80016620 con 0x7fec881eeba0 2023-08-25T10:05:40.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.549+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec800167a0 con 0x7fec881eeba0 2023-08-25T10:05:40.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.549+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80016b50 con 0x7fec881eeba0 2023-08-25T10:05:40.561 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.549+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec881ef560 con 0x7fec881eeba0 2023-08-25T10:05:40.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.549+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec88159160 con 0x7fec881eeba0 2023-08-25T10:05:40.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.550+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec88159380 con 0x7fec881eeba0 2023-08-25T10:05:40.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.551+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fec80016cb0 con 0x7fec881eeba0 2023-08-25T10:05:40.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.553+0000 7fec7dffb640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fec6c06a050 0x7fec6c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.562 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.553+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fec80089940 con 0x7fec881eeba0 2023-08-25T10:05:40.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.553+0000 7fec8d2f1640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fec6c06a050 0x7fec6c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.553+0000 7fec8d2f1640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fec6c06a050 0x7fec6c06c510 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fec84001eb0 tx=0x7fec84000f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.563 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.556+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fec80054ad0 con 0x7fec881eeba0 2023-08-25T10:05:40.726 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.724+0000 7f5e218b4640 1 Processor -- start 2023-08-25T10:05:40.727 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.724+0000 7f5e218b4640 1 -- start start 2023-08-25T10:05:40.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e218b4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c15cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.728 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e218b4640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5e1c159830 0x7f5e1c069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e218b4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c159c80 con 0x7f5e1c06a530 2023-08-25T10:05:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e218b4640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c15ca50 con 0x7f5e1c159830 2023-08-25T10:05:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e218b4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c14fc80 con 0x7f5e1c06bed0 2023-08-25T10:05:40.729 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e1b7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5e1c159830 0x7f5e1c069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e1b7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5e1c159830 0x7f5e1c069850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59836/0 (socket says 172.21.15.71:59836) 2023-08-25T10:05:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e1b7fe640 1 -- 172.21.15.71:0/2005168122 learned_addr learned my addr 172.21.15.71:0/2005168122 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.726+0000 7f5e1a7fc640 1 --2- 172.21.15.71:0/2005168122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c15cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.730 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.727+0000 7f5e1b7fe640 1 -- 172.21.15.71:0/2005168122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 msgr2=0x7f5e1c15cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.727+0000 7f5e1b7fe640 1 --2- 172.21.15.71:0/2005168122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c15cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.727+0000 7f5e1b7fe640 1 -- 172.21.15.71:0/2005168122 >> v1:172.21.15.164:6789/0 conn(0x7f5e1c06a530 legacy=0x7f5e1c0c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.731 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.727+0000 7f5e1b7fe640 1 -- 172.21.15.71:0/2005168122 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c06dae0 con 0x7f5e1c159830 2023-08-25T10:05:40.732 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.728+0000 7f5e1b7fe640 1 --2- 172.21.15.71:0/2005168122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c159830 0x7f5e1c069850 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009870 tx=0x7f5e1402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5108f55682238839 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e19ffb640 1 -- 172.21.15.71:0/2005168122 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14031640 con 0x7f5e1c159830 2023-08-25T10:05:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e19ffb640 1 -- 172.21.15.71:0/2005168122 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e140317a0 con 0x7f5e1c159830 2023-08-25T10:05:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e19ffb640 1 -- 172.21.15.71:0/2005168122 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14031b30 con 0x7f5e1c159830 2023-08-25T10:05:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e218b4640 1 -- 172.21.15.71:0/2005168122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c159830 msgr2=0x7f5e1c069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.733 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2005168122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c159830 0x7f5e1c069850 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009870 tx=0x7f5e1402b8f0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e218b4640 1 -- 172.21.15.71:0/2005168122 shutdown_connections 2023-08-25T10:05:40.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2005168122 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c159830 0x7f5e1c069850 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.734 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2005168122 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c15cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.729+0000 7f5e218b4640 1 -- 172.21.15.71:0/2005168122 >> 172.21.15.71:0/2005168122 conn(0x7f5e1c0654f0 msgr2=0x7f5e1c065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.730+0000 7f5e218b4640 1 -- 172.21.15.71:0/2005168122 shutdown_connections 2023-08-25T10:05:40.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.730+0000 7f5e218b4640 1 -- 172.21.15.71:0/2005168122 wait complete. 2023-08-25T10:05:40.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.731+0000 7f5e218b4640 1 Processor -- start 2023-08-25T10:05:40.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.731+0000 7f5e218b4640 1 -- start start 2023-08-25T10:05:40.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e218b4640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 0x7f5e1c1492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e218b4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c1ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e218b4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e1c159830 0x7f5e1c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e218b4640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c0c4fd0 con 0x7f5e1c06a530 2023-08-25T10:05:40.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e218b4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c145c00 con 0x7f5e1c159830 2023-08-25T10:05:40.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e218b4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c1edf90 con 0x7f5e1c06bed0 2023-08-25T10:05:40.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.732+0000 7f5e1affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 0x7f5e1c1492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 0x7f5e1c1492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59842/0 (socket says 172.21.15.71:59842) 2023-08-25T10:05:40.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 -- 172.21.15.71:0/2841048879 learned_addr learned my addr 172.21.15.71:0/2841048879 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:40.740 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1a7fc640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c1ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1b7fe640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e1c159830 0x7f5e1c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 -- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 msgr2=0x7f5e1c1ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c1ee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 -- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e1c159830 msgr2=0x7f5e1c152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.741 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e1c159830 0x7f5e1c152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 -- 172.21.15.71:0/2841048879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140094d0 con 0x7f5e1c06a530 2023-08-25T10:05:40.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e1affd640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 0x7f5e1c1492f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00c8a0 tx=0x7f5e0c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c003b70 con 0x7f5e1c06a530 2023-08-25T10:05:40.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.733+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0c003cf0 con 0x7f5e1c06a530 2023-08-25T10:05:40.742 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.734+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c0c4a50 con 0x7f5e1c06a530 2023-08-25T10:05:40.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.734+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c00aa90 con 0x7f5e1c06a530 2023-08-25T10:05:40.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.734+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c0c4cf0 con 0x7f5e1c06a530 2023-08-25T10:05:40.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.735+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c069160 con 0x7f5e1c06a530 2023-08-25T10:05:40.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.736+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5e0c00abf0 con 0x7f5e1c06a530 2023-08-25T10:05:40.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.736+0000 7f5e03fff640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5e0806a050 0x7f5e0806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:40.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.737+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f5e0c088790 con 0x7f5e1c06a530 2023-08-25T10:05:40.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.737+0000 7f5e1a7fc640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5e0806a050 0x7f5e0806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:40.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.738+0000 7f5e1a7fc640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5e0806a050 0x7f5e0806c510 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c149830 tx=0x7f5e10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:40.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.741+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f5e0c0539c0 con 0x7f5e1c06a530 2023-08-25T10:05:40.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.749+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec88069160 con 0x7fec881eeba0 2023-08-25T10:05:40.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.749+0000 7fec7dffb640 1 -- 172.21.15.71:0/3508848132 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec800580f0 con 0x7fec881eeba0 2023-08-25T10:05:40.753 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:40.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.751+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fec6c06a050 msgr2=0x7fec6c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.751+0000 7fec8f555640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fec6c06a050 0x7fec6c06c510 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fec84001eb0 tx=0x7fec84000f10 comp rx=0 tx=0).stop 2023-08-25T10:05:40.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 msgr2=0x7fec881ef020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 0x7fec881ef020 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fec88074280 tx=0x7fec8000f520 comp rx=0 tx=0).stop 2023-08-25T10:05:40.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 shutdown_connections 2023-08-25T10:05:40.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec88148350 0x7fec881487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fec6c06a050 0x7fec6c06c510 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fec881eeba0 0x7fec881ef020 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 --2- 172.21.15.71:0/3508848132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec740047c0 0x7fec88145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 >> 172.21.15.71:0/3508848132 conn(0x7fec880654f0 msgr2=0x7fec881513b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 shutdown_connections 2023-08-25T10:05:40.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.752+0000 7fec8f555640 1 -- 172.21.15.71:0/3508848132 wait complete. 2023-08-25T10:05:40.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T10:05:40.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.936+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e1c159c80 con 0x7f5e1c06a530 2023-08-25T10:05:40.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.943+0000 7f5e03fff640 1 -- 172.21.15.71:0/2841048879 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20877 (secure 0 0 0) 0x7f5e1c159c80 con 0x7f5e1c06a530 2023-08-25T10:05:40.946 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:40.946 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":69,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:59:40.756440+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6837/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.71:6834/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6832/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.71:6836/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6833/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.164:6835/2707317155":"2023-08-26T09:52:30.557077+0000","172.21.15.71:6835/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.164:6834/2707317155":"2023-08-26T09:52:30.557077+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":504,"length":2}]},{"pool":5,"snaps":[{"begin":504,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T10:05:40.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5e0806a050 msgr2=0x7f5e0806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.947 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5e0806a050 0x7f5e0806c510 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c149830 tx=0x7f5e10008040 comp rx=0 tx=0).stop 2023-08-25T10:05:40.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 msgr2=0x7f5e1c1492f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:40.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 0x7f5e1c1492f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00c8a0 tx=0x7f5e0c00cd60 comp rx=0 tx=0).stop 2023-08-25T10:05:40.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 shutdown_connections 2023-08-25T10:05:40.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e1c159830 0x7f5e1c152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.948 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f5e0806a050 0x7f5e0806c510 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e1c06bed0 0x7f5e1c1ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 --2- 172.21.15.71:0/2841048879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5e1c06a530 0x7f5e1c1492f0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:40.949 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.945+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 >> 172.21.15.71:0/2841048879 conn(0x7f5e1c0654f0 msgr2=0x7f5e1c15b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:40.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.946+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 shutdown_connections 2023-08-25T10:05:40.950 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:40.946+0000 7f5e218b4640 1 -- 172.21.15.71:0/2841048879 wait complete. 2023-08-25T10:05:40.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T10:05:40.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T10:05:41.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.022+0000 7f3712226640 1 Processor -- start 2023-08-25T10:05:41.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.022+0000 7f3712226640 1 -- start start 2023-08-25T10:05:41.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f3712226640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f370c152d80 0x7f370c0c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f3712226640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c0c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f3712226640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f370c15c740 0x7f370c15cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f3712226640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c15d0d0 con 0x7f370c152d80 2023-08-25T10:05:41.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f3712226640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c1591c0 con 0x7f370c15c740 2023-08-25T10:05:41.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f3712226640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1594a0 con 0x7f370c0cacb0 2023-08-25T10:05:41.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f370b7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f370c152d80 0x7f370c0c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f370b7fe640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f370c152d80 0x7f370c0c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59858/0 (socket says 172.21.15.71:59858) 2023-08-25T10:05:41.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f370b7fe640 1 -- 172.21.15.71:0/3566385625 learned_addr learned my addr 172.21.15.71:0/3566385625 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f370affd640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c0c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.024+0000 7f370bfff640 1 --2- 172.21.15.71:0/3566385625 >> v2:172.21.15.164:3300/0 conn(0x7f370c15c740 0x7f370c15cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370b7fe640 1 -- 172.21.15.71:0/3566385625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 msgr2=0x7f370c0c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370b7fe640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c0c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370b7fe640 1 -- 172.21.15.71:0/3566385625 >> v2:172.21.15.164:3300/0 conn(0x7f370c15c740 msgr2=0x7f370c15cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370b7fe640 1 --2- 172.21.15.71:0/3566385625 >> v2:172.21.15.164:3300/0 conn(0x7f370c15c740 0x7f370c15cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370b7fe640 1 -- 172.21.15.71:0/3566385625 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c159780 con 0x7f370c152d80 2023-08-25T10:05:41.030 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370affd640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:41.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.025+0000 7f370bfff640 1 --2- 172.21.15.71:0/3566385625 >> v2:172.21.15.164:3300/0 conn(0x7f370c15c740 0x7f370c15cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:41.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.026+0000 7f370b7fe640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c0c9360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f370c0b2670 tx=0x7f36fc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f566318c3b0df36 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.026+0000 7f370a7fc640 1 -- 172.21.15.71:0/3566385625 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc030850 con 0x7f370c152d80 2023-08-25T10:05:41.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.026+0000 7f370a7fc640 1 -- 172.21.15.71:0/3566385625 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36fc0309b0 con 0x7f370c152d80 2023-08-25T10:05:41.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f370a7fc640 1 -- 172.21.15.71:0/3566385625 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36fc030d40 con 0x7f370c152d80 2023-08-25T10:05:41.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 -- 172.21.15.71:0/3566385625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 msgr2=0x7f370c0c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c0c9360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f370c0b2670 tx=0x7f36fc02b990 comp rx=0 tx=0).stop 2023-08-25T10:05:41.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 -- 172.21.15.71:0/3566385625 shutdown_connections 2023-08-25T10:05:41.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 --2- 172.21.15.71:0/3566385625 >> v2:172.21.15.164:3300/0 conn(0x7f370c15c740 0x7f370c15cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c0c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 --2- 172.21.15.71:0/3566385625 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c0c9360 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 -- 172.21.15.71:0/3566385625 >> 172.21.15.71:0/3566385625 conn(0x7f370c06b8f0 msgr2=0x7f370c0654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 -- 172.21.15.71:0/3566385625 shutdown_connections 2023-08-25T10:05:41.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.027+0000 7f3712226640 1 -- 172.21.15.71:0/3566385625 wait complete. 2023-08-25T10:05:41.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.029+0000 7f3712226640 1 Processor -- start 2023-08-25T10:05:41.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.029+0000 7f3712226640 1 -- start start 2023-08-25T10:05:41.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f3712226640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f3712226640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c13ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f3712226640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f370c15c740 0x7f370c13f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f3712226640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c150020 con 0x7f370c152d80 2023-08-25T10:05:41.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f3712226640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1ee9d0 con 0x7f370c15c740 2023-08-25T10:05:41.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f3712226640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c1eec90 con 0x7f370c0cacb0 2023-08-25T10:05:41.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f370affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c13ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f370affd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c13ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59860/0 (socket says 172.21.15.71:59860) 2023-08-25T10:05:41.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.030+0000 7f370affd640 1 -- 172.21.15.71:0/4210405182 learned_addr learned my addr 172.21.15.71:0/4210405182 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370affd640 1 -- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 msgr2=0x7f370c143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370affd640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370affd640 1 -- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f370c15c740 msgr2=0x7f370c13f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370bfff640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f370c15c740 0x7f370c13f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370affd640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f370c15c740 0x7f370c13f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370affd640 1 -- 172.21.15.71:0/4210405182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc0094d0 con 0x7f370c152d80 2023-08-25T10:05:41.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f370affd640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c13ef40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f370000c7f0 tx=0x7f370000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700004220 con 0x7f370c152d80 2023-08-25T10:05:41.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700004380 con 0x7f370c152d80 2023-08-25T10:05:41.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c1eef70 con 0x7f370c152d80 2023-08-25T10:05:41.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700007860 con 0x7f370c152d80 2023-08-25T10:05:41.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.031+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c1ef530 con 0x7f370c152d80 2023-08-25T10:05:41.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.033+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c0b1a60 con 0x7f370c152d80 2023-08-25T10:05:41.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.033+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f3700007bd0 con 0x7f370c152d80 2023-08-25T10:05:41.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.034+0000 7f36f3fff640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36f806a050 0x7f36f806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.034+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f3700088b70 con 0x7f370c152d80 2023-08-25T10:05:41.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.035+0000 7f370b7fe640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36f806a050 0x7f36f806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.035+0000 7f370b7fe640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36f806a050 0x7f36f806c510 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f36fc036000 tx=0x7f36fc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.039+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f3700053be0 con 0x7f370c152d80 2023-08-25T10:05:41.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.155+0000 7f8ec0912640 1 Processor -- start 2023-08-25T10:05:41.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.155+0000 7f8ec0912640 1 -- start start 2023-08-25T10:05:41.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8ec0912640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 0x7f8ebc0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8ec0912640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8ebc0c3940 0x7f8ebc1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8ec0912640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ebc06a960 con 0x7f8ebc06cc70 2023-08-25T10:05:41.159 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8ec0912640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc06ac20 con 0x7f8ebc0c3940 2023-08-25T10:05:41.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8ec0912640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc06af00 con 0x7f8ebc1594d0 2023-08-25T10:05:41.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8eb9d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8ebc06cc70 0x7f8ebc14f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44146/0 (socket says 172.21.15.71:44146) 2023-08-25T10:05:41.160 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8eb9d9b640 1 -- 172.21.15.71:0/1535874622 learned_addr learned my addr 172.21.15.71:0/1535874622 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:41.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8ebad9d640 1 --2- 172.21.15.71:0/1535874622 >> v2:172.21.15.164:3300/0 conn(0x7f8ebc0c3940 0x7f8ebc1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.163 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8eba59c640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 0x7f8ebc0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384743767 0 0) 0x7f8ebc06a960 con 0x7f8ebc06cc70 2023-08-25T10:05:41.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.157+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea8003520 con 0x7f8ebc06cc70 2023-08-25T10:05:41.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889019857 0 0) 0x7f8ea8003520 con 0x7f8ebc06cc70 2023-08-25T10:05:41.164 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 msgr2=0x7f8ebc0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 0x7f8ebc0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 >> v2:172.21.15.164:3300/0 conn(0x7f8ebc0c3940 msgr2=0x7f8ebc1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 --2- 172.21.15.71:0/1535874622 >> v2:172.21.15.164:3300/0 conn(0x7f8ebc0c3940 0x7f8ebc1598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0c2e60 con 0x7f8ebc06cc70 2023-08-25T10:05:41.165 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f8ea4001670 con 0x7f8ebc06cc70 2023-08-25T10:05:41.166 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eba59c640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 0x7f8ebc0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:05:41.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 >> v1:172.21.15.71:6789/0 conn(0x7f8ebc06cc70 legacy=0x7f8ebc14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ea80039e0 0x7f8ea8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ea80047c0 0x7f8ea8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 0x7f8ea8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc06ac20 con 0x7f8ea80047c0 2023-08-25T10:05:41.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc06af00 con 0x7f8ea80039e0 2023-08-25T10:05:41.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea800ba60 con 0x7f8ebc1594d0 2023-08-25T10:05:41.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8ebad9d640 1 --2- 172.21.15.71:0/1535874622 >> v2:172.21.15.164:3300/0 conn(0x7f8ebc0c3940 0x7f8ebc1598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:05:41.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eb9d9b640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ea80047c0 0x7f8ea8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.158+0000 7f8eba59c640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ea80039e0 0x7f8ea8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.159+0000 7f8eb9d9b640 1 -- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 msgr2=0x7f8ea8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.159+0000 7f8eb9d9b640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ebc1594d0 0x7f8ea8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.159+0000 7f8eb9d9b640 1 -- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ea80039e0 msgr2=0x7f8ea8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.159+0000 7f8eb9d9b640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ea80039e0 0x7f8ea8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.159+0000 7f8eb9d9b640 1 -- 172.21.15.71:0/1535874622 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8003520 con 0x7f8ea80047c0 2023-08-25T10:05:41.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8eb9d9b640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ea80047c0 0x7f8ea8004c40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c210 tx=0x7f8ea80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83664de7d5724dbe server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4005ad0 con 0x7f8ea80047c0 2023-08-25T10:05:41.170 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4005c30 con 0x7f8ea80047c0 2023-08-25T10:05:41.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8eb959a640 1 -- 172.21.15.71:0/1535874622 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea402c8a0 con 0x7f8ea80047c0 2023-08-25T10:05:41.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8ec0912640 1 -- 172.21.15.71:0/1535874622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ea80047c0 msgr2=0x7f8ea8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8ec0912640 1 --2- 172.21.15.71:0/1535874622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ea80047c0 0x7f8ea8004c40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8ea400c210 tx=0x7f8ea80057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.171 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.160+0000 7f8ebad9d640 1 -- 172.21.15.71:0/1535874622 reap_dead start 2023-08-25T10:05:41.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.161+0000 7f8ec0912640 1 -- 172.21.15.71:0/1535874622 shutdown_connections 2023-08-25T10:05:41.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.161+0000 7f8ec0912640 1 -- 172.21.15.71:0/1535874622 >> 172.21.15.71:0/1535874622 conn(0x7f8ebc0654f0 msgr2=0x7f8ebc065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.161+0000 7f8ec0912640 1 -- 172.21.15.71:0/1535874622 shutdown_connections 2023-08-25T10:05:41.172 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.161+0000 7f8ec0912640 1 -- 172.21.15.71:0/1535874622 wait complete. 2023-08-25T10:05:41.174 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.162+0000 7f8ec0912640 1 Processor -- start 2023-08-25T10:05:41.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.162+0000 7f8ec0912640 1 -- start start 2023-08-25T10:05:41.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.163+0000 7f8ec0912640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ea80047c0 0x7f8ebc0c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.163+0000 7f8ec0912640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ebc158c40 0x7f8ebc1eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.163+0000 7f8ec0912640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 0x7f8ebc13dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.175 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.163+0000 7f8ec0912640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc1623f0 con 0x7f8ebc1ef470 2023-08-25T10:05:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.163+0000 7f8ec0912640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc1ee800 con 0x7f8ebc158c40 2023-08-25T10:05:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.163+0000 7f8ec0912640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0ca100 con 0x7f8ea80047c0 2023-08-25T10:05:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 0x7f8ebc13dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.176 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 0x7f8ebc13dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59882/0 (socket says 172.21.15.71:59882) 2023-08-25T10:05:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 -- 172.21.15.71:0/3976881044 learned_addr learned my addr 172.21.15.71:0/3976881044 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8eba59c640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ea80047c0 0x7f8ebc0c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8eb9d9b640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ebc158c40 0x7f8ebc1eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 -- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ea80047c0 msgr2=0x7f8ebc0c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.177 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ea80047c0 0x7f8ebc0c8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 -- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ebc158c40 msgr2=0x7f8ebc1eef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ebc158c40 0x7f8ebc1eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 -- 172.21.15.71:0/3976881044 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc161b70 con 0x7f8ebc1ef470 2023-08-25T10:05:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.164+0000 7f8ebad9d640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 0x7f8ebc13dbe0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc071d10 tx=0x7f8eb400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.178 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.165+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400de30 con 0x7f8ebc1ef470 2023-08-25T10:05:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.165+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400f040 con 0x7f8ebc1ef470 2023-08-25T10:05:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.165+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4003bd0 con 0x7f8ebc1ef470 2023-08-25T10:05:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.165+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc1618d0 con 0x7f8ebc1ef470 2023-08-25T10:05:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.165+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc0c9b40 con 0x7f8ebc1ef470 2023-08-25T10:05:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.166+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc14f320 con 0x7f8ebc1ef470 2023-08-25T10:05:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.168+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f8eb4003d30 con 0x7f8ebc1ef470 2023-08-25T10:05:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.169+0000 7f8eb2ffd640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e9c06a080 0x7f8e9c06c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.169+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f8eb4088910 con 0x7f8ebc1ef470 2023-08-25T10:05:41.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.169+0000 7f8eba59c640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e9c06a080 0x7f8e9c06c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.169+0000 7f8eba59c640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e9c06a080 0x7f8e9c06c540 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8eac006fd0 tx=0x7f8eac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.173+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f8eb4053b40 con 0x7f8ebc1ef470 2023-08-25T10:05:41.235 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.232+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f370c069160 con 0x7f370c152d80 2023-08-25T10:05:41.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.233+0000 7f36f3fff640 1 -- 172.21.15.71:0/4210405182 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f370001e0a0 con 0x7f370c152d80 2023-08-25T10:05:41.236 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T10:05:41.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36f806a050 msgr2=0x7f36f806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.240 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36f806a050 0x7f36f806c510 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f36fc036000 tx=0x7f36fc03b040 comp rx=0 tx=0).stop 2023-08-25T10:05:41.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 msgr2=0x7f370c13ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.241 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c13ef40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f370000c7f0 tx=0x7f370000ccb0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 shutdown_connections 2023-08-25T10:05:41.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f370c15c740 0x7f370c13f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.242 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f36f806a050 0x7f36f806c510 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f370c152d80 0x7f370c13ef40 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.243 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 --2- 172.21.15.71:0/4210405182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f370c0cacb0 0x7f370c143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 >> 172.21.15.71:0/4210405182 conn(0x7f370c06b8f0 msgr2=0x7f370c0c4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.244 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 shutdown_connections 2023-08-25T10:05:41.245 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.236+0000 7f3712226640 1 -- 172.21.15.71:0/4210405182 wait complete. 2023-08-25T10:05:41.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T10:05:41.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.363+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ebc069160 con 0x7f8ebc1ef470 2023-08-25T10:05:41.365 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.363+0000 7f8eb2ffd640 1 -- 172.21.15.71:0/3976881044 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8eb4057160 con 0x7f8ebc1ef470 2023-08-25T10:05:41.366 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T10:05:41.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e9c06a080 msgr2=0x7f8e9c06c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e9c06a080 0x7f8e9c06c540 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8eac006fd0 tx=0x7f8eac008040 comp rx=0 tx=0).stop 2023-08-25T10:05:41.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 msgr2=0x7f8ebc13dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 0x7f8ebc13dbe0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc071d10 tx=0x7f8eb400d4a0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 shutdown_connections 2023-08-25T10:05:41.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8ebc158c40 0x7f8ebc1eef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f8e9c06a080 0x7f8e9c06c540 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ebc1ef470 0x7f8ebc13dbe0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 --2- 172.21.15.71:0/3976881044 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8ea80047c0 0x7f8ebc0c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.366+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 >> 172.21.15.71:0/3976881044 conn(0x7f8ebc0654f0 msgr2=0x7f8ebc0c1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.367+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 shutdown_connections 2023-08-25T10:05:41.372 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.367+0000 7f8ec0912640 1 -- 172.21.15.71:0/3976881044 wait complete. 2023-08-25T10:05:41.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T10:05:41.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.449+0000 7fed99eec640 1 Processor -- start 2023-08-25T10:05:41.451 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.449+0000 7fed99eec640 1 -- start start 2023-08-25T10:05:41.452 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.450+0000 7fed99eec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed940c3660 0x7fed940c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.450+0000 7fed99eec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed9406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.450+0000 7fed99eec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940c1dc0 con 0x7fed941509d0 2023-08-25T10:05:41.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.450+0000 7fed99eec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed94157a80 con 0x7fed940c3660 2023-08-25T10:05:41.453 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.451+0000 7fed99eec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9406cf70 con 0x7fed940c3a40 2023-08-25T10:05:41.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.451+0000 7fed937fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed940c3660 0x7fed940c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.454 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.451+0000 7fed92ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed9406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.451+0000 7fed92ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed9406ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:53186/0 (socket says 172.21.15.71:53186) 2023-08-25T10:05:41.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.451+0000 7fed92ffd640 1 -- 172.21.15.71:0/4226461211 learned_addr learned my addr 172.21.15.71:0/4226461211 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed927fc640 1 -- 172.21.15.71:0/4226461211 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892864586 0 0) 0x7fed940c1dc0 con 0x7fed941509d0 2023-08-25T10:05:41.455 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed927fc640 1 -- 172.21.15.71:0/4226461211 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7c003520 con 0x7fed941509d0 2023-08-25T10:05:41.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed92ffd640 1 -- 172.21.15.71:0/4226461211 >> v2:172.21.15.164:3300/0 conn(0x7fed940c3660 msgr2=0x7fed940c1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed92ffd640 1 --2- 172.21.15.71:0/4226461211 >> v2:172.21.15.164:3300/0 conn(0x7fed940c3660 0x7fed940c1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed92ffd640 1 -- 172.21.15.71:0/4226461211 >> v1:172.21.15.164:6789/0 conn(0x7fed941509d0 legacy=0x7fed94150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed92ffd640 1 -- 172.21.15.71:0/4226461211 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94156e00 con 0x7fed940c3a40 2023-08-25T10:05:41.456 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.452+0000 7fed937fe640 1 --2- 172.21.15.71:0/4226461211 >> v2:172.21.15.164:3300/0 conn(0x7fed940c3660 0x7fed940c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T10:05:41.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.453+0000 7fed92ffd640 1 --2- 172.21.15.71:0/4226461211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed9406ca30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fed88009870 tx=0x7fed8802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcfa677a43596e8a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.454+0000 7fed927fc640 1 -- 172.21.15.71:0/4226461211 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8802f830 con 0x7fed940c3a40 2023-08-25T10:05:41.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.454+0000 7fed927fc640 1 -- 172.21.15.71:0/4226461211 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8802f990 con 0x7fed940c3a40 2023-08-25T10:05:41.460 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.454+0000 7fed927fc640 1 -- 172.21.15.71:0/4226461211 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8802fd20 con 0x7fed940c3a40 2023-08-25T10:05:41.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.454+0000 7fed99eec640 1 -- 172.21.15.71:0/4226461211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 msgr2=0x7fed9406ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.454+0000 7fed99eec640 1 --2- 172.21.15.71:0/4226461211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed9406ca30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fed88009870 tx=0x7fed8802ba10 comp rx=0 tx=0).stop 2023-08-25T10:05:41.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.454+0000 7fed99eec640 1 -- 172.21.15.71:0/4226461211 shutdown_connections 2023-08-25T10:05:41.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.455+0000 7fed99eec640 1 --2- 172.21.15.71:0/4226461211 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed9406ca30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.455+0000 7fed99eec640 1 --2- 172.21.15.71:0/4226461211 >> v2:172.21.15.164:3300/0 conn(0x7fed940c3660 0x7fed940c1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.455+0000 7fed99eec640 1 -- 172.21.15.71:0/4226461211 >> 172.21.15.71:0/4226461211 conn(0x7fed940654f0 msgr2=0x7fed94065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.455+0000 7fed99eec640 1 -- 172.21.15.71:0/4226461211 shutdown_connections 2023-08-25T10:05:41.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.455+0000 7fed99eec640 1 -- 172.21.15.71:0/4226461211 wait complete. 2023-08-25T10:05:41.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.456+0000 7fed99eec640 1 Processor -- start 2023-08-25T10:05:41.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.457+0000 7fed99eec640 1 -- start start 2023-08-25T10:05:41.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.457+0000 7fed99eec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed940c3660 0x7fed94153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.457+0000 7fed99eec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed941eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed99eec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 0x7fed941f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed99eec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed941f2440 con 0x7fed941509d0 2023-08-25T10:05:41.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed99eec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed94152e70 con 0x7fed940c3660 2023-08-25T10:05:41.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed99eec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed941544d0 con 0x7fed940c3a40 2023-08-25T10:05:41.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed93fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 0x7fed941f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed93fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 0x7fed941f1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59886/0 (socket says 172.21.15.71:59886) 2023-08-25T10:05:41.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed93fff640 1 -- 172.21.15.71:0/3754629227 learned_addr learned my addr 172.21.15.71:0/3754629227 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed937fe640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed940c3660 0x7fed94153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.458+0000 7fed92ffd640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed941eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed93fff640 1 -- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 msgr2=0x7fed941eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed93fff640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed941eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed93fff640 1 -- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed940c3660 msgr2=0x7fed94153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed93fff640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed940c3660 0x7fed94153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed93fff640 1 -- 172.21.15.71:0/3754629227 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84009400 con 0x7fed941509d0 2023-08-25T10:05:41.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed92ffd640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed941eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-08-25T10:05:41.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed93fff640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 0x7fed941f1860 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00c790 tx=0x7fed8c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c00cea0 con 0x7fed941509d0 2023-08-25T10:05:41.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8c003b70 con 0x7fed941509d0 2023-08-25T10:05:41.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed880094d0 con 0x7fed941509d0 2023-08-25T10:05:41.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.459+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c00a850 con 0x7fed941509d0 2023-08-25T10:05:41.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.460+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed941f2160 con 0x7fed941509d0 2023-08-25T10:05:41.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.461+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed940c29c0 con 0x7fed941509d0 2023-08-25T10:05:41.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.462+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fed8c00a9b0 con 0x7fed941509d0 2023-08-25T10:05:41.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.464+0000 7fed7bfff640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed8006a050 0x7fed8006c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.464+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fed8c088920 con 0x7fed941509d0 2023-08-25T10:05:41.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.464+0000 7fed937fe640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed8006a050 0x7fed8006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.465+0000 7fed937fe640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed8006a050 0x7fed8006c510 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fed84009c30 tx=0x7fed84009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.467+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fed8c053ab0 con 0x7fed941509d0 2023-08-25T10:05:41.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.592+0000 7f44f86e1640 1 Processor -- start 2023-08-25T10:05:41.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.592+0000 7f44f86e1640 1 -- start start 2023-08-25T10:05:41.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f86e1640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44f0152d80 0x7f44f006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f86e1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f44f0070cf0 0x7f44f01637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f86e1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f006eb60 con 0x7f44f00c4560 2023-08-25T10:05:41.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f86e1640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f006ca30 con 0x7f44f0152d80 2023-08-25T10:05:41.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f86e1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f006cf90 con 0x7f44f0070cf0 2023-08-25T10:05:41.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f647d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44f0152d80 0x7f44f006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f647d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f44f0152d80 0x7f44f006e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59890/0 (socket says 172.21.15.71:59890) 2023-08-25T10:05:41.598 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.594+0000 7f44f647d640 1 -- 172.21.15.71:0/547958497 learned_addr learned my addr 172.21.15.71:0/547958497 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.595+0000 7f44f6c7e640 1 --2- 172.21.15.71:0/547958497 >> v2:172.21.15.164:3300/0 conn(0x7f44f0070cf0 0x7f44f01637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.599 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.595+0000 7f44f647d640 1 -- 172.21.15.71:0/547958497 >> v2:172.21.15.164:3300/0 conn(0x7f44f0070cf0 msgr2=0x7f44f01637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.595+0000 7f44f647d640 1 --2- 172.21.15.71:0/547958497 >> v2:172.21.15.164:3300/0 conn(0x7f44f0070cf0 0x7f44f01637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.595+0000 7f44f647d640 1 -- 172.21.15.71:0/547958497 >> v1:172.21.15.164:6789/0 conn(0x7f44f00c4560 legacy=0x7f44f00c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.595+0000 7f44f647d640 1 -- 172.21.15.71:0/547958497 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f0158fd0 con 0x7f44f0152d80 2023-08-25T10:05:41.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.596+0000 7f44f647d640 1 --2- 172.21.15.71:0/547958497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f006e620 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f44f00c1390 tx=0x7f44ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e2340cfc077806 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.600 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.596+0000 7f44f547b640 1 -- 172.21.15.71:0/547958497 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec0317e0 con 0x7f44f0152d80 2023-08-25T10:05:41.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.596+0000 7f44f547b640 1 -- 172.21.15.71:0/547958497 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec031940 con 0x7f44f0152d80 2023-08-25T10:05:41.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.596+0000 7f44f547b640 1 -- 172.21.15.71:0/547958497 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec031cd0 con 0x7f44f0152d80 2023-08-25T10:05:41.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 -- 172.21.15.71:0/547958497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 msgr2=0x7f44f006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 --2- 172.21.15.71:0/547958497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f006e620 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f44f00c1390 tx=0x7f44ec02b880 comp rx=0 tx=0).stop 2023-08-25T10:05:41.601 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 -- 172.21.15.71:0/547958497 shutdown_connections 2023-08-25T10:05:41.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 --2- 172.21.15.71:0/547958497 >> v2:172.21.15.164:3300/0 conn(0x7f44f0070cf0 0x7f44f01637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.602 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 --2- 172.21.15.71:0/547958497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f006e620 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 -- 172.21.15.71:0/547958497 >> 172.21.15.71:0/547958497 conn(0x7f44f00654f0 msgr2=0x7f44f0065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 -- 172.21.15.71:0/547958497 shutdown_connections 2023-08-25T10:05:41.603 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.597+0000 7f44f86e1640 1 -- 172.21.15.71:0/547958497 wait complete. 2023-08-25T10:05:41.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.599+0000 7f44f86e1640 1 Processor -- start 2023-08-25T10:05:41.605 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.599+0000 7f44f86e1640 1 -- start start 2023-08-25T10:05:41.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.599+0000 7f44f86e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f0070cf0 0x7f44f00c1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.606 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f86e1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f00c4560 0x7f44f01f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.607 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f86e1640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f01f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f86e1640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f014c370 con 0x7f44f0152d80 2023-08-25T10:05:41.609 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f86e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f01f2690 con 0x7f44f0070cf0 2023-08-25T10:05:41.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f86e1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00c3400 con 0x7f44f00c4560 2023-08-25T10:05:41.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f01f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.610 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f01f2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59906/0 (socket says 172.21.15.71:59906) 2023-08-25T10:05:41.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 -- 172.21.15.71:0/139398982 learned_addr learned my addr 172.21.15.71:0/139398982 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f647d640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f0070cf0 0x7f44f00c1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.611 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f5c7c640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f00c4560 0x7f44f01f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 -- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f00c4560 msgr2=0x7f44f01f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f00c4560 0x7f44f01f1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 -- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f0070cf0 msgr2=0x7f44f00c1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.612 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f0070cf0 0x7f44f00c1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f6c7e640 1 -- 172.21.15.71:0/139398982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0094d0 con 0x7f44f0152d80 2023-08-25T10:05:41.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.600+0000 7f44f5c7c640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f00c4560 0x7f44f01f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:41.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.601+0000 7f44f6c7e640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f01f2070 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f44e8007e90 tx=0x7f44e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.601+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8014420 con 0x7f44f0152d80 2023-08-25T10:05:41.613 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.601+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8014580 con 0x7f44f0152d80 2023-08-25T10:05:41.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.601+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8014930 con 0x7f44f0152d80 2023-08-25T10:05:41.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.601+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f00c36c0 con 0x7f44f0152d80 2023-08-25T10:05:41.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.601+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f00c9e90 con 0x7f44f0152d80 2023-08-25T10:05:41.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.602+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f014c520 con 0x7f44f0152d80 2023-08-25T10:05:41.614 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.603+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f44e8014a90 con 0x7f44f0152d80 2023-08-25T10:05:41.615 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.604+0000 7f44e6ffd640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44d406a050 0x7f44d406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.604+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f44e8089420 con 0x7f44f0152d80 2023-08-25T10:05:41.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.605+0000 7f44f647d640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44d406a050 0x7f44d406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.605+0000 7f44f647d640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44d406a050 0x7f44d406c510 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f44ec036000 tx=0x7f44ec03d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.616 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.609+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f44f014c520 con 0x7f44f0152d80 2023-08-25T10:05:41.664 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.662+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fed94069160 con 0x7fed941509d0 2023-08-25T10:05:41.665 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.663+0000 7fed7bfff640 1 -- 172.21.15.71:0/3754629227 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v69) v1 ==== 103+0+11 (secure 0 0 0) 0x7fed8c0570d0 con 0x7fed941509d0 2023-08-25T10:05:41.666 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:41.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed8006a050 msgr2=0x7fed8006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.669 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed8006a050 0x7fed8006c510 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fed84009c30 tx=0x7fed84009190 comp rx=0 tx=0).stop 2023-08-25T10:05:41.670 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 msgr2=0x7fed941f1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 0x7fed941f1860 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00c790 tx=0x7fed8c00cc50 comp rx=0 tx=0).stop 2023-08-25T10:05:41.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 shutdown_connections 2023-08-25T10:05:41.671 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fed8006a050 0x7fed8006c510 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fed941509d0 0x7fed941f1860 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.672 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed940c3a40 0x7fed941eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 --2- 172.21.15.71:0/3754629227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed940c3660 0x7fed94153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.666+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 >> 172.21.15.71:0/3754629227 conn(0x7fed940654f0 msgr2=0x7fed940658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.673 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.667+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 shutdown_connections 2023-08-25T10:05:41.674 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.667+0000 7fed99eec640 1 -- 172.21.15.71:0/3754629227 wait complete. 2023-08-25T10:05:41.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T10:05:41.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.813+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f44f014c520 con 0x7f44f0152d80 2023-08-25T10:05:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.814+0000 7f44e6ffd640 1 -- 172.21.15.71:0/139398982 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7f44f014c520 con 0x7f44f0152d80 2023-08-25T10:05:41.817 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T10:05:41.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44d406a050 msgr2=0x7f44d406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44d406a050 0x7f44d406c510 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f44ec036000 tx=0x7f44ec03d040 comp rx=0 tx=0).stop 2023-08-25T10:05:41.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 msgr2=0x7f44f01f2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f01f2070 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f44e8007e90 tx=0x7f44e800e4b0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 shutdown_connections 2023-08-25T10:05:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f44d406a050 0x7f44d406c510 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f44f0152d80 0x7f44f01f2070 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f44f00c4560 0x7f44f01f1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 --2- 172.21.15.71:0/139398982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f44f0070cf0 0x7f44f00c1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 >> 172.21.15.71:0/139398982 conn(0x7f44f00654f0 msgr2=0x7f44f00658d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 shutdown_connections 2023-08-25T10:05:41.824 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.817+0000 7f44f86e1640 1 -- 172.21.15.71:0/139398982 wait complete. 2023-08-25T10:05:41.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T10:05:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.908+0000 7fc4e170c640 1 Processor -- start 2023-08-25T10:05:41.910 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.908+0000 7fc4e170c640 1 -- start start 2023-08-25T10:05:41.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.909+0000 7fc4e170c640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc4dc0c8600 0x7fc4dc069850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.909+0000 7fc4e170c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.914 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4e170c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc4dc0b1760 0x7fc4dc0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4e170c640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc0b4430 con 0x7fc4dc0c8600 2023-08-25T10:05:41.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4e170c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc0b2390 con 0x7fc4dc0b1760 2023-08-25T10:05:41.915 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4e170c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc1536c0 con 0x7fc4dc15c4f0 2023-08-25T10:05:41.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc4dc0c8600 0x7fc4dc069850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.916 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fc4dc0c8600 0x7fc4dc069850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59908/0 (socket says 172.21.15.71:59908) 2023-08-25T10:05:41.917 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 -- 172.21.15.71:0/2751894536 learned_addr learned my addr 172.21.15.71:0/2751894536 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4db7fe640 1 --2- 172.21.15.71:0/2751894536 >> v2:172.21.15.164:3300/0 conn(0x7fc4dc0b1760 0x7fc4dc0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4da7fc640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 -- 172.21.15.71:0/2751894536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 msgr2=0x7fc4dc0b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc0b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.918 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 -- 172.21.15.71:0/2751894536 >> v2:172.21.15.164:3300/0 conn(0x7fc4dc0b1760 msgr2=0x7fc4dc0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:41.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 --2- 172.21.15.71:0/2751894536 >> v2:172.21.15.164:3300/0 conn(0x7fc4dc0b1760 0x7fc4dc0b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.910+0000 7fc4daffd640 1 -- 172.21.15.71:0/2751894536 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0c89e0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.911+0000 7fc4db7fe640 1 --2- 172.21.15.71:0/2751894536 >> v2:172.21.15.164:3300/0 conn(0x7fc4dc0b1760 0x7fc4dc0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:41.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.911+0000 7fc4da7fc640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc0b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:41.919 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.912+0000 7fc4daffd640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc069850 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc15d560 tx=0x7fc4cc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855e6237d98d1ee3 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.912+0000 7fc4d9ffb640 1 -- 172.21.15.71:0/2751894536 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc031880 con 0x7fc4dc0c8600 2023-08-25T10:05:41.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.912+0000 7fc4d9ffb640 1 -- 172.21.15.71:0/2751894536 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4cc0319e0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.912+0000 7fc4d9ffb640 1 -- 172.21.15.71:0/2751894536 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc031d70 con 0x7fc4dc0c8600 2023-08-25T10:05:41.922 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 -- 172.21.15.71:0/2751894536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 msgr2=0x7fc4dc069850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc069850 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc15d560 tx=0x7fc4cc02b920 comp rx=0 tx=0).stop 2023-08-25T10:05:41.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 -- 172.21.15.71:0/2751894536 shutdown_connections 2023-08-25T10:05:41.923 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 --2- 172.21.15.71:0/2751894536 >> v2:172.21.15.164:3300/0 conn(0x7fc4dc0b1760 0x7fc4dc0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc0b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 --2- 172.21.15.71:0/2751894536 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc069850 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 -- 172.21.15.71:0/2751894536 >> 172.21.15.71:0/2751894536 conn(0x7fc4dc0ce720 msgr2=0x7fc4dc0ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:41.924 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 -- 172.21.15.71:0/2751894536 shutdown_connections 2023-08-25T10:05:41.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.913+0000 7fc4e170c640 1 -- 172.21.15.71:0/2751894536 wait complete. 2023-08-25T10:05:41.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.915+0000 7fc4e170c640 1 Processor -- start 2023-08-25T10:05:41.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.915+0000 7fc4e170c640 1 -- start start 2023-08-25T10:05:41.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.916+0000 7fc4e170c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4dc0b1760 0x7fc4dc158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.916+0000 7fc4e170c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc1f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.916+0000 7fc4e170c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.916+0000 7fc4e170c640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc1589e0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.916+0000 7fc4e170c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc154a10 con 0x7fc4dc0b1760 2023-08-25T10:05:41.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.916+0000 7fc4e170c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc153e90 con 0x7fc4dc15c4f0 2023-08-25T10:05:41.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc1f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.926 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc1f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59920/0 (socket says 172.21.15.71:59920) 2023-08-25T10:05:41.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 -- 172.21.15.71:0/3142400609 learned_addr learned my addr 172.21.15.71:0/3142400609 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:41.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4daffd640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4dc0b1760 0x7fc4dc158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4db7fe640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 -- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 msgr2=0x7fc4dc1f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc1f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 -- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4dc0b1760 msgr2=0x7fc4dc158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:41.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4dc0b1760 0x7fc4dc158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:41.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 -- 172.21.15.71:0/3142400609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0094d0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4db7fe640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:41.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.917+0000 7fc4da7fc640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc1f19d0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c7f0 tx=0x7fc4d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.918+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0007a30 con 0x7fc4dc0c8600 2023-08-25T10:05:41.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.918+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0007b90 con 0x7fc4dc0c8600 2023-08-25T10:05:41.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.918+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000ce40 con 0x7fc4dc0c8600 2023-08-25T10:05:41.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.918+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc154170 con 0x7fc4dc0c8600 2023-08-25T10:05:41.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.918+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc154730 con 0x7fc4dc0c8600 2023-08-25T10:05:41.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.919+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc0c20c0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.920+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fc4d00113f0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.921+0000 7fc4c37fe640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4c806a0a0 0x7fc4c806c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:41.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.921+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fc4d0088bf0 con 0x7fc4dc0c8600 2023-08-25T10:05:41.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.921+0000 7fc4daffd640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4c806a0a0 0x7fc4c806c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:41.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.922+0000 7fc4daffd640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4c806a0a0 0x7fc4c806c560 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc036000 tx=0x7fc4cc03c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:41.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:41.925+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fc4d0053e00 con 0x7fc4dc0c8600 2023-08-25T10:05:42.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.038+0000 7fb64e42d640 1 Processor -- start 2023-08-25T10:05:42.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.038+0000 7fb64e42d640 1 -- start start 2023-08-25T10:05:42.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb64e42d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb64806a6b0 0x7fb64806bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb64e42d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 0x7fb648070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb64e42d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb648159a60 con 0x7fb648153300 2023-08-25T10:05:42.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb64e42d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64806b920 con 0x7fb64806a6b0 2023-08-25T10:05:42.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb64e42d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb648159d40 con 0x7fb64806f620 2023-08-25T10:05:42.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb647fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fb648153300 0x7fb6480c5d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44154/0 (socket says 172.21.15.71:44154) 2023-08-25T10:05:42.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.040+0000 7fb647fff640 1 -- 172.21.15.71:0/1201579975 learned_addr learned my addr 172.21.15.71:0/1201579975 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:42.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb6477fe640 1 --2- 172.21.15.71:0/1201579975 >> v2:172.21.15.164:3300/0 conn(0x7fb64806a6b0 0x7fb64806bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb64c9ca640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 0x7fb648070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630213970 0 0) 0x7fb648159a60 con 0x7fb648153300 2023-08-25T10:05:42.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634003520 con 0x7fb648153300 2023-08-25T10:05:42.045 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855349258 0 0) 0x7fb634003520 con 0x7fb648153300 2023-08-25T10:05:42.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 msgr2=0x7fb648070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 0x7fb648070eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 >> v2:172.21.15.164:3300/0 conn(0x7fb64806a6b0 msgr2=0x7fb64806bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 --2- 172.21.15.71:0/1201579975 >> v2:172.21.15.164:3300/0 conn(0x7fb64806a6b0 0x7fb64806bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.046 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6480698f0 con 0x7fb648153300 2023-08-25T10:05:42.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7fb638001670 con 0x7fb648153300 2023-08-25T10:05:42.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 >> v1:172.21.15.71:6789/0 conn(0x7fb648153300 legacy=0x7fb6480c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6340039e0 0x7fb634003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb634004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.047 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.041+0000 7fb646ffd640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 0x7fb634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb648159d40 con 0x7fb6340047c0 2023-08-25T10:05:42.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63400b980 con 0x7fb6340039e0 2023-08-25T10:05:42.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63400bac0 con 0x7fb64806f620 2023-08-25T10:05:42.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb6477fe640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb634004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.048 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb647fff640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6340039e0 0x7fb634003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb6477fe640 1 -- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 msgr2=0x7fb634005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb6477fe640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb64806f620 0x7fb634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb6477fe640 1 -- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6340039e0 msgr2=0x7fb634003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb6477fe640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6340039e0 0x7fb634003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.049 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.042+0000 7fb6477fe640 1 -- 172.21.15.71:0/1201579975 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634003520 con 0x7fb6340047c0 2023-08-25T10:05:42.050 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.043+0000 7fb6477fe640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb634004c40 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb630009af0 tx=0x7fb6340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a834a93d964b1 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.043+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63002c710 con 0x7fb6340047c0 2023-08-25T10:05:42.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63002c870 con 0x7fb6340047c0 2023-08-25T10:05:42.054 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb646ffd640 1 -- 172.21.15.71:0/1201579975 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63002cc00 con 0x7fb6340047c0 2023-08-25T10:05:42.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64e42d640 1 -- 172.21.15.71:0/1201579975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 msgr2=0x7fb634004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64e42d640 1 --2- 172.21.15.71:0/1201579975 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb634004c40 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb630009af0 tx=0x7fb6340057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64c9ca640 1 -- 172.21.15.71:0/1201579975 reap_dead start 2023-08-25T10:05:42.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64e42d640 1 -- 172.21.15.71:0/1201579975 shutdown_connections 2023-08-25T10:05:42.055 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64e42d640 1 -- 172.21.15.71:0/1201579975 >> 172.21.15.71:0/1201579975 conn(0x7fb6480654f0 msgr2=0x7fb648065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64e42d640 1 -- 172.21.15.71:0/1201579975 shutdown_connections 2023-08-25T10:05:42.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.044+0000 7fb64e42d640 1 -- 172.21.15.71:0/1201579975 wait complete. 2023-08-25T10:05:42.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.046+0000 7fb64e42d640 1 Processor -- start 2023-08-25T10:05:42.056 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.046+0000 7fb64e42d640 1 -- start start 2023-08-25T10:05:42.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.047+0000 7fb64e42d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb6480ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.047+0000 7fb64e42d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb64815a020 0x7fb64815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.047+0000 7fb64e42d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6480c3180 0x7fb64815b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.047+0000 7fb64e42d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480c5060 con 0x7fb6340047c0 2023-08-25T10:05:42.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.047+0000 7fb64e42d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480c52d0 con 0x7fb64815a020 2023-08-25T10:05:42.057 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.047+0000 7fb64e42d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb648159800 con 0x7fb6480c3180 2023-08-25T10:05:42.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb6480ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb6480ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59946/0 (socket says 172.21.15.71:59946) 2023-08-25T10:05:42.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 -- 172.21.15.71:0/4002346268 learned_addr learned my addr 172.21.15.71:0/4002346268 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:42.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 -- 172.21.15.71:0/4002346268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6480c3180 msgr2=0x7fb64815b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.058 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6480c3180 0x7fb64815b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 -- 172.21.15.71:0/4002346268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb64815a020 msgr2=0x7fb64815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb64815a020 0x7fb64815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 -- 172.21.15.71:0/4002346268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6480c4190 con 0x7fb6340047c0 2023-08-25T10:05:42.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb647fff640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb6480ca5b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb6380018e0 tx=0x7fb63800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.059 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.048+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63800ee00 con 0x7fb6340047c0 2023-08-25T10:05:42.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.049+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63800c7c0 con 0x7fb6340047c0 2023-08-25T10:05:42.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.049+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63800cb70 con 0x7fb6340047c0 2023-08-25T10:05:42.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.049+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6480c43e0 con 0x7fb6340047c0 2023-08-25T10:05:42.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.049+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6480c49a0 con 0x7fb6340047c0 2023-08-25T10:05:42.060 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.050+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb648069160 con 0x7fb6340047c0 2023-08-25T10:05:42.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.051+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fb63800ccd0 con 0x7fb6340047c0 2023-08-25T10:05:42.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.052+0000 7fb644ff9640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb62806a050 0x7fb62806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.052+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fb6380888e0 con 0x7fb6340047c0 2023-08-25T10:05:42.061 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.052+0000 7fb6477fe640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb62806a050 0x7fb62806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.053+0000 7fb6477fe640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb62806a050 0x7fb62806c510 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb630036000 tx=0x7fb63002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.062 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.056+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fb638057590 con 0x7fb6340047c0 2023-08-25T10:05:42.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.135+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc4dc069160 con 0x7fc4dc0c8600 2023-08-25T10:05:42.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.136+0000 7fc4c37fe640 1 -- 172.21.15.71:0/3142400609 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v69) v1 ==== 80+0+492 (secure 0 0 0) 0x7fc4d001f080 con 0x7fc4dc0c8600 2023-08-25T10:05:42.139 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6837/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:42.140 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6834/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:42.142 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6832/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:42.142 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6836/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:42.142 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6833/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:42.143 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6835/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:42.143 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6835/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:42.143 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6834/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:42.144 INFO:teuthology.orchestra.run.smithi071.stderr:listed 8 entries 2023-08-25T10:05:42.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4c806a0a0 msgr2=0x7fc4c806c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.144 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4c806a0a0 0x7fc4c806c560 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc036000 tx=0x7fc4cc03c040 comp rx=0 tx=0).stop 2023-08-25T10:05:42.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 msgr2=0x7fc4dc1f19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc1f19d0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc4d000c7f0 tx=0x7fc4d000cbf0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 shutdown_connections 2023-08-25T10:05:42.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fc4c806a0a0 0x7fc4c806c560 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4dc15c4f0 0x7fc4dc1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc4dc0c8600 0x7fc4dc1f19d0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 --2- 172.21.15.71:0/3142400609 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4dc0b1760 0x7fc4dc158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 >> 172.21.15.71:0/3142400609 conn(0x7fc4dc0ce720 msgr2=0x7fc4dc0c8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 shutdown_connections 2023-08-25T10:05:42.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.139+0000 7fc4e170c640 1 -- 172.21.15.71:0/3142400609 wait complete. 2023-08-25T10:05:42.158 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2023-08-25T10:05:42.158 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/proc/self/mounts of=/dev/stdout 2023-08-25T10:05:42.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-08-25T10:05:42.246 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.244+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb64806abc0 con 0x7fb6340047c0 2023-08-25T10:05:42.247 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.245+0000 7fb644ff9640 1 -- 172.21.15.71:0/4002346268 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb638057120 con 0x7fb6340047c0 2023-08-25T10:05:42.247 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:42.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb62806a050 msgr2=0x7fb62806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb62806a050 0x7fb62806c510 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb630036000 tx=0x7fb63002e000 comp rx=0 tx=0).stop 2023-08-25T10:05:42.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 msgr2=0x7fb6480ca5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.255 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb6480ca5b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fb6380018e0 tx=0x7fb63800ec10 comp rx=0 tx=0).stop 2023-08-25T10:05:42.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 shutdown_connections 2023-08-25T10:05:42.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb64815a020 0x7fb64815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.256 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fb62806a050 0x7fb62806c510 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6480c3180 0x7fb64815b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 --2- 172.21.15.71:0/4002346268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6340047c0 0x7fb6480ca5b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 >> 172.21.15.71:0/4002346268 conn(0x7fb6480654f0 msgr2=0x7fb64814f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.257 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 shutdown_connections 2023-08-25T10:05:42.258 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.247+0000 7fb64e42d640 1 -- 172.21.15.71:0/4002346268 wait complete. 2023-08-25T10:05:42.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T10:05:42.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.460+0000 7f562c0ec640 1 Processor -- start 2023-08-25T10:05:42.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.461+0000 7f562c0ec640 1 -- start start 2023-08-25T10:05:42.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f562c0ec640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5624152ff0 0x7f562415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f562c0ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56240c9ee0 0x7f5624159280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f562c0ec640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240c9060 con 0x7f56240ca630 2023-08-25T10:05:42.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f562c0ec640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562406ac60 con 0x7f5624152ff0 2023-08-25T10:05:42.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f562c0ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562406a6e0 con 0x7f56240c9ee0 2023-08-25T10:05:42.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f5629e88640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5624152ff0 0x7f562415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f5629e88640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f5624152ff0 0x7f562415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59962/0 (socket says 172.21.15.71:59962) 2023-08-25T10:05:42.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.462+0000 7f5629e88640 1 -- 172.21.15.71:0/1342504567 learned_addr learned my addr 172.21.15.71:0/1342504567 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:42.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.463+0000 7f562a689640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56240c9ee0 0x7f5624159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.463+0000 7f5629e88640 1 -- 172.21.15.71:0/1342504567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56240c9ee0 msgr2=0x7f5624159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.463+0000 7f5629e88640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56240c9ee0 0x7f5624159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.463+0000 7f5629e88640 1 -- 172.21.15.71:0/1342504567 >> v1:172.21.15.71:6789/0 conn(0x7f56240ca630 legacy=0x7f56240cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.463+0000 7f5629e88640 1 -- 172.21.15.71:0/1342504567 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56240c92b0 con 0x7f5624152ff0 2023-08-25T10:05:42.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.463+0000 7f562a689640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56240c9ee0 0x7f5624159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:42.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.464+0000 7f5629e88640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5624152ff0 0x7f562415cea0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5620009870 tx=0x7f562002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266a85581d073e24 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.464+0000 7f5628e86640 1 -- 172.21.15.71:0/1342504567 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620031640 con 0x7f5624152ff0 2023-08-25T10:05:42.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.464+0000 7f5628e86640 1 -- 172.21.15.71:0/1342504567 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56200317a0 con 0x7f5624152ff0 2023-08-25T10:05:42.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.464+0000 7f5628e86640 1 -- 172.21.15.71:0/1342504567 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620031b30 con 0x7f5624152ff0 2023-08-25T10:05:42.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 -- 172.21.15.71:0/1342504567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5624152ff0 msgr2=0x7f562415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5624152ff0 0x7f562415cea0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5620009870 tx=0x7f562002b8f0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 -- 172.21.15.71:0/1342504567 shutdown_connections 2023-08-25T10:05:42.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f56240c9ee0 0x7f5624159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 --2- 172.21.15.71:0/1342504567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5624152ff0 0x7f562415cea0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 -- 172.21.15.71:0/1342504567 >> 172.21.15.71:0/1342504567 conn(0x7f5624065910 msgr2=0x7f5624065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 -- 172.21.15.71:0/1342504567 shutdown_connections 2023-08-25T10:05:42.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.465+0000 7f562c0ec640 1 -- 172.21.15.71:0/1342504567 wait complete. 2023-08-25T10:05:42.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.467+0000 7f562c0ec640 1 Processor -- start 2023-08-25T10:05:42.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.467+0000 7f562c0ec640 1 -- start start 2023-08-25T10:05:42.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f562c0ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56240c9ee0 0x7f5624149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f562c0ec640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 0x7f562414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f562c0ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5624152ff0 0x7f562414a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f562c0ec640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562414daf0 con 0x7f56240ca630 2023-08-25T10:05:42.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f562c0ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562414b340 con 0x7f56240c9ee0 2023-08-25T10:05:42.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f562c0ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f562414c490 con 0x7f5624152ff0 2023-08-25T10:05:42.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f5629687640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 0x7f562414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f5629687640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 0x7f562414a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59968/0 (socket says 172.21.15.71:59968) 2023-08-25T10:05:42.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.468+0000 7f5629687640 1 -- 172.21.15.71:0/682607876 learned_addr learned my addr 172.21.15.71:0/682607876 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:42.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629e88640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56240c9ee0 0x7f5624149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f562a689640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5624152ff0 0x7f562414a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629687640 1 -- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5624152ff0 msgr2=0x7f562414a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629687640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5624152ff0 0x7f562414a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629687640 1 -- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56240c9ee0 msgr2=0x7f5624149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629687640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56240c9ee0 0x7f5624149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629687640 1 -- 172.21.15.71:0/682607876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200094d0 con 0x7f56240ca630 2023-08-25T10:05:42.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f5629687640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 0x7f562414a370 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f561800c400 tx=0x7f561800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56180052d0 con 0x7f56240ca630 2023-08-25T10:05:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5618005430 con 0x7f56240ca630 2023-08-25T10:05:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618003830 con 0x7f56240ca630 2023-08-25T10:05:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.469+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562414b5a0 con 0x7f56240ca630 2023-08-25T10:05:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.470+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562414bb60 con 0x7f56240ca630 2023-08-25T10:05:42.478 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.471+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5624072790 con 0x7f56240ca630 2023-08-25T10:05:42.479 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.472+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f5618003990 con 0x7f56240ca630 2023-08-25T10:05:42.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.474+0000 7f56167fc640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f560806a080 0x7f560806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.474+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f5618088060 con 0x7f56240ca630 2023-08-25T10:05:42.480 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.474+0000 7f5629e88640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f560806a080 0x7f560806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.475+0000 7f5629e88640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f560806a080 0x7f560806c540 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5620036000 tx=0x7f562003c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.481 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.478+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f56180ba990 con 0x7f56240ca630 2023-08-25T10:05:42.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.510+0000 7fd4192ed640 1 Processor -- start 2023-08-25T10:05:42.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.510+0000 7fd4192ed640 1 -- start start 2023-08-25T10:05:42.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.511+0000 7fd4192ed640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 0x7fd4140ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.511+0000 7fd4192ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41415cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.511+0000 7fd4192ed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4140b1af0 con 0x7fd4140c7f10 2023-08-25T10:05:42.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.511+0000 7fd4192ed640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4140c98f0 con 0x7fd4140cb190 2023-08-25T10:05:42.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.511+0000 7fd4192ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140b0fb0 con 0x7fd4140cb570 2023-08-25T10:05:42.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.512+0000 7fd413fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 0x7fd4140ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.512+0000 7fd4137fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41415cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.512+0000 7fd413fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 0x7fd4140ca910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:57820/0 (socket says 172.21.15.71:57820) 2023-08-25T10:05:42.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.512+0000 7fd413fff640 1 -- 172.21.15.71:0/858455079 learned_addr learned my addr 172.21.15.71:0/858455079 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:42.516 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.513+0000 7fd4137fe640 1 -- 172.21.15.71:0/858455079 >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 msgr2=0x7fd4140ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.513+0000 7fd4137fe640 1 --2- 172.21.15.71:0/858455079 >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 0x7fd4140ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.517 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.513+0000 7fd4137fe640 1 -- 172.21.15.71:0/858455079 >> v1:172.21.15.164:6789/0 conn(0x7fd4140c7f10 legacy=0x7fd414158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.513+0000 7fd4137fe640 1 -- 172.21.15.71:0/858455079 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4140b1530 con 0x7fd4140cb570 2023-08-25T10:05:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.513+0000 7fd413fff640 1 --2- 172.21.15.71:0/858455079 >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 0x7fd4140ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-08-25T10:05:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.514+0000 7fd4137fe640 1 --2- 172.21.15.71:0/858455079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41415cd00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd4080097a0 tx=0x7fd40802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce8e9860ad3720a server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd412ffd640 1 -- 172.21.15.71:0/858455079 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408030800 con 0x7fd4140cb570 2023-08-25T10:05:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd412ffd640 1 -- 172.21.15.71:0/858455079 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd408030960 con 0x7fd4140cb570 2023-08-25T10:05:42.519 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd412ffd640 1 -- 172.21.15.71:0/858455079 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408030cf0 con 0x7fd4140cb570 2023-08-25T10:05:42.521 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 -- 172.21.15.71:0/858455079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 msgr2=0x7fd41415cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 --2- 172.21.15.71:0/858455079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41415cd00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd4080097a0 tx=0x7fd40802b940 comp rx=0 tx=0).stop 2023-08-25T10:05:42.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 -- 172.21.15.71:0/858455079 shutdown_connections 2023-08-25T10:05:42.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 --2- 172.21.15.71:0/858455079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41415cd00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.522 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 --2- 172.21.15.71:0/858455079 >> v2:172.21.15.164:3300/0 conn(0x7fd4140cb190 0x7fd4140ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 -- 172.21.15.71:0/858455079 >> 172.21.15.71:0/858455079 conn(0x7fd41406ae20 msgr2=0x7fd41406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 -- 172.21.15.71:0/858455079 shutdown_connections 2023-08-25T10:05:42.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.515+0000 7fd4192ed640 1 -- 172.21.15.71:0/858455079 wait complete. 2023-08-25T10:05:42.523 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.517+0000 7fd4192ed640 1 Processor -- start 2023-08-25T10:05:42.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.517+0000 7fd4192ed640 1 -- start start 2023-08-25T10:05:42.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4192ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4140c7f10 0x7fd414146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4192ed640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 0x7fd4140c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.524 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4192ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41414db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4192ed640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140cd510 con 0x7fd4140cb190 2023-08-25T10:05:42.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4192ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140cc410 con 0x7fd4140c7f10 2023-08-25T10:05:42.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4192ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140cc6b0 con 0x7fd4140cb570 2023-08-25T10:05:42.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4137fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 0x7fd4140c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.525 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.518+0000 7fd4137fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 0x7fd4140c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:59972/0 (socket says 172.21.15.71:59972) 2023-08-25T10:05:42.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 -- 172.21.15.71:0/3660357 learned_addr learned my addr 172.21.15.71:0/3660357 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:42.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd413fff640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4140c7f10 0x7fd414146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd418aec640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41414db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 -- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 msgr2=0x7fd41414db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.526 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41414db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 -- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4140c7f10 msgr2=0x7fd414146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4140c7f10 0x7fd414146dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 -- 172.21.15.71:0/3660357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4040094d0 con 0x7fd4140cb190 2023-08-25T10:05:42.527 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd413fff640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4140c7f10 0x7fd414146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:42.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd418aec640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41414db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:42.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.519+0000 7fd4137fe640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 0x7fd4140c5e70 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd40802bf50 tx=0x7fd4080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.520+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40803e070 con 0x7fd4140cb190 2023-08-25T10:05:42.528 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.520+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd408039a10 con 0x7fd4140cb190 2023-08-25T10:05:42.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.520+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd408039da0 con 0x7fd4140cb190 2023-08-25T10:05:42.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.520+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd408009400 con 0x7fd4140cb190 2023-08-25T10:05:42.529 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.520+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4140ccc70 con 0x7fd4140cb190 2023-08-25T10:05:42.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.521+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41404e420 con 0x7fd4140cb190 2023-08-25T10:05:42.530 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.522+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fd4080423f0 con 0x7fd4140cb190 2023-08-25T10:05:42.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.523+0000 7fd410ff9640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd40006a080 0x7fd40006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.523+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fd4080aa500 con 0x7fd4140cb190 2023-08-25T10:05:42.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.524+0000 7fd413fff640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd40006a080 0x7fd40006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.524+0000 7fd413fff640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd40006a080 0x7fd40006c540 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd404009c50 tx=0x7fd404008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.531 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.528+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fd408075730 con 0x7fd4140cb190 2023-08-25T10:05:42.699 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.697+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5624069160 con 0x7f56240ca630 2023-08-25T10:05:42.701 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.699+0000 7f56167fc640 1 -- 172.21.15.71:0/682607876 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20877 (secure 0 0 0) 0x7f56180531f0 con 0x7f56240ca630 2023-08-25T10:05:42.701 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-08-25T10:05:42.702 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":69,"fsid":"d3b06f22-7372-48e3-a060-a2382f01af3f","created":"2023-08-25T09:49:05.740037+0000","modified":"2023-08-25T09:59:40.756440+0000","last_up_change":"2023-08-25T09:49:18.272681+0000","last_in_change":"2023-08-25T09:49:09.296445+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-25T09:49:18.704589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-25T09:49:25.686846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-08-25T09:49:31.674431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-08-25T09:49:33.068099+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-08-25T09:49:36.676444+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":67,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ec3e1497-d839-448b-b911-2c867ac9c98e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6811","nonce":555709626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6816","nonce":555709626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6821","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6823","nonce":555709626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":555709626},{"type":"v1","addr":"172.21.15.71:6820","nonce":555709626}]},"public_addr":"172.21.15.71:6811/555709626","cluster_addr":"172.21.15.71:6816/555709626","heartbeat_back_addr":"172.21.15.71:6823/555709626","heartbeat_front_addr":"172.21.15.71:6820/555709626","state":["exists","up"]},{"osd":1,"uuid":"b113539d-3236-4a8e-8000-b76810490755","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6806","nonce":3242507061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6809","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6812","nonce":3242507061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6819","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6822","nonce":3242507061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6815","nonce":3242507061},{"type":"v1","addr":"172.21.15.71:6817","nonce":3242507061}]},"public_addr":"172.21.15.71:6806/3242507061","cluster_addr":"172.21.15.71:6812/3242507061","heartbeat_back_addr":"172.21.15.71:6822/3242507061","heartbeat_front_addr":"172.21.15.71:6817/3242507061","state":["exists","up"]},{"osd":2,"uuid":"25701b56-737a-4fe4-9cf8-355b7e2491c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6801","nonce":3037968157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6803","nonce":3037968157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6813","nonce":3037968157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6805","nonce":3037968157},{"type":"v1","addr":"172.21.15.71:6807","nonce":3037968157}]},"public_addr":"172.21.15.71:6801/3037968157","cluster_addr":"172.21.15.71:6803/3037968157","heartbeat_back_addr":"172.21.15.71:6813/3037968157","heartbeat_front_addr":"172.21.15.71:6807/3037968157","state":["exists","up"]},{"osd":3,"uuid":"0d2b548b-2377-4ab5-b195-e5d93c0893b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6825","nonce":1330919310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6827","nonce":1330919310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6831","nonce":1330919310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1330919310},{"type":"v1","addr":"172.21.15.71:6829","nonce":1330919310}]},"public_addr":"172.21.15.71:6825/1330919310","cluster_addr":"172.21.15.71:6827/1330919310","heartbeat_back_addr":"172.21.15.71:6831/1330919310","heartbeat_front_addr":"172.21.15.71:6829/1330919310","state":["exists","up"]},{"osd":4,"uuid":"953c2978-0014-4223-8f3b-20fc431cfabe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6801","nonce":2125881619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6803","nonce":2125881619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6807","nonce":2125881619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2125881619},{"type":"v1","addr":"172.21.15.164:6805","nonce":2125881619}]},"public_addr":"172.21.15.164:6801/2125881619","cluster_addr":"172.21.15.164:6803/2125881619","heartbeat_back_addr":"172.21.15.164:6807/2125881619","heartbeat_front_addr":"172.21.15.164:6805/2125881619","state":["exists","up"]},{"osd":5,"uuid":"6bcee708-4862-41c2-ae1d-c2eccd7d97b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6809","nonce":1764416968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6811","nonce":1764416968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6816","nonce":1764416968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1764416968},{"type":"v1","addr":"172.21.15.164:6813","nonce":1764416968}]},"public_addr":"172.21.15.164:6809/1764416968","cluster_addr":"172.21.15.164:6811/1764416968","heartbeat_back_addr":"172.21.15.164:6816/1764416968","heartbeat_front_addr":"172.21.15.164:6813/1764416968","state":["exists","up"]},{"osd":6,"uuid":"300928fe-b078-4c53-9e5f-fc450e8773ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6825","nonce":4127242778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6827","nonce":4127242778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6831","nonce":4127242778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":4127242778},{"type":"v1","addr":"172.21.15.164:6829","nonce":4127242778}]},"public_addr":"172.21.15.164:6825/4127242778","cluster_addr":"172.21.15.164:6827/4127242778","heartbeat_back_addr":"172.21.15.164:6831/4127242778","heartbeat_front_addr":"172.21.15.164:6829/4127242778","state":["exists","up"]},{"osd":7,"uuid":"558a2369-e26a-44de-b968-f2ce93b5bdc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6817","nonce":2744841889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6819","nonce":2744841889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6823","nonce":2744841889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2744841889},{"type":"v1","addr":"172.21.15.164:6821","nonce":2744841889}]},"public_addr":"172.21.15.164:6817/2744841889","cluster_addr":"172.21.15.164:6819/2744841889","heartbeat_back_addr":"172.21.15.164:6823/2744841889","heartbeat_front_addr":"172.21.15.164:6821/2744841889","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.387541+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.611000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.395439+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.720029+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.658711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.874427+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.727073+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-25T09:49:14.695275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.a","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.71:6837/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.71:6834/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6832/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.71:6836/1556118065":"2023-08-26T09:56:20.645474+0000","172.21.15.164:6833/2402500458":"2023-08-26T09:55:05.613929+0000","172.21.15.164:6835/2707317155":"2023-08-26T09:52:30.557077+0000","172.21.15.71:6835/3404832976":"2023-08-26T09:57:50.703151+0000","172.21.15.164:6834/2707317155":"2023-08-26T09:52:30.557077+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":4,"snaps":[{"begin":504,"length":2}]},{"pool":5,"snaps":[{"begin":504,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-25T10:05:42.704 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f560806a080 msgr2=0x7f560806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f560806a080 0x7f560806c540 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5620036000 tx=0x7f562003c040 comp rx=0 tx=0).stop 2023-08-25T10:05:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 msgr2=0x7f562414a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.705 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 0x7f562414a370 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f561800c400 tx=0x7f561800c8c0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 shutdown_connections 2023-08-25T10:05:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f560806a080 0x7f560806c540 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5624152ff0 0x7f562414a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56240ca630 0x7f562414a370 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 --2- 172.21.15.71:0/682607876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f56240c9ee0 0x7f5624149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 >> 172.21.15.71:0/682607876 conn(0x7f5624065910 msgr2=0x7f5624150d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 shutdown_connections 2023-08-25T10:05:42.707 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.702+0000 7f562c0ec640 1 -- 172.21.15.71:0/682607876 wait complete. 2023-08-25T10:05:42.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-08-25T09:49:18.704589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-08-25T09:49:25.686846+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-08-25T09:49:31.674431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-08-25T09:49:33.068099+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-08-25T09:49:36.676444+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 67, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-08-25T10:05:42.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-08-25T10:05:42.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.750+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4140c9df0 con 0x7fd4140cb190 2023-08-25T10:05:42.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.751+0000 7fd410ff9640 1 -- 172.21.15.71:0/3660357 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd4080480a0 con 0x7fd4140cb190 2023-08-25T10:05:42.753 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T10:05:42.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.753+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd40006a080 msgr2=0x7fd40006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.753+0000 7fd4192ed640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd40006a080 0x7fd40006c540 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd404009c50 tx=0x7fd404008040 comp rx=0 tx=0).stop 2023-08-25T10:05:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.753+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 msgr2=0x7fd4140c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.753+0000 7fd4192ed640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 0x7fd4140c5e70 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd40802bf50 tx=0x7fd4080046c0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 shutdown_connections 2023-08-25T10:05:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fd40006a080 0x7fd40006c540 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd4140cb570 0x7fd41414db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd4140cb190 0x7fd4140c5e70 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 --2- 172.21.15.71:0/3660357 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd4140c7f10 0x7fd414146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 >> 172.21.15.71:0/3660357 conn(0x7fd41406ae20 msgr2=0x7fd414151470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 shutdown_connections 2023-08-25T10:05:42.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.754+0000 7fd4192ed640 1 -- 172.21.15.71:0/3660357 wait complete. 2023-08-25T10:05:42.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T10:05:42.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.923+0000 7f6fca6e0640 1 Processor -- start 2023-08-25T10:05:42.925 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.923+0000 7f6fca6e0640 1 -- start start 2023-08-25T10:05:42.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.925+0000 7f6fca6e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 0x7f6fc406a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.927 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fca6e0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc41590c0 con 0x7f6fc40c4510 2023-08-25T10:05:42.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fca6e0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc4159380 con 0x7f6fc40cec00 2023-08-25T10:05:42.928 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fca6e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4159660 con 0x7f6fc4153310 2023-08-25T10:05:42.929 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fc8c7d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f6fc40c4510 0x7f6fc4163670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44182/0 (socket says 172.21.15.71:44182) 2023-08-25T10:05:42.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fc8c7d640 1 -- 172.21.15.71:0/638348694 learned_addr learned my addr 172.21.15.71:0/638348694 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:42.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fc3fff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 0x7f6fc406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674831520 0 0) 0x7f6fc41590c0 con 0x7f6fc40c4510 2023-08-25T10:05:42.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.926+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb0003520 con 0x7f6fc40c4510 2023-08-25T10:05:42.930 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775837891 0 0) 0x7f6fb0003520 con 0x7f6fc40c4510 2023-08-25T10:05:42.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 msgr2=0x7f6fc406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 0x7f6fc406a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 >> v1:172.21.15.164:6789/0 conn(0x7f6fc40cec00 legacy=0x7f6fc40c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.931 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4159940 con 0x7f6fc40c4510 2023-08-25T10:05:42.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f6fbc0019f0 con 0x7f6fc40c4510 2023-08-25T10:05:42.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 >> v1:172.21.15.71:6789/0 conn(0x7f6fc40c4510 legacy=0x7f6fc4163670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb0003a10 0x7f6fb0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.932 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 0x7f6fb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc4159660 con 0x7f6fb00047c0 2023-08-25T10:05:42.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb000bb10 con 0x7f6fb0003a10 2023-08-25T10:05:42.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb000bcd0 con 0x7f6fc4153310 2023-08-25T10:05:42.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.927+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.933 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fc3fff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb0003a10 0x7f6fb0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fc8c7d640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 0x7f6fb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fbbfff640 1 -- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 msgr2=0x7f6fb0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc4153310 0x7f6fb0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fbbfff640 1 -- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb0003a10 msgr2=0x7f6fb0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb0003a10 0x7f6fb0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.934 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fbbfff640 1 -- 172.21.15.71:0/638348694 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb0003520 con 0x7f6fb00047c0 2023-08-25T10:05:42.935 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.928+0000 7f6fc3fff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fb0003a10 0x7f6fb0003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:42.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.929+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fb0004c40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6fac001750 tx=0x7f6fb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa3551479c0da00 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.929+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0052d0 con 0x7f6fb00047c0 2023-08-25T10:05:42.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.929+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac02e050 con 0x7f6fb00047c0 2023-08-25T10:05:42.937 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.929+0000 7f6fc37fe640 1 -- 172.21.15.71:0/638348694 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0035f0 con 0x7f6fb00047c0 2023-08-25T10:05:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fca6e0640 1 -- 172.21.15.71:0/638348694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 msgr2=0x7f6fb0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/638348694 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fb0004c40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f6fac001750 tx=0x7f6fb00057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fc8c7d640 1 -- 172.21.15.71:0/638348694 reap_dead start 2023-08-25T10:05:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fca6e0640 1 -- 172.21.15.71:0/638348694 shutdown_connections 2023-08-25T10:05:42.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fca6e0640 1 -- 172.21.15.71:0/638348694 >> 172.21.15.71:0/638348694 conn(0x7f6fc4065910 msgr2=0x7f6fc4065d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fca6e0640 1 -- 172.21.15.71:0/638348694 shutdown_connections 2023-08-25T10:05:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.930+0000 7f6fca6e0640 1 -- 172.21.15.71:0/638348694 wait complete. 2023-08-25T10:05:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.932+0000 7f6fca6e0640 1 Processor -- start 2023-08-25T10:05:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.932+0000 7f6fca6e0640 1 -- start start 2023-08-25T10:05:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fca6e0640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fc414c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fca6e0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc41487f0 0x7f6fc4151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fca6e0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc41479a0 0x7f6fc41f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fca6e0640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc414ae50 con 0x7f6fb00047c0 2023-08-25T10:05:42.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fca6e0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc414c0c0 con 0x7f6fc41487f0 2023-08-25T10:05:42.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fca6e0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc414b1d0 con 0x7f6fc41479a0 2023-08-25T10:05:42.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fc414c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fc414c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60000/0 (socket says 172.21.15.71:60000) 2023-08-25T10:05:42.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.933+0000 7f6fc3fff640 1 -- 172.21.15.71:0/1770862405 learned_addr learned my addr 172.21.15.71:0/1770862405 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:42.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc3fff640 1 -- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc41479a0 msgr2=0x7f6fc41f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc3fff640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc41479a0 0x7f6fc41f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc3fff640 1 -- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc41487f0 msgr2=0x7f6fc4151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:42.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc41487f0 0x7f6fc4151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc3fff640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc41487f0 0x7f6fc4151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:42.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc3fff640 1 -- 172.21.15.71:0/1770862405 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc414b4b0 con 0x7f6fb00047c0 2023-08-25T10:05:42.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc3fff640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fc414c550 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40108b0 tx=0x7f6fb4010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.942 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.934+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400eb60 con 0x7f6fb00047c0 2023-08-25T10:05:42.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.935+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb400ecc0 con 0x7f6fb00047c0 2023-08-25T10:05:42.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.935+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb401c400 con 0x7f6fb00047c0 2023-08-25T10:05:42.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.935+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc414b790 con 0x7f6fb00047c0 2023-08-25T10:05:42.943 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.935+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc414bd50 con 0x7f6fb00047c0 2023-08-25T10:05:42.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.936+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc414bf10 con 0x7f6fb00047c0 2023-08-25T10:05:42.944 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.938+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f6fb401c560 con 0x7f6fb00047c0 2023-08-25T10:05:42.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.939+0000 7f6fc17fa640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6fa406a080 0x7f6fa406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:42.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.939+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f6fb40890f0 con 0x7f6fb00047c0 2023-08-25T10:05:42.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.939+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6fa406a080 0x7f6fa406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:42.945 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.940+0000 7f6fbbfff640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6fa406a080 0x7f6fa406c540 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6fac033000 tx=0x7f6fac009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:42.946 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:42.943+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f6fb4057dd0 con 0x7f6fb00047c0 2023-08-25T10:05:43.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.015+0000 7f0ef1142640 1 Processor -- start 2023-08-25T10:05:43.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.015+0000 7f0ef1142640 1 -- start start 2023-08-25T10:05:43.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0ef1142640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 0x7f0eec0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.022 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0ef1142640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec0b1830 con 0x7f0eec0ceff0 2023-08-25T10:05:43.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0ef1142640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec159100 con 0x7f0eec1531e0 2023-08-25T10:05:43.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0ef1142640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1593e0 con 0x7f0eec0c9e00 2023-08-25T10:05:43.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0eeb59e640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0eec0ceff0 0x7f0eec0cb5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44186/0 (socket says 172.21.15.71:44186) 2023-08-25T10:05:43.023 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0eeb59e640 1 -- 172.21.15.71:0/834771556 learned_addr learned my addr 172.21.15.71:0/834771556 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:43.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.018+0000 7f0eea59c640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 0x7f0eec0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579318390 0 0) 0x7f0eec0b1830 con 0x7f0eec0ceff0 2023-08-25T10:05:43.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed4003520 con 0x7f0eec0ceff0 2023-08-25T10:05:43.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770364609 0 0) 0x7f0ed4003520 con 0x7f0eec0ceff0 2023-08-25T10:05:43.024 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 msgr2=0x7f0eec0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:43.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 0x7f0eec0ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 >> v1:172.21.15.164:6789/0 conn(0x7f0eec1531e0 legacy=0x7f0eec06a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec1596c0 con 0x7f0eec0ceff0 2023-08-25T10:05:43.025 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f0ee4001a40 con 0x7f0eec0ceff0 2023-08-25T10:05:43.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 >> v1:172.21.15.71:6789/0 conn(0x7f0eec0ceff0 legacy=0x7f0eec0cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed4003a10 0x7f0ed4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 0x7f0ed40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ed4004d00 0x7f0ed4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.026 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1593e0 con 0x7f0ed4004d00 2023-08-25T10:05:43.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed400bac0 con 0x7f0ed4003a10 2023-08-25T10:05:43.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed400bc80 con 0x7f0eec0c9e00 2023-08-25T10:05:43.027 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.019+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ed4004d00 0x7f0ed4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.020+0000 7f0eead9d640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed4003a10 0x7f0ed4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.020+0000 7f0eeb59e640 1 -- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 msgr2=0x7f0ed40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:43.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.020+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0eec0c9e00 0x7f0ed40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.020+0000 7f0eeb59e640 1 -- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed4003a10 msgr2=0x7f0ed4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:43.028 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.020+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ed4003a10 0x7f0ed4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.029 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.020+0000 7f0eeb59e640 1 -- 172.21.15.71:0/834771556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed4003520 con 0x7f0ed4004d00 2023-08-25T10:05:43.031 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.021+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ed4004d00 0x7f0ed4005180 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4002280 tx=0x7f0ed40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bc5ec018f0fffd server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.021+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4034630 con 0x7f0ed4004d00 2023-08-25T10:05:43.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.021+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee402f070 con 0x7f0ed4004d00 2023-08-25T10:05:43.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.021+0000 7f0ee9d9b640 1 -- 172.21.15.71:0/834771556 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4034a10 con 0x7f0ed4004d00 2023-08-25T10:05:43.032 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0ef1142640 1 -- 172.21.15.71:0/834771556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ed4004d00 msgr2=0x7f0ed4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0ef1142640 1 --2- 172.21.15.71:0/834771556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ed4004d00 0x7f0ed4005180 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4002280 tx=0x7f0ed40057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0eeb59e640 1 -- 172.21.15.71:0/834771556 reap_dead start 2023-08-25T10:05:43.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0ef1142640 1 -- 172.21.15.71:0/834771556 shutdown_connections 2023-08-25T10:05:43.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0ef1142640 1 -- 172.21.15.71:0/834771556 >> 172.21.15.71:0/834771556 conn(0x7f0eec06b240 msgr2=0x7f0eec06b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.033 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0ef1142640 1 -- 172.21.15.71:0/834771556 shutdown_connections 2023-08-25T10:05:43.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.022+0000 7f0ef1142640 1 -- 172.21.15.71:0/834771556 wait complete. 2023-08-25T10:05:43.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.024+0000 7f0ef1142640 1 Processor -- start 2023-08-25T10:05:43.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.024+0000 7f0ef1142640 1 -- start start 2023-08-25T10:05:43.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0ef1142640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ed4004d00 0x7f0eec15cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.034 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0ef1142640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec156420 0x7f0eec149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0ef1142640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 0x7f0eec15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0ef1142640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec15b230 con 0x7f0eec1495c0 2023-08-25T10:05:43.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0ef1142640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec149a40 con 0x7f0eec156420 2023-08-25T10:05:43.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0ef1142640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec15c720 con 0x7f0ed4004d00 2023-08-25T10:05:43.035 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0eeb59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 0x7f0eec15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0eeb59e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 0x7f0eec15a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60014/0 (socket says 172.21.15.71:60014) 2023-08-25T10:05:43.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0eeb59e640 1 -- 172.21.15.71:0/3715536105 learned_addr learned my addr 172.21.15.71:0/3715536105 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:43.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0eead9d640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ed4004d00 0x7f0eec15cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.036 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.025+0000 7f0eea59c640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec156420 0x7f0eec149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0eeb59e640 1 -- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ed4004d00 msgr2=0x7f0eec15cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ed4004d00 0x7f0eec15cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0eeb59e640 1 -- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec156420 msgr2=0x7f0eec149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec156420 0x7f0eec149080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.037 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0eeb59e640 1 -- 172.21.15.71:0/3715536105 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec1503e0 con 0x7f0eec1495c0 2023-08-25T10:05:43.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0eeb59e640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 0x7f0eec15a160 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40011e0 tx=0x7f0eec06a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4034630 con 0x7f0eec1495c0 2023-08-25T10:05:43.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee4001b30 con 0x7f0eec1495c0 2023-08-25T10:05:43.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4034990 con 0x7f0eec1495c0 2023-08-25T10:05:43.038 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.026+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec150140 con 0x7f0eec1495c0 2023-08-25T10:05:43.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.027+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec1ef4b0 con 0x7f0eec1495c0 2023-08-25T10:05:43.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.028+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec151c60 con 0x7f0eec1495c0 2023-08-25T10:05:43.039 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.029+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f0ee4031030 con 0x7f0eec1495c0 2023-08-25T10:05:43.040 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.030+0000 7f0ed37fe640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed806a080 0x7f0ed806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.030+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f0ee40aa6a0 con 0x7f0eec1495c0 2023-08-25T10:05:43.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.031+0000 7f0eead9d640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed806a080 0x7f0ed806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.031+0000 7f0eead9d640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed806a080 0x7f0ed806c540 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0edc007fd0 tx=0x7f0edc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.034+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f0ee40758c0 con 0x7f0eec1495c0 2023-08-25T10:05:43.142 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.140+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6fc4069160 con 0x7f6fb00047c0 2023-08-25T10:05:43.143 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.141+0000 7f6fc17fa640 1 -- 172.21.15.71:0/1770862405 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fb4057960 con 0x7f6fb00047c0 2023-08-25T10:05:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 1 2023-08-25T10:05:43.145 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.143+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6fa406a080 msgr2=0x7f6fa406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.143+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6fa406a080 0x7f6fa406c540 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6fac033000 tx=0x7f6fac009030 comp rx=0 tx=0).stop 2023-08-25T10:05:43.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.143+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 msgr2=0x7f6fc414c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.143+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fc414c550 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40108b0 tx=0x7f6fb4010d70 comp rx=0 tx=0).stop 2023-08-25T10:05:43.146 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 shutdown_connections 2023-08-25T10:05:43.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fc41487f0 0x7f6fc4151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f6fa406a080 0x7f6fa406c540 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fc41479a0 0x7f6fc41f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 --2- 172.21.15.71:0/1770862405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6fb00047c0 0x7f6fc414c550 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.147 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 >> 172.21.15.71:0/1770862405 conn(0x7f6fc4065910 msgr2=0x7f6fc4156c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 shutdown_connections 2023-08-25T10:05:43.148 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.144+0000 7f6fca6e0640 1 -- 172.21.15.71:0/1770862405 wait complete. 2023-08-25T10:05:43.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-08-25T10:05:43.231 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.229+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0eec151c60 con 0x7f0eec1495c0 2023-08-25T10:05:43.232 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.230+0000 7f0ed37fe640 1 -- 172.21.15.71:0/3715536105 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v69) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0ee4078ee0 con 0x7f0eec1495c0 2023-08-25T10:05:43.232 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:43.236 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed806a080 msgr2=0x7f0ed806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed806a080 0x7f0ed806c540 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0edc007fd0 tx=0x7f0edc009040 comp rx=0 tx=0).stop 2023-08-25T10:05:43.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 msgr2=0x7f0eec15a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 0x7f0eec15a160 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40011e0 tx=0x7f0eec06a280 comp rx=0 tx=0).stop 2023-08-25T10:05:43.237 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 shutdown_connections 2023-08-25T10:05:43.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0eec156420 0x7f0eec149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f0ed806a080 0x7f0ed806c540 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0eec1495c0 0x7f0eec15a160 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 --2- 172.21.15.71:0/3715536105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ed4004d00 0x7f0eec15cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.238 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.232+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 >> 172.21.15.71:0/3715536105 conn(0x7f0eec06b240 msgr2=0x7f0eec0ca530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.233+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 shutdown_connections 2023-08-25T10:05:43.239 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.233+0000 7f0ef1142640 1 -- 172.21.15.71:0/3715536105 wait complete. 2023-08-25T10:05:43.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T10:05:43.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.368+0000 7f352c342640 1 Processor -- start 2023-08-25T10:05:43.371 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.368+0000 7f352c342640 1 -- start start 2023-08-25T10:05:43.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352c342640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3524152d80 0x7f35240c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.373 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352c342640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 0x7f35240c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352c342640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35241591c0 con 0x7f35240cacb0 2023-08-25T10:05:43.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352c342640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35241594a0 con 0x7f3524152d80 2023-08-25T10:05:43.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352c342640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3524159780 con 0x7f352415ce50 2023-08-25T10:05:43.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352a0de640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3524152d80 0x7f35240c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.374 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352a8df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 0x7f35240c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352a8df640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 0x7f35240c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.71:53372/0 (socket says 172.21.15.71:53372) 2023-08-25T10:05:43.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352a0de640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3524152d80 0x7f35240c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.71:57868/0 (socket says 172.21.15.71:57868) 2023-08-25T10:05:43.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f352a8df640 1 -- 172.21.15.71:0/628858627 learned_addr learned my addr 172.21.15.71:0/628858627 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:43.375 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.370+0000 7f35298dd640 1 --1- 172.21.15.71:0/628858627 >> v1:172.21.15.164:6789/0 conn(0x7f35240cacb0 0x7f352415c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.71:37398/0 (socket says 172.21.15.71:37398) 2023-08-25T10:05:43.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.371+0000 7f35290dc640 1 -- 172.21.15.71:0/628858627 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644675511 0 0) 0x7f35241591c0 con 0x7f35240cacb0 2023-08-25T10:05:43.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.371+0000 7f352a8df640 1 -- 172.21.15.71:0/628858627 >> v2:172.21.15.164:3300/0 conn(0x7f3524152d80 msgr2=0x7f35240c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:43.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.371+0000 7f352a8df640 1 --2- 172.21.15.71:0/628858627 >> v2:172.21.15.164:3300/0 conn(0x7f3524152d80 0x7f35240c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.376 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.371+0000 7f352a8df640 1 -- 172.21.15.71:0/628858627 >> v1:172.21.15.164:6789/0 conn(0x7f35240cacb0 legacy=0x7f352415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.377 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.371+0000 7f352a8df640 1 -- 172.21.15.71:0/628858627 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524159a60 con 0x7f352415ce50 2023-08-25T10:05:43.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.372+0000 7f352a8df640 1 --2- 172.21.15.71:0/628858627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 0x7f35240c98a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f351c009e50 tx=0x7f351c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4ca481d057cbf0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.378 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.373+0000 7f35290dc640 1 -- 172.21.15.71:0/628858627 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c02cb90 con 0x7f352415ce50 2023-08-25T10:05:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.373+0000 7f35290dc640 1 -- 172.21.15.71:0/628858627 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351c02ccf0 con 0x7f352415ce50 2023-08-25T10:05:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.373+0000 7f35290dc640 1 -- 172.21.15.71:0/628858627 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c034610 con 0x7f352415ce50 2023-08-25T10:05:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 -- 172.21.15.71:0/628858627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 msgr2=0x7f35240c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.379 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 --2- 172.21.15.71:0/628858627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 0x7f35240c98a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f351c009e50 tx=0x7f351c02bff0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 -- 172.21.15.71:0/628858627 shutdown_connections 2023-08-25T10:05:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 --2- 172.21.15.71:0/628858627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f352415ce50 0x7f35240c98a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 --2- 172.21.15.71:0/628858627 >> v2:172.21.15.164:3300/0 conn(0x7f3524152d80 0x7f35240c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 -- 172.21.15.71:0/628858627 >> 172.21.15.71:0/628858627 conn(0x7f352406b8f0 msgr2=0x7f35240654f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.380 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 -- 172.21.15.71:0/628858627 shutdown_connections 2023-08-25T10:05:43.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.374+0000 7f352c342640 1 -- 172.21.15.71:0/628858627 wait complete. 2023-08-25T10:05:43.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.376+0000 7f352c342640 1 Processor -- start 2023-08-25T10:05:43.381 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.376+0000 7f352c342640 1 -- start start 2023-08-25T10:05:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352c342640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35240cacb0 0x7f3524147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352c342640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3524152d80 0x7f35241efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352c342640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 0x7f35241f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.385 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352c342640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352414d400 con 0x7f352415ce50 2023-08-25T10:05:43.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352c342640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352415c220 con 0x7f35240cacb0 2023-08-25T10:05:43.386 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352c342640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f352415b410 con 0x7f3524152d80 2023-08-25T10:05:43.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352a8df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 0x7f35241f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.387 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352a8df640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 0x7f35241f02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60024/0 (socket says 172.21.15.71:60024) 2023-08-25T10:05:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.377+0000 7f352a8df640 1 -- 172.21.15.71:0/1222262145 learned_addr learned my addr 172.21.15.71:0/1222262145 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a0de640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35240cacb0 0x7f3524147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.388 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f35298dd640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3524152d80 0x7f35241efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a8df640 1 -- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3524152d80 msgr2=0x7f35241efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.389 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a8df640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3524152d80 0x7f35241efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a8df640 1 -- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35240cacb0 msgr2=0x7f3524147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.390 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a8df640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35240cacb0 0x7f3524147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a8df640 1 -- 172.21.15.71:0/1222262145 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c009b30 con 0x7f352415ce50 2023-08-25T10:05:43.391 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f352a8df640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 0x7f35241f02b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f351c036040 tx=0x7f351c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.378+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c040670 con 0x7f352415ce50 2023-08-25T10:05:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.379+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351c040810 con 0x7f352415ce50 2023-08-25T10:05:43.392 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.379+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352415ab70 con 0x7f352415ce50 2023-08-25T10:05:43.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.379+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c040bc0 con 0x7f352415ce50 2023-08-25T10:05:43.393 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.379+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f352415b130 con 0x7f352415ce50 2023-08-25T10:05:43.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.380+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35240b1a60 con 0x7f352415ce50 2023-08-25T10:05:43.394 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.381+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f351c040d20 con 0x7f352415ce50 2023-08-25T10:05:43.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.381+0000 7f351affd640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f350806a050 0x7f350806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.395 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.382+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f351c0ab9f0 con 0x7f352415ce50 2023-08-25T10:05:43.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.382+0000 7f352a0de640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f350806a050 0x7f350806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.396 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.382+0000 7f352a0de640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f350806a050 0x7f350806c510 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3520009ad0 tx=0x7f3520008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.398 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.386+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f351c076c10 con 0x7f352415ce50 2023-08-25T10:05:43.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.459+0000 7fe2c940d640 1 Processor -- start 2023-08-25T10:05:43.461 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.459+0000 7fe2c940d640 1 -- start start 2023-08-25T10:05:43.462 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.460+0000 7fe2c940d640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe2c41574a0 0x7fe2c415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c940d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe2c4069850 0x7fe2c40c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c940d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c415d8c0 con 0x7fe2c40c3110 2023-08-25T10:05:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c940d640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c415db80 con 0x7fe2c41574a0 2023-08-25T10:05:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c940d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c415de60 con 0x7fe2c4069850 2023-08-25T10:05:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c37fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fe2c40c3110 0x7fe2c40c6110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44200/0 (socket says 172.21.15.71:44200) 2023-08-25T10:05:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c2ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe2c41574a0 0x7fe2c415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.464 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c37fe640 1 -- 172.21.15.71:0/1792972466 learned_addr learned my addr 172.21.15.71:0/1792972466 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:43.465 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c27fc640 1 --2- 172.21.15.71:0/1792972466 >> v2:172.21.15.164:3300/0 conn(0x7fe2c4069850 0x7fe2c40c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/1792972466 >> v2:172.21.15.164:3300/0 conn(0x7fe2c4069850 msgr2=0x7fe2c40c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c2ffd640 1 --2- 172.21.15.71:0/1792972466 >> v2:172.21.15.164:3300/0 conn(0x7fe2c4069850 0x7fe2c40c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/1792972466 >> v1:172.21.15.71:6789/0 conn(0x7fe2c40c3110 legacy=0x7fe2c40c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.462+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/1792972466 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c415e140 con 0x7fe2c41574a0 2023-08-25T10:05:43.466 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.461+0000 7fe2c1ffb640 1 -- 172.21.15.71:0/1792972466 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063458379 0 0) 0x7fe2c415d8c0 con 0x7fe2c40c3110 2023-08-25T10:05:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.463+0000 7fe2c2ffd640 1 --2- 172.21.15.71:0/1792972466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c41574a0 0x7fe2c415d2d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4069ca0 tx=0x7fe2b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9e8e2028917ef8 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.463+0000 7fe2c1ffb640 1 -- 172.21.15.71:0/1792972466 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4030610 con 0x7fe2c41574a0 2023-08-25T10:05:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.463+0000 7fe2c1ffb640 1 -- 172.21.15.71:0/1792972466 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4030770 con 0x7fe2c41574a0 2023-08-25T10:05:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.463+0000 7fe2c1ffb640 1 -- 172.21.15.71:0/1792972466 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4030b00 con 0x7fe2c41574a0 2023-08-25T10:05:43.467 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.463+0000 7fe2c940d640 1 -- 172.21.15.71:0/1792972466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c41574a0 msgr2=0x7fe2c415d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.463+0000 7fe2c940d640 1 --2- 172.21.15.71:0/1792972466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c41574a0 0x7fe2c415d2d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4069ca0 tx=0x7fe2b402b960 comp rx=0 tx=0).stop 2023-08-25T10:05:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.464+0000 7fe2c940d640 1 -- 172.21.15.71:0/1792972466 shutdown_connections 2023-08-25T10:05:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.464+0000 7fe2c940d640 1 --2- 172.21.15.71:0/1792972466 >> v2:172.21.15.164:3300/0 conn(0x7fe2c4069850 0x7fe2c40c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.464+0000 7fe2c940d640 1 --2- 172.21.15.71:0/1792972466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c41574a0 0x7fe2c415d2d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.468 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.464+0000 7fe2c940d640 1 -- 172.21.15.71:0/1792972466 >> 172.21.15.71:0/1792972466 conn(0x7fe2c40c75c0 msgr2=0x7fe2c40c79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.464+0000 7fe2c940d640 1 -- 172.21.15.71:0/1792972466 shutdown_connections 2023-08-25T10:05:43.469 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.464+0000 7fe2c940d640 1 -- 172.21.15.71:0/1792972466 wait complete. 2023-08-25T10:05:43.470 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.465+0000 7fe2c940d640 1 Processor -- start 2023-08-25T10:05:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 -- start start 2023-08-25T10:05:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 0x7fe2c41f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2c40c3110 0x7fe2c415b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2c41574a0 0x7fe2c406f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.471 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c41f3760 con 0x7fe2c4069850 2023-08-25T10:05:43.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c406f0f0 con 0x7fe2c40c3110 2023-08-25T10:05:43.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.466+0000 7fe2c940d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c4071e00 con 0x7fe2c41574a0 2023-08-25T10:05:43.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 0x7fe2c41f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 0x7fe2c41f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60036/0 (socket says 172.21.15.71:60036) 2023-08-25T10:05:43.472 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/17530313 learned_addr learned my addr 172.21.15.71:0/17530313 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:43.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2c41574a0 msgr2=0x7fe2c406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2c41574a0 0x7fe2c406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c27fc640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2c40c3110 0x7fe2c415b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2c40c3110 msgr2=0x7fe2c415b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2c40c3110 0x7fe2c415b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.473 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.467+0000 7fe2c2ffd640 1 -- 172.21.15.71:0/17530313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b4009400 con 0x7fe2c4069850 2023-08-25T10:05:43.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.468+0000 7fe2c2ffd640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 0x7fe2c41f3220 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009890 tx=0x7fe2b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.468+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b403e070 con 0x7fe2c4069850 2023-08-25T10:05:43.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.468+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b4006e10 con 0x7fe2c4069850 2023-08-25T10:05:43.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.468+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4039c30 con 0x7fe2c4069850 2023-08-25T10:05:43.474 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.468+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c4071560 con 0x7fe2c4069850 2023-08-25T10:05:43.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.468+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c4071b20 con 0x7fe2c4069850 2023-08-25T10:05:43.475 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.469+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c40c7f60 con 0x7fe2c4069850 2023-08-25T10:05:43.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.470+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe2b404b400 con 0x7fe2c4069850 2023-08-25T10:05:43.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.471+0000 7fe2ab7fe640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe2b006a080 0x7fe2b006c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.471+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe2b40aac80 con 0x7fe2c4069850 2023-08-25T10:05:43.476 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.471+0000 7fe2c27fc640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe2b006a080 0x7fe2b006c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.472+0000 7fe2c27fc640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe2b006a080 0x7fe2b006c540 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4152b80 tx=0x7fe2b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.477 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.475+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe2b4075f50 con 0x7fe2c4069850 2023-08-25T10:05:43.591 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.590+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3524069160 con 0x7f352415ce50 2023-08-25T10:05:43.592 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.590+0000 7f351affd640 1 -- 172.21.15.71:0/1222262145 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v69) v1 ==== 92+0+10 (secure 0 0 0) 0x7f351c009510 con 0x7f352415ce50 2023-08-25T10:05:43.592 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 8 2023-08-25T10:05:43.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f350806a050 msgr2=0x7f350806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f350806a050 0x7f350806c510 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3520009ad0 tx=0x7f3520008040 comp rx=0 tx=0).stop 2023-08-25T10:05:43.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 msgr2=0x7f35241f02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.595 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 0x7f35241f02b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f351c036040 tx=0x7f351c004a40 comp rx=0 tx=0).stop 2023-08-25T10:05:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 shutdown_connections 2023-08-25T10:05:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f350806a050 0x7f350806c510 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f352415ce50 0x7f35241f02b0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3524152d80 0x7f35241efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.596 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 --2- 172.21.15.71:0/1222262145 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35240cacb0 0x7f3524147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 >> 172.21.15.71:0/1222262145 conn(0x7f352406b8f0 msgr2=0x7f35240c9160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 shutdown_connections 2023-08-25T10:05:43.597 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.593+0000 7f352c342640 1 -- 172.21.15.71:0/1222262145 wait complete. 2023-08-25T10:05:43.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-08-25T10:05:43.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.681+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe2c4069160 con 0x7fe2c4069850 2023-08-25T10:05:43.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.682+0000 7fe2ab7fe640 1 -- 172.21.15.71:0/17530313 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v69) v1 ==== 80+0+492 (secure 0 0 0) 0x7fe2b4079570 con 0x7fe2c4069850 2023-08-25T10:05:43.685 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6837/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:43.685 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6834/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:43.686 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6832/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:43.686 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6836/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:43.686 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6833/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:43.687 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6835/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:43.687 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6835/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:43.687 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6834/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:43.688 INFO:teuthology.orchestra.run.smithi071.stderr:listed 8 entries 2023-08-25T10:05:43.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.684+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe2b006a080 msgr2=0x7fe2b006c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.684+0000 7fe2c940d640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe2b006a080 0x7fe2b006c540 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4152b80 tx=0x7fe2b8008040 comp rx=0 tx=0).stop 2023-08-25T10:05:43.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.684+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 msgr2=0x7fe2c41f3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.684+0000 7fe2c940d640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 0x7fe2c41f3220 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009890 tx=0x7fe2b40046c0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.684+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 shutdown_connections 2023-08-25T10:05:43.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.684+0000 7fe2c940d640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2c40c3110 0x7fe2c415b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.685+0000 7fe2c940d640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe2b006a080 0x7fe2b006c540 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.685+0000 7fe2c940d640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe2c41574a0 0x7fe2c406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.685+0000 7fe2c940d640 1 --2- 172.21.15.71:0/17530313 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe2c4069850 0x7fe2c41f3220 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.685+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 >> 172.21.15.71:0/17530313 conn(0x7fe2c40c75c0 msgr2=0x7fe2c406a880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.685+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 shutdown_connections 2023-08-25T10:05:43.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.685+0000 7fe2c940d640 1 -- 172.21.15.71:0/17530313 wait complete. 2023-08-25T10:05:43.703 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi071.front.sepia.ceph.com... 2023-08-25T10:05:43.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T10:05:43.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-08-25T10:05:43.773 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:05:43.773 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T10:05:43.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-08-25T10:05:43.775 DEBUG:teuthology.orchestra.run.smithi071:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-08-25T10:05:43.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.809+0000 7f608759e640 1 Processor -- start 2023-08-25T10:05:43.811 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.809+0000 7f608759e640 1 -- start start 2023-08-25T10:05:43.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f608759e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f6088159490 0x7f60880ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.813 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f608759e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880b1af0 con 0x7f60880cb190 2023-08-25T10:05:43.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f608759e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880b0fb0 con 0x7f60880cb570 2023-08-25T10:05:43.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f608759e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60880c98f0 con 0x7f6088159490 2023-08-25T10:05:43.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f608659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f60880cb190 0x7f60880c7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44202/0 (socket says 172.21.15.71:44202) 2023-08-25T10:05:43.814 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f608659c640 1 -- 172.21.15.71:0/3337949079 learned_addr learned my addr 172.21.15.71:0/3337949079 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:43.815 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.811+0000 7f6086d9d640 1 --2- 172.21.15.71:0/3337949079 >> v2:172.21.15.71:3300/0 conn(0x7f6088159490 0x7f60880ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.817 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.812+0000 7f6086d9d640 1 -- 172.21.15.71:0/3337949079 >> v1:172.21.15.164:6789/0 conn(0x7f60880cb570 legacy=0x7f6088158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:43.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.812+0000 7f6086d9d640 1 -- 172.21.15.71:0/3337949079 >> v1:172.21.15.71:6789/0 conn(0x7f60880cb190 legacy=0x7f60880c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.818 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.812+0000 7f6086d9d640 1 -- 172.21.15.71:0/3337949079 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880b1530 con 0x7f6088159490 2023-08-25T10:05:43.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.812+0000 7f608559a640 1 -- 172.21.15.71:0/3337949079 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767096348 0 0) 0x7f60880b1af0 con 0x7f60880cb190 2023-08-25T10:05:43.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.813+0000 7f6086d9d640 1 --2- 172.21.15.71:0/3337949079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6088159490 0x7f60880ca910 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f60880b1720 tx=0x7f608002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c11d46dbcdd16d6 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.819 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.813+0000 7f608559a640 1 -- 172.21.15.71:0/3337949079 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608002cce0 con 0x7f6088159490 2023-08-25T10:05:43.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.813+0000 7f608559a640 1 -- 172.21.15.71:0/3337949079 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608002ce40 con 0x7f6088159490 2023-08-25T10:05:43.820 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.813+0000 7f608559a640 1 -- 172.21.15.71:0/3337949079 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080034660 con 0x7f6088159490 2023-08-25T10:05:43.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 -- 172.21.15.71:0/3337949079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6088159490 msgr2=0x7f60880ca910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.821 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 --2- 172.21.15.71:0/3337949079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6088159490 0x7f60880ca910 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f60880b1720 tx=0x7f608002bf30 comp rx=0 tx=0).stop 2023-08-25T10:05:43.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 -- 172.21.15.71:0/3337949079 shutdown_connections 2023-08-25T10:05:43.822 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 --2- 172.21.15.71:0/3337949079 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6088159490 0x7f60880ca910 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 -- 172.21.15.71:0/3337949079 >> 172.21.15.71:0/3337949079 conn(0x7f608806ae20 msgr2=0x7f608806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:43.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 -- 172.21.15.71:0/3337949079 shutdown_connections 2023-08-25T10:05:43.823 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.814+0000 7f608759e640 1 -- 172.21.15.71:0/3337949079 wait complete. 2023-08-25T10:05:43.827 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.816+0000 7f608759e640 1 Processor -- start 2023-08-25T10:05:43.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.816+0000 7f608759e640 1 -- start start 2023-08-25T10:05:43.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60880cb190 0x7f60881f1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.828 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608759e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 0x7f6088155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6088154c10 0x7f60881562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.829 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608759e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6088154a60 con 0x7f60880cb570 2023-08-25T10:05:43.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608759e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60881558e0 con 0x7f60880cb190 2023-08-25T10:05:43.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608759e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60881544c0 con 0x7f6088154c10 2023-08-25T10:05:43.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 0x7f6088155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.830 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 0x7f6088155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60056/0 (socket says 172.21.15.71:60056) 2023-08-25T10:05:43.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 -- 172.21.15.71:0/2188655557 learned_addr learned my addr 172.21.15.71:0/2188655557 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:43.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f608659c640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60880cb190 0x7f60881f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 -- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6088154c10 msgr2=0x7f60881562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.831 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6088154c10 0x7f60881562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 -- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60880cb190 msgr2=0x7f60881f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:43.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60880cb190 0x7f60881f1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:43.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.817+0000 7f6085d9b640 1 -- 172.21.15.71:0/2188655557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6080009a70 con 0x7f60880cb570 2023-08-25T10:05:43.832 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.818+0000 7f6085d9b640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 0x7f6088155d70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f607c00c400 tx=0x7f607c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.818+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c011790 con 0x7f60880cb570 2023-08-25T10:05:43.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.818+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607c011930 con 0x7f60880cb570 2023-08-25T10:05:43.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.818+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607c011ce0 con 0x7f60880cb570 2023-08-25T10:05:43.833 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.818+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60881f2220 con 0x7f60880cb570 2023-08-25T10:05:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.818+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6088154780 con 0x7f60880cb570 2023-08-25T10:05:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.819+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608804e420 con 0x7f60880cb570 2023-08-25T10:05:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.820+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f607c01c460 con 0x7f60880cb570 2023-08-25T10:05:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.821+0000 7f6076ffd640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f606806a080 0x7f606806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:43.834 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.821+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f607c088a60 con 0x7f60880cb570 2023-08-25T10:05:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.821+0000 7f608659c640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f606806a080 0x7f606806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.822+0000 7f608659c640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f606806a080 0x7f606806c540 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6078007fd0 tx=0x7f6078009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:43.835 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:43.825+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f607c053b60 con 0x7f60880cb570 2023-08-25T10:05:44.015 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.013+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6088155a20 con 0x7f60880cb570 2023-08-25T10:05:44.016 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.014+0000 7f6076ffd640 1 -- 172.21.15.71:0/2188655557 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f607c057180 con 0x7f60880cb570 2023-08-25T10:05:44.016 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:44.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f606806a080 msgr2=0x7f606806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f606806a080 0x7f606806c540 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6078007fd0 tx=0x7f6078009040 comp rx=0 tx=0).stop 2023-08-25T10:05:44.018 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 msgr2=0x7f6088155d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 0x7f6088155d70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f607c00c400 tx=0x7f607c00c8c0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 shutdown_connections 2023-08-25T10:05:44.019 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f606806a080 0x7f606806c540 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6088154c10 0x7f60881562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f60880cb570 0x7f6088155d70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.020 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 --2- 172.21.15.71:0/2188655557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60880cb190 0x7f60881f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 >> 172.21.15.71:0/2188655557 conn(0x7f608806ae20 msgr2=0x7f60880c6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:44.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 shutdown_connections 2023-08-25T10:05:44.021 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.016+0000 7f608759e640 1 -- 172.21.15.71:0/2188655557 wait complete. 2023-08-25T10:05:44.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-08-25T10:05:44.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.299+0000 7f34f1383640 1 Processor -- start 2023-08-25T10:05:44.301 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.299+0000 7f34f1383640 1 -- start start 2023-08-25T10:05:44.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.300+0000 7f34f1383640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f34ec06cc70 0x7f34ec0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.303 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.300+0000 7f34f1383640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec0c3940 0x7f34ec1598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.300+0000 7f34f1383640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec06a960 con 0x7f34ec1594d0 2023-08-25T10:05:44.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.300+0000 7f34f1383640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ec06ac20 con 0x7f34ec06cc70 2023-08-25T10:05:44.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.300+0000 7f34f1383640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec06af00 con 0x7f34ec0c3940 2023-08-25T10:05:44.304 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f34ec06cc70 0x7f34ec0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f34ec06cc70 0x7f34ec0c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60064/0 (socket says 172.21.15.71:60064) 2023-08-25T10:05:44.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 -- 172.21.15.71:0/724169156 learned_addr learned my addr 172.21.15.71:0/724169156 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:44.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34eb7fe640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec0c3940 0x7f34ec1598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 -- 172.21.15.71:0/724169156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec0c3940 msgr2=0x7f34ec1598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:44.305 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec0c3940 0x7f34ec1598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 -- 172.21.15.71:0/724169156 >> v1:172.21.15.164:6789/0 conn(0x7f34ec1594d0 legacy=0x7f34ec14f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:44.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34e3fff640 1 -- 172.21.15.71:0/724169156 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec0c2e60 con 0x7f34ec06cc70 2023-08-25T10:05:44.306 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.301+0000 7f34eb7fe640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec0c3940 0x7f34ec1598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:44.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.302+0000 7f34e3fff640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec0c4c90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009870 tx=0x7f34d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dd6b57abf18311 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:44.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34ea7fc640 1 -- 172.21.15.71:0/724169156 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d4031640 con 0x7f34ec06cc70 2023-08-25T10:05:44.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34ea7fc640 1 -- 172.21.15.71:0/724169156 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d40317a0 con 0x7f34ec06cc70 2023-08-25T10:05:44.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34ea7fc640 1 -- 172.21.15.71:0/724169156 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d4031b30 con 0x7f34ec06cc70 2023-08-25T10:05:44.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 -- 172.21.15.71:0/724169156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 msgr2=0x7f34ec0c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec0c4c90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f34d4009870 tx=0x7f34d402b8f0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 -- 172.21.15.71:0/724169156 shutdown_connections 2023-08-25T10:05:44.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec0c3940 0x7f34ec1598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 --2- 172.21.15.71:0/724169156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec0c4c90 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 -- 172.21.15.71:0/724169156 >> 172.21.15.71:0/724169156 conn(0x7f34ec0654f0 msgr2=0x7f34ec065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:44.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 -- 172.21.15.71:0/724169156 shutdown_connections 2023-08-25T10:05:44.309 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.303+0000 7f34f1383640 1 -- 172.21.15.71:0/724169156 wait complete. 2023-08-25T10:05:44.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.305+0000 7f34f1383640 1 Processor -- start 2023-08-25T10:05:44.310 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.305+0000 7f34f1383640 1 -- start start 2023-08-25T10:05:44.311 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34f1383640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34f1383640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34ec0c3940 0x7f34ec0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34f1383640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec1594d0 0x7f34ec1ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34f1383640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec13f3f0 con 0x7f34ec06cc70 2023-08-25T10:05:44.312 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34f1383640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec139ff0 con 0x7f34ec0c3940 2023-08-25T10:05:44.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34f1383640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec0703e0 con 0x7f34ec1594d0 2023-08-25T10:05:44.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34eaffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34eaffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec13eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60068/0 (socket says 172.21.15.71:60068) 2023-08-25T10:05:44.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.306+0000 7f34eaffd640 1 -- 172.21.15.71:0/1476351972 learned_addr learned my addr 172.21.15.71:0/1476351972 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:44.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eb7fe640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec1594d0 0x7f34ec1ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.313 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34e3fff640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34ec0c3940 0x7f34ec0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eaffd640 1 -- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec1594d0 msgr2=0x7f34ec1ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eaffd640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec1594d0 0x7f34ec1ef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eaffd640 1 -- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34ec0c3940 msgr2=0x7f34ec0c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eaffd640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34ec0c3940 0x7f34ec0c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.314 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eaffd640 1 -- 172.21.15.71:0/1476351972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40094d0 con 0x7f34ec06cc70 2023-08-25T10:05:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34e3fff640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34ec0c3940 0x7f34ec0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34eaffd640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec13eeb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00c430 tx=0x7f34dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc011790 con 0x7f34ec06cc70 2023-08-25T10:05:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc0118f0 con 0x7f34ec06cc70 2023-08-25T10:05:44.315 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc011ce0 con 0x7f34ec06cc70 2023-08-25T10:05:44.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.307+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ec0706c0 con 0x7f34ec06cc70 2023-08-25T10:05:44.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.308+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ec13a570 con 0x7f34ec06cc70 2023-08-25T10:05:44.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.309+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ec14ef00 con 0x7f34ec06cc70 2023-08-25T10:05:44.316 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.310+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f34dc01c460 con 0x7f34ec06cc70 2023-08-25T10:05:44.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.311+0000 7f34e37fe640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f34cc06a050 0x7f34cc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.311+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f34dc088a30 con 0x7f34ec06cc70 2023-08-25T10:05:44.317 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.311+0000 7f34e3fff640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f34cc06a050 0x7f34cc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.312+0000 7f34e3fff640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f34cc06a050 0x7f34cc06c510 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f34d4036000 tx=0x7f34d403c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:44.318 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.316+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f34dc053c50 con 0x7f34ec06cc70 2023-08-25T10:05:44.508 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.506+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34ec069160 con 0x7f34ec06cc70 2023-08-25T10:05:44.509 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.507+0000 7f34e37fe640 1 -- 172.21.15.71:0/1476351972 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34dc057270 con 0x7f34ec06cc70 2023-08-25T10:05:44.509 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 64 2023-08-25T10:05:44.512 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f34cc06a050 msgr2=0x7f34cc06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f34cc06a050 0x7f34cc06c510 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f34d4036000 tx=0x7f34d403c040 comp rx=0 tx=0).stop 2023-08-25T10:05:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 msgr2=0x7f34ec13eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec13eeb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f34dc00c430 tx=0x7f34dc00c8f0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 shutdown_connections 2023-08-25T10:05:44.513 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f34ec0c3940 0x7f34ec0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f34cc06a050 0x7f34cc06c510 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34ec1594d0 0x7f34ec1ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 --2- 172.21.15.71:0/1476351972 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f34ec06cc70 0x7f34ec13eeb0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 >> 172.21.15.71:0/1476351972 conn(0x7f34ec0654f0 msgr2=0x7f34ec0c6300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:44.514 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 shutdown_connections 2023-08-25T10:05:44.515 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.510+0000 7f34f1383640 1 -- 172.21.15.71:0/1476351972 wait complete. 2023-08-25T10:05:44.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-08-25T10:05:44.747 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.745+0000 7f4ab92bf640 1 Processor -- start 2023-08-25T10:05:44.748 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.745+0000 7f4ab92bf640 1 -- start start 2023-08-25T10:05:44.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.747+0000 7f4ab92bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 0x7f4ab40c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.747+0000 7f4ab92bf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab406a960 con 0x7f4ab406cc70 2023-08-25T10:05:44.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.747+0000 7f4ab92bf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab406ac20 con 0x7f4ab40c3940 2023-08-25T10:05:44.749 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.747+0000 7f4ab92bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab406af00 con 0x7f4ab41594d0 2023-08-25T10:05:44.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.747+0000 7f4aaa7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4ab406cc70 0x7f4ab414f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44216/0 (socket says 172.21.15.71:44216) 2023-08-25T10:05:44.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.747+0000 7f4aaa7fc640 1 -- 172.21.15.71:0/1154523509 learned_addr learned my addr 172.21.15.71:0/1154523509 (peer_addr_for_me v1:172.21.15.71:0/0) 2023-08-25T10:05:44.751 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab2ffd640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 0x7f4ab40c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249577396 0 0) 0x7f4ab406a960 con 0x7f4ab406cc70 2023-08-25T10:05:44.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa0003520 con 0x7f4ab406cc70 2023-08-25T10:05:44.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110363383 0 0) 0x7f4aa0003520 con 0x7f4ab406cc70 2023-08-25T10:05:44.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 msgr2=0x7f4ab40c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:44.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 0x7f4ab40c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 >> v1:172.21.15.164:6789/0 conn(0x7f4ab40c3940 legacy=0x7f4ab41598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab40c2e60 con 0x7f4ab406cc70 2023-08-25T10:05:44.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2136166316 0 0) 0x7f4a9c0019f0 con 0x7f4ab406cc70 2023-08-25T10:05:44.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 >> v1:172.21.15.71:6789/0 conn(0x7f4ab406cc70 legacy=0x7f4ab414f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa0003a10 0x7f4aa0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.748+0000 7f4ab27fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4aa00047c0 0x7f4aa0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab27fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 0x7f4aa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab406af00 con 0x7f4aa00047c0 2023-08-25T10:05:44.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa000ba80 con 0x7f4aa0003a10 2023-08-25T10:05:44.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa000bc40 con 0x7f4ab41594d0 2023-08-25T10:05:44.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4aaa7fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4aa00047c0 0x7f4aa0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab2ffd640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa0003a10 0x7f4aa0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4aaa7fc640 1 -- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 msgr2=0x7f4aa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:44.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab37fe640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 0x7f4aa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4aaa7fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ab41594d0 0x7f4aa0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4aaa7fc640 1 -- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa0003a10 msgr2=0x7f4aa0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:44.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4aaa7fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa0003a10 0x7f4aa0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4aaa7fc640 1 -- 172.21.15.71:0/1154523509 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0003520 con 0x7f4aa00047c0 2023-08-25T10:05:44.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.749+0000 7f4ab2ffd640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4aa0003a10 0x7f4aa0003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:44.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.750+0000 7f4aaa7fc640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4aa00047c0 0x7f4aa0004c40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00a830 tx=0x7f4aa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5478a323a65bb001 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:44.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0072d0 con 0x7f4aa00047c0 2023-08-25T10:05:44.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c02f070 con 0x7f4aa00047c0 2023-08-25T10:05:44.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab27fc640 1 -- 172.21.15.71:0/1154523509 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0076b0 con 0x7f4aa00047c0 2023-08-25T10:05:44.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1154523509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4aa00047c0 msgr2=0x7f4aa0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1154523509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4aa00047c0 0x7f4aa0004c40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00a830 tx=0x7f4aa00057f0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab37fe640 1 -- 172.21.15.71:0/1154523509 reap_dead start 2023-08-25T10:05:44.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1154523509 shutdown_connections 2023-08-25T10:05:44.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1154523509 >> 172.21.15.71:0/1154523509 conn(0x7f4ab40654f0 msgr2=0x7f4ab4065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:44.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.751+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1154523509 shutdown_connections 2023-08-25T10:05:44.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.752+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1154523509 wait complete. 2023-08-25T10:05:44.761 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.753+0000 7f4ab92bf640 1 Processor -- start 2023-08-25T10:05:44.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.753+0000 7f4ab92bf640 1 -- start start 2023-08-25T10:05:44.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.754+0000 7f4ab92bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa00047c0 0x7f4ab41425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.754+0000 7f4ab92bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ab406edc0 0x7f4ab40c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.754+0000 7f4ab92bf640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 0x7f4ab406fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.762 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.754+0000 7f4ab92bf640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40c8950 con 0x7f4ab40c9670 2023-08-25T10:05:44.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.754+0000 7f4ab92bf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4142b20 con 0x7f4ab406edc0 2023-08-25T10:05:44.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.754+0000 7f4ab92bf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40c80f0 con 0x7f4aa00047c0 2023-08-25T10:05:44.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 0x7f4ab406fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 0x7f4ab406fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60086/0 (socket says 172.21.15.71:60086) 2023-08-25T10:05:44.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 -- 172.21.15.71:0/1055186712 learned_addr learned my addr 172.21.15.71:0/1055186712 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:44.763 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab2ffd640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa00047c0 0x7f4ab41425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4aaa7fc640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ab406edc0 0x7f4ab40c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 -- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa00047c0 msgr2=0x7f4ab41425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa00047c0 0x7f4ab41425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.764 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 -- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ab406edc0 msgr2=0x7f4ab40c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ab406edc0 0x7f4ab40c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 -- 172.21.15.71:0/1055186712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab406f530 con 0x7f4ab40c9670 2023-08-25T10:05:44.765 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4aaa7fc640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ab406edc0 0x7f4ab40c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-08-25T10:05:44.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.755+0000 7f4ab37fe640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 0x7f4ab406fa80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4ab406db60 tx=0x7f4aac00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:44.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.756+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac00b070 con 0x7f4ab40c9670 2023-08-25T10:05:44.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.756+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac0054e0 con 0x7f4ab40c9670 2023-08-25T10:05:44.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.756+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac003810 con 0x7f4ab40c9670 2023-08-25T10:05:44.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.756+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab40c9b20 con 0x7f4ab40c9670 2023-08-25T10:05:44.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.756+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab4143c40 con 0x7f4ab40c9670 2023-08-25T10:05:44.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.757+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab41e2670 con 0x7f4ab40c9670 2023-08-25T10:05:44.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.758+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f4aac003970 con 0x7f4ab40c9670 2023-08-25T10:05:44.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.759+0000 7f4aabfff640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a9406a080 0x7f4a9406c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:44.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.759+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f4aac087f50 con 0x7f4ab40c9670 2023-08-25T10:05:44.768 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.759+0000 7f4ab2ffd640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a9406a080 0x7f4a9406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:44.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.760+0000 7f4ab2ffd640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a9406a080 0x7f4a9406c540 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40099c0 tx=0x7f4aa4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:44.769 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.763+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f4aac0530d0 con 0x7f4ab40c9670 2023-08-25T10:05:44.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.952+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4ab406f530 con 0x7f4ab40c9670 2023-08-25T10:05:44.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.952+0000 7f4aabfff640 1 -- 172.21.15.71:0/1055186712 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v69) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4ab406f530 con 0x7f4ab40c9670 2023-08-25T10:05:44.956 INFO:teuthology.orchestra.run.smithi071.stdout:pg_num: 32 2023-08-25T10:05:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a9406a080 msgr2=0x7f4a9406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a9406a080 0x7f4a9406c540 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40099c0 tx=0x7f4aa4008040 comp rx=0 tx=0).stop 2023-08-25T10:05:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 msgr2=0x7f4ab406fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 0x7f4ab406fa80 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4ab406db60 tx=0x7f4aac00d890 comp rx=0 tx=0).stop 2023-08-25T10:05:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 shutdown_connections 2023-08-25T10:05:44.958 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ab406edc0 0x7f4ab40c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f4a9406a080 0x7f4a9406c540 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab40c9670 0x7f4ab406fa80 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 --2- 172.21.15.71:0/1055186712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4aa00047c0 0x7f4ab41425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:44.959 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.955+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 >> 172.21.15.71:0/1055186712 conn(0x7f4ab40654f0 msgr2=0x7f4ab41e1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.956+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 shutdown_connections 2023-08-25T10:05:44.960 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:44.956+0000 7f4ab92bf640 1 -- 172.21.15.71:0/1055186712 wait complete. 2023-08-25T10:05:44.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-08-25T10:05:45.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.192+0000 7fe0f359e640 1 Processor -- start 2023-08-25T10:05:45.194 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.192+0000 7fe0f359e640 1 -- start start 2023-08-25T10:05:45.195 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0f359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe0f4159890 0x7fe0f406a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0f359e640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0f406b490 0x7fe0f40c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0f359e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f41580c0 con 0x7fe0f40c6300 2023-08-25T10:05:45.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0f359e640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f40c4d20 con 0x7fe0f406b490 2023-08-25T10:05:45.196 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0f359e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f40c5000 con 0x7fe0f4159890 2023-08-25T10:05:45.199 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0ebfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0f406b490 0x7fe0f40c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0ebfff640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7fe0f406b490 0x7fe0f40c5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60088/0 (socket says 172.21.15.71:60088) 2023-08-25T10:05:45.200 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0ebfff640 1 -- 172.21.15.71:0/2588266354 learned_addr learned my addr 172.21.15.71:0/2588266354 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:45.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.194+0000 7fe0f259c640 1 --2- 172.21.15.71:0/2588266354 >> v2:172.21.15.164:3300/0 conn(0x7fe0f4159890 0x7fe0f406a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.195+0000 7fe0ebfff640 1 -- 172.21.15.71:0/2588266354 >> v2:172.21.15.164:3300/0 conn(0x7fe0f4159890 msgr2=0x7fe0f406a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:45.201 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.195+0000 7fe0ebfff640 1 --2- 172.21.15.71:0/2588266354 >> v2:172.21.15.164:3300/0 conn(0x7fe0f4159890 0x7fe0f406a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.195+0000 7fe0ebfff640 1 -- 172.21.15.71:0/2588266354 >> v1:172.21.15.164:6789/0 conn(0x7fe0f40c6300 legacy=0x7fe0f4156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:45.202 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.195+0000 7fe0ebfff640 1 -- 172.21.15.71:0/2588266354 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40c52e0 con 0x7fe0f406b490 2023-08-25T10:05:45.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.196+0000 7fe0ebfff640 1 --2- 172.21.15.71:0/2588266354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c5dc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe0f406dc20 tx=0x7fe0dc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f9900b26db0155 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:45.203 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.196+0000 7fe0f1d9b640 1 -- 172.21.15.71:0/2588266354 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc030860 con 0x7fe0f406b490 2023-08-25T10:05:45.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.196+0000 7fe0f1d9b640 1 -- 172.21.15.71:0/2588266354 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0309c0 con 0x7fe0f406b490 2023-08-25T10:05:45.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.196+0000 7fe0f1d9b640 1 -- 172.21.15.71:0/2588266354 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc030d50 con 0x7fe0f406b490 2023-08-25T10:05:45.204 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 -- 172.21.15.71:0/2588266354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 msgr2=0x7fe0f40c5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 --2- 172.21.15.71:0/2588266354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c5dc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe0f406dc20 tx=0x7fe0dc02b9a0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.205 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 -- 172.21.15.71:0/2588266354 shutdown_connections 2023-08-25T10:05:45.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 --2- 172.21.15.71:0/2588266354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c5dc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 --2- 172.21.15.71:0/2588266354 >> v2:172.21.15.164:3300/0 conn(0x7fe0f4159890 0x7fe0f406a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.206 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 -- 172.21.15.71:0/2588266354 >> 172.21.15.71:0/2588266354 conn(0x7fe0f40c6ca0 msgr2=0x7fe0f40c70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:45.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 -- 172.21.15.71:0/2588266354 shutdown_connections 2023-08-25T10:05:45.207 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.197+0000 7fe0f359e640 1 -- 172.21.15.71:0/2588266354 wait complete. 2023-08-25T10:05:45.210 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.199+0000 7fe0f359e640 1 Processor -- start 2023-08-25T10:05:45.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.199+0000 7fe0f359e640 1 -- start start 2023-08-25T10:05:45.211 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f359e640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0f40c6300 0x7fe0f40c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.212 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f359e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0f4159890 0x7fe0f40c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f359e640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f41ef430 con 0x7fe0f406b490 2023-08-25T10:05:45.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f359e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f4157de0 con 0x7fe0f40c6300 2023-08-25T10:05:45.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f359e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40c1920 con 0x7fe0f4159890 2023-08-25T10:05:45.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.214 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60100/0 (socket says 172.21.15.71:60100) 2023-08-25T10:05:45.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 -- 172.21.15.71:0/14289562 learned_addr learned my addr 172.21.15.71:0/14289562 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:45.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 -- 172.21.15.71:0/14289562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0f4159890 msgr2=0x7fe0f40c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.215 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0f4159890 0x7fe0f40c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 -- 172.21.15.71:0/14289562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0f40c6300 msgr2=0x7fe0f40c1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.216 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0f40c6300 0x7fe0f40c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.200+0000 7fe0f259c640 1 -- 172.21.15.71:0/14289562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc0094d0 con 0x7fe0f406b490 2023-08-25T10:05:45.217 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.201+0000 7fe0f259c640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c0530 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400d620 tx=0x7fe0e400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:45.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.201+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400b620 con 0x7fe0f406b490 2023-08-25T10:05:45.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.201+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e400b780 con 0x7fe0f406b490 2023-08-25T10:05:45.218 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.201+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f40c3590 con 0x7fe0f406b490 2023-08-25T10:05:45.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.201+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400bb30 con 0x7fe0f406b490 2023-08-25T10:05:45.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.201+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f40703f0 con 0x7fe0f406b490 2023-08-25T10:05:45.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.202+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f404e420 con 0x7fe0f406b490 2023-08-25T10:05:45.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.203+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7fe0e400bc90 con 0x7fe0f406b490 2023-08-25T10:05:45.219 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.205+0000 7fe0eaffd640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0d406a050 0x7fe0d406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.205+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7fe0e4088ec0 con 0x7fe0f406b490 2023-08-25T10:05:45.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.205+0000 7fe0ebfff640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0d406a050 0x7fe0d406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.220 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.206+0000 7fe0ebfff640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0d406a050 0x7fe0d406c510 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc036000 tx=0x7fe0dc03b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:45.221 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.209+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7fe0e40540f0 con 0x7fe0f406b490 2023-08-25T10:05:45.413 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.411+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe0f40c2c60 con 0x7fe0f406b490 2023-08-25T10:05:45.414 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.412+0000 7fe0eaffd640 1 -- 172.21.15.71:0/14289562 <== mon.0 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v69) v1 ==== 80+0+492 (secure 0 0 0) 0x7fe0e4057710 con 0x7fe0f406b490 2023-08-25T10:05:45.416 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6837/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:45.416 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6834/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:45.416 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6832/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:45.416 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6836/1556118065 2023-08-26T09:56:20.645474+0000 2023-08-25T10:05:45.417 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6833/2402500458 2023-08-26T09:55:05.613929+0000 2023-08-25T10:05:45.417 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6835/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:45.417 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71:6835/3404832976 2023-08-26T09:57:50.703151+0000 2023-08-25T10:05:45.417 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.164:6834/2707317155 2023-08-26T09:52:30.557077+0000 2023-08-25T10:05:45.418 INFO:teuthology.orchestra.run.smithi071.stderr:listed 8 entries 2023-08-25T10:05:45.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0d406a050 msgr2=0x7fe0d406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.418 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0d406a050 0x7fe0d406c510 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc036000 tx=0x7fe0dc03b040 comp rx=0 tx=0).stop 2023-08-25T10:05:45.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 msgr2=0x7fe0f40c0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c0530 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400d620 tx=0x7fe0e400dae0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 shutdown_connections 2023-08-25T10:05:45.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe0f40c6300 0x7fe0f40c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.419 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7fe0d406a050 0x7fe0d406c510 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe0f4159890 0x7fe0f40c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 --2- 172.21.15.71:0/14289562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe0f406b490 0x7fe0f40c0530 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.420 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 >> 172.21.15.71:0/14289562 conn(0x7fe0f40c6ca0 msgr2=0x7fe0f40c8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:45.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 shutdown_connections 2023-08-25T10:05:45.421 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.415+0000 7fe0f359e640 1 -- 172.21.15.71:0/14289562 wait complete. 2023-08-25T10:05:45.434 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_fuse.py", line 175, in task yield all_mounts File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/cephfs/fuse_mount.py", line 396, in umount_wait self.umount(cleanup=False) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/cephfs/fuse_mount.py", line 317, in umount if self.is_blocked(): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/cephfs/mount.py", line 197, in is_blocked return self.addr in output TypeError: 'in ' requires string as left operand, not NoneType 2023-08-25T10:05:45.441 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-08-25T10:05:45.458 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-08-25T10:05:45.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-08-25T10:05:45.679 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.677+0000 7f1dfd38b640 1 Processor -- start 2023-08-25T10:05:45.680 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.678+0000 7f1dfd38b640 1 -- start start 2023-08-25T10:05:45.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.679+0000 7f1dfd38b640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1df80c3660 0x7f1df80c1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.679+0000 7f1dfd38b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.681 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.679+0000 7f1dfd38b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df80c1dc0 con 0x7f1df81509d0 2023-08-25T10:05:45.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.679+0000 7f1dfd38b640 1 -- --> v2:172.21.15.71:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df8157a80 con 0x7f1df80c3660 2023-08-25T10:05:45.682 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.679+0000 7f1dfd38b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df806cf70 con 0x7f1df80c3a40 2023-08-25T10:05:45.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.679+0000 7f1df6ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1df80c3660 0x7f1df80c1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1df6ffd640 1 --2- >> v2:172.21.15.71:3300/0 conn(0x7f1df80c3660 0x7f1df80c1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60110/0 (socket says 172.21.15.71:60110) 2023-08-25T10:05:45.683 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1df6ffd640 1 -- 172.21.15.71:0/839400487 learned_addr learned my addr 172.21.15.71:0/839400487 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:45.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1deffff640 1 --2- 172.21.15.71:0/839400487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1df6ffd640 1 -- 172.21.15.71:0/839400487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1df806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:45.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/839400487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df806ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1df6ffd640 1 -- 172.21.15.71:0/839400487 >> v1:172.21.15.164:6789/0 conn(0x7f1df81509d0 legacy=0x7f1df8150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-08-25T10:05:45.684 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.680+0000 7f1df6ffd640 1 -- 172.21.15.71:0/839400487 --> v2:172.21.15.71:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8156e00 con 0x7f1df80c3660 2023-08-25T10:05:45.685 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.681+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/839400487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df80c1710 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009870 tx=0x7f1de802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54bf93da8833b05 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:45.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.682+0000 7f1df67fc640 1 -- 172.21.15.71:0/839400487 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8030840 con 0x7f1df80c3660 2023-08-25T10:05:45.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.682+0000 7f1df67fc640 1 -- 172.21.15.71:0/839400487 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de80309a0 con 0x7f1df80c3660 2023-08-25T10:05:45.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.682+0000 7f1df67fc640 1 -- 172.21.15.71:0/839400487 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8030d30 con 0x7f1df80c3660 2023-08-25T10:05:45.686 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.682+0000 7f1dfd38b640 1 -- 172.21.15.71:0/839400487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 msgr2=0x7f1df80c1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.682+0000 7f1dfd38b640 1 --2- 172.21.15.71:0/839400487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df80c1710 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1de8009870 tx=0x7f1de802b980 comp rx=0 tx=0).stop 2023-08-25T10:05:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.683+0000 7f1dfd38b640 1 -- 172.21.15.71:0/839400487 shutdown_connections 2023-08-25T10:05:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.683+0000 7f1dfd38b640 1 --2- 172.21.15.71:0/839400487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df806ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.683+0000 7f1dfd38b640 1 --2- 172.21.15.71:0/839400487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df80c1710 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.683+0000 7f1dfd38b640 1 -- 172.21.15.71:0/839400487 >> 172.21.15.71:0/839400487 conn(0x7f1df80654f0 msgr2=0x7f1df8065900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-08-25T10:05:45.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.683+0000 7f1dfd38b640 1 -- 172.21.15.71:0/839400487 shutdown_connections 2023-08-25T10:05:45.688 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.683+0000 7f1dfd38b640 1 -- 172.21.15.71:0/839400487 wait complete. 2023-08-25T10:05:45.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.685+0000 7f1dfd38b640 1 Processor -- start 2023-08-25T10:05:45.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.685+0000 7f1dfd38b640 1 -- start start 2023-08-25T10:05:45.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1dfd38b640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1dfd38b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df80c0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1dfd38b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1df80c0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1dfd38b640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df80c1080 con 0x7f1df80c3660 2023-08-25T10:05:45.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1dfd38b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df81462d0 con 0x7f1df81509d0 2023-08-25T10:05:45.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1dfd38b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df815a450 con 0x7f1df80c3a40 2023-08-25T10:05:45.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:60126/0 (socket says 172.21.15.71:60126) 2023-08-25T10:05:45.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 learned_addr learned my addr 172.21.15.71:0/3596385450 (peer_addr_for_me v2:172.21.15.71:0/0) 2023-08-25T10:05:45.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df80c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1df80c0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1df80c0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1df80c0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 msgr2=0x7f1df80c0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-08-25T10:05:45.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1df80c0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.686+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de80094d0 con 0x7f1df80c3660 2023-08-25T10:05:45.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.687+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1de80090b0 tx=0x7f1de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:45.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.687+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 <== mon.0 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8039b50 con 0x7f1df80c3660 2023-08-25T10:05:45.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.687+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 <== mon.0 v2:172.21.15.71:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de8039cf0 con 0x7f1df80c3660 2023-08-25T10:05:45.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.687+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 <== mon.0 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8043650 con 0x7f1df80c3660 2023-08-25T10:05:45.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.687+0000 7f1dfd38b640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df815a730 con 0x7f1df80c3660 2023-08-25T10:05:45.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.687+0000 7f1dfd38b640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df815acf0 con 0x7f1df80c3660 2023-08-25T10:05:45.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.688+0000 7f1dfd38b640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df80c29c0 con 0x7f1df80c3660 2023-08-25T10:05:45.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.689+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 <== mon.0 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 5) v1 ==== 88341+0+0 (secure 0 0 0) 0x7f1de80437b0 con 0x7f1df80c3660 2023-08-25T10:05:45.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.690+0000 7f1def7fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1dd806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.690+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 <== mon.0 v2:172.21.15.71:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7271+0+0 (secure 0 0 0) 0x7f1de80aacb0 con 0x7f1df80c3660 2023-08-25T10:05:45.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.690+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1dd806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-08-25T10:05:45.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.691+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1dd806c510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-08-25T10:05:45.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.695+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 <== mon.0 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178558 (secure 0 0 0) 0x7f1de8075e40 con 0x7f1df80c3660 2023-08-25T10:05:45.765 INFO:tasks.ceph.osd.0:Sent signal 15 2023-08-25T10:05:45.766 INFO:tasks.ceph.osd.1:Sent signal 15 2023-08-25T10:05:45.766 INFO:tasks.ceph.osd.2:Sent signal 15 2023-08-25T10:05:45.767 INFO:tasks.ceph.osd.3:Sent signal 15 2023-08-25T10:05:45.768 INFO:tasks.ceph.osd.4:Sent signal 15 2023-08-25T10:05:45.768 INFO:tasks.ceph.osd.5:Sent signal 15 2023-08-25T10:05:45.769 INFO:tasks.ceph.osd.6:Sent signal 15 2023-08-25T10:05:45.770 INFO:tasks.ceph.osd.7:Sent signal 15 2023-08-25T10:05:45.770 INFO:tasks.ceph.mds.a:Sent signal 15 2023-08-25T10:05:45.770 INFO:tasks.ceph.mds.c:Sent signal 15 2023-08-25T10:05:45.771 INFO:tasks.ceph.mds.e:Sent signal 15 2023-08-25T10:05:45.772 INFO:tasks.ceph.mds.b:Sent signal 15 2023-08-25T10:05:45.772 INFO:tasks.ceph.mds.d:Sent signal 15 2023-08-25T10:05:45.773 INFO:tasks.ceph.mds.f:Sent signal 15 2023-08-25T10:05:45.773 INFO:tasks.ceph.mon.a:Sent signal 15 2023-08-25T10:05:45.774 INFO:tasks.ceph.mon.b:Sent signal 15 2023-08-25T10:05:45.775 INFO:tasks.ceph.mon.c:Sent signal 15 2023-08-25T10:05:45.775 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-08-25T10:05:45.776 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-08-25T10:05:45.777 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T10:05:45.764+0000 7f69f6f0e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 89432) UID: 0 2023-08-25T10:05:45.778 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T10:05:45.764+0000 7f69f6f0e640 -1 osd.0 69 *** Got signal Terminated *** 2023-08-25T10:05:45.778 INFO:tasks.ceph.osd.0.smithi071.stderr:2023-08-25T10:05:45.764+0000 7f69f6f0e640 -1 osd.0 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.778 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T10:05:45.767+0000 7fb28f8e4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 89607) UID: 0 2023-08-25T10:05:45.778 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T10:05:45.767+0000 7fb28f8e4640 -1 osd.4 69 *** Got signal Terminated *** 2023-08-25T10:05:45.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2023-08-25T10:05:45.767+0000 7fb28f8e4640 -1 osd.4 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T10:05:45.768+0000 7f89af097640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 89610) UID: 0 2023-08-25T10:05:45.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T10:05:45.768+0000 7f89af097640 -1 osd.5 69 *** Got signal Terminated *** 2023-08-25T10:05:45.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2023-08-25T10:05:45.768+0000 7f89af097640 -1 osd.5 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.781 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T10:05:45.765+0000 7f62f117a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 89428) UID: 0 2023-08-25T10:05:45.781 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T10:05:45.765+0000 7f62f117a640 -1 osd.1 69 *** Got signal Terminated *** 2023-08-25T10:05:45.781 INFO:tasks.ceph.osd.1.smithi071.stderr:2023-08-25T10:05:45.765+0000 7f62f117a640 -1 osd.1 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T10:05:45.768+0000 7fbb74be9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 89612) UID: 0 2023-08-25T10:05:45.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T10:05:45.768+0000 7fbb74be9640 -1 osd.6 69 *** Got signal Terminated *** 2023-08-25T10:05:45.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2023-08-25T10:05:45.768+0000 7fbb74be9640 -1 osd.6 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.783 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T10:05:45.766+0000 7f9f06267640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 89433) UID: 0 2023-08-25T10:05:45.783 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T10:05:45.766+0000 7f9f06267640 -1 osd.2 69 *** Got signal Terminated *** 2023-08-25T10:05:45.783 INFO:tasks.ceph.osd.2.smithi071.stderr:2023-08-25T10:05:45.766+0000 7f9f06267640 -1 osd.2 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T10:05:45.769+0000 7f27f13fa640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 89613) UID: 0 2023-08-25T10:05:45.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T10:05:45.769+0000 7f27f13fa640 -1 osd.7 69 *** Got signal Terminated *** 2023-08-25T10:05:45.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2023-08-25T10:05:45.769+0000 7f27f13fa640 -1 osd.7 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.785 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T10:05:45.767+0000 7f275d319640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 89434) UID: 0 2023-08-25T10:05:45.785 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T10:05:45.767+0000 7f275d319640 -1 osd.3 69 *** Got signal Terminated *** 2023-08-25T10:05:45.785 INFO:tasks.ceph.osd.3.smithi071.stderr:2023-08-25T10:05:45.767+0000 7f275d319640 -1 osd.3 69 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-25T10:05:45.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2023-08-25T10:05:45.771+0000 7fe994dbf640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i b (PID: 93962) UID: 0 2023-08-25T10:05:45.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2023-08-25T10:05:45.771+0000 7fe994dbf640 -1 mds.b *** got signal Terminated *** 2023-08-25T10:05:45.787 INFO:tasks.ceph.mds.a.smithi071.stderr:2023-08-25T10:05:45.769+0000 7f2b45d90640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i a (PID: 104687) UID: 0 2023-08-25T10:05:45.788 INFO:tasks.ceph.mds.a.smithi071.stderr:2023-08-25T10:05:45.769+0000 7f2b45d90640 -1 mds.a *** got signal Terminated *** 2023-08-25T10:05:45.789 INFO:tasks.ceph.mds.d.smithi164.stderr:2023-08-25T10:05:45.771+0000 7f93c64b2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 93359) UID: 0 2023-08-25T10:05:45.789 INFO:tasks.ceph.mds.d.smithi164.stderr:2023-08-25T10:05:45.771+0000 7f93c64b2640 -1 mds.d *** got signal Terminated *** 2023-08-25T10:05:45.790 INFO:tasks.ceph.mds.c.smithi071.stderr:2023-08-25T10:05:45.770+0000 7fad17ac7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 105812) UID: 0 2023-08-25T10:05:45.790 INFO:tasks.ceph.mds.c.smithi071.stderr:2023-08-25T10:05:45.770+0000 7fad17ac7640 -1 mds.c *** got signal Terminated *** 2023-08-25T10:05:45.791 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T10:05:45.773+0000 7f5fc5b20640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i f (PID: 93121) UID: 0 2023-08-25T10:05:45.792 INFO:tasks.ceph.mds.f.smithi164.stderr:2023-08-25T10:05:45.773+0000 7f5fc5b20640 -1 mds.f *** got signal Terminated *** 2023-08-25T10:05:45.792 INFO:tasks.ceph.mds.e.smithi071.stderr:2023-08-25T10:05:45.771+0000 7f09771d4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i e (PID: 93302) UID: 0 2023-08-25T10:05:45.793 INFO:tasks.ceph.mds.e.smithi071.stderr:2023-08-25T10:05:45.771+0000 7f09771d4640 -1 mds.e *** got signal Terminated *** 2023-08-25T10:05:45.794 INFO:tasks.ceph.mon.b.smithi164.stderr:2023-08-25T10:05:45.774+0000 7f5db3ff0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 89028) UID: 0 2023-08-25T10:05:45.794 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-08-25T10:05:45.773+0000 7f1bca605640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 89179) UID: 0 2023-08-25T10:05:45.795 INFO:tasks.ceph.mon.a.smithi071.stderr:2023-08-25T10:05:45.773+0000 7f1bca605640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-08-25T10:05:45.795 INFO:tasks.ceph.mon.c.smithi164.stderr:2023-08-25T10:05:45.774+0000 7f1336a1c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 89029) UID: 0 2023-08-25T10:05:45.796 INFO:tasks.ceph.mon.c.smithi164.stderr:2023-08-25T10:05:45.774+0000 7f1336a1c640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-08-25T10:05:45.797 INFO:tasks.ceph.mon.b.smithi164.stderr:2023-08-25T10:05:45.774+0000 7f5db3ff0640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-08-25T10:05:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.776+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 msgr2=0x7f1df8145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-08-25T10:05:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.776+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 msgr2=0x7f1df8145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-25T10:05:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.776+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1de80090b0 tx=0x7f1de8004730 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T10:05:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.776+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1de80090b0 tx=0x7f1de8004730 comp rx=0 tx=0).stop 2023-08-25T10:05:45.799 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.776+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-08-25T10:05:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.776+0000 7f1def7fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1df80c3660 0x7f1df8145d90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1def7fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1dd8075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.800 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1def7fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd8078720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1def7fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 0x7f1dd807b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:45.801 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd8079130 2023-08-25T10:05:45.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1df81509d0 2023-08-25T10:05:45.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1df80c3a40 2023-08-25T10:05:45.802 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.777+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 msgr2=0x7f1dd807b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:45.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.778+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 0x7f1dd807b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:45.803 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.778+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 msgr2=0x7f1dd8075f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:45.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.778+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1dd8075f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:45.804 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.778+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd8078720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:45.805 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.778+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd8078720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:45.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.841+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1dd806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2023-08-25T10:05:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.841+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1dd806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-08-25T10:05:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.841+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1dd806c510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-08-25T10:05:45.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.841+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1dd806c510 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0008040 comp rx=0 tx=0).stop 2023-08-25T10:05:45.845 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.841+0000 7f1def7fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1dd806c510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-08-25T10:05:45.846 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.841+0000 7f1def7fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1dd806c510 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:45.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:daemon-helper: command crashed with signal 15 2023-08-25T10:05:45.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.979+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd8078720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:45.981 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.979+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 msgr2=0x7f1dd807b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:45.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.979+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd8078720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:45.982 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.979+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 0x7f1dd807b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:45.983 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.980+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 msgr2=0x7f1dd8075f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:45.984 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:45.980+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1dd8075f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:46.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.380+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd8078720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:46.382 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.380+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 msgr2=0x7f1dd807b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:46.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.380+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd8078720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:46.383 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.380+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 0x7f1dd807b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:46.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.381+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 msgr2=0x7f1dd8075f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:46.384 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.381+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1dd8075f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:46.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.690+0000 7f1dedffb640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:46.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.690+0000 7f1dedffb640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1df81573c0 con 0x7f1dd806a050 2023-08-25T10:05:46.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.691+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:46.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.691+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:46.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.892+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:46.894 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:46.892+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:47.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.181+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 msgr2=0x7f1dd807b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:47.185 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.181+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 0x7f1dd807b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:47.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.181+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd8078720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:47.187 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.181+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd8078720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:47.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.182+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 msgr2=0x7f1dd8075f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:47.188 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.182+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1dd8075f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:47.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.293+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:47.295 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:47.293+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:48.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.094+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:48.096 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.094+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:48.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd8078720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:05:48.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd8078720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:48.689 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 msgr2=0x7f1dd8075f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:05:48.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df81509d0 0x7f1dd8075f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:48.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 msgr2=0x7f1dd807b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:05:48.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd8079130 0x7f1dd807b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:48.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:48.690 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:48.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:48.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.686+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd4006040 2023-08-25T10:05:48.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4003610 2023-08-25T10:05:48.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1df80c3a40 2023-08-25T10:05:48.691 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:48.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:48.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:48.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:48.692 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:48.693 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.687+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:48.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.888+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:48.890 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.888+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:48.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.888+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:48.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.888+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:48.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.888+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:48.891 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:48.888+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:49.291 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.289+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:49.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.289+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:49.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.289+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:49.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.289+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:49.292 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.289+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:49.293 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.289+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:49.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.695+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:49.698 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:49.695+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:50.092 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:50.090+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:50.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:50.090+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:50.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:50.090+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:50.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:50.090+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:50.093 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:50.090+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:50.094 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:50.091+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:51.694 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:51.692+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:51.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:51.692+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:51.695 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:51.692+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:51.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:51.692+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:51.696 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:51.692+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:51.697 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:51.692+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:52.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:52.898+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:52.901 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:52.899+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:05:54.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:54.896+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:54.898 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:54.896+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:05:54.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:54.896+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:54.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:54.896+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:05:54.899 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:54.896+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:54.900 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:54.896+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:05:55.439 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd400acc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:05:55.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd400acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:55.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd4005a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:05:55.440 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd4005a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:55.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:05:55.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd4006040 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:05:55.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:55.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:55.441 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:05:55.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd400b7b0 2023-08-25T10:05:55.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4003610 2023-08-25T10:05:55.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1df80c3a40 2023-08-25T10:05:55.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:55.442 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:55.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:55.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.437+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:55.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.438+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:55.443 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.438+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:05:55.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.638+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:55.640 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.638+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:55.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.638+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:55.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.638+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:55.641 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.639+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:55.642 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:55.639+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:05:56.041 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.039+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:56.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.039+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:56.042 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.039+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:56.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.039+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:56.043 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.040+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:56.044 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.040+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:05:56.842 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.840+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:56.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.840+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:56.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.840+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:56.843 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.840+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:56.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.841+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:56.844 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:56.841+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:05:57.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:05:57.382 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:05:58.444 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:58.442+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:05:58.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:58.442+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:58.445 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:58.442+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:05:58.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:58.442+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:05:58.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:58.442+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:58.446 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:58.443+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:05:59.307 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:59.305+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:05:59.308 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:05:59.305+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:01.648 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:01.646+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:01.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:01.646+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:01.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:01.646+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:01.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:01.646+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:01.649 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:01.646+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:01.650 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:01.647+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:05.565 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40084d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:05.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40084d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:05.566 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:05.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003610 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:05.567 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:05.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:05.568 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:05.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:05.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:05.569 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd400b7b0 2023-08-25T10:06:05.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4003f30 2023-08-25T10:06:05.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.562+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1df80c3a40 2023-08-25T10:06:05.570 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.563+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:05.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.563+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:05.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.563+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:05.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.563+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:05.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.563+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:05.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.563+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:05.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.764+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:05.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.764+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:05.766 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.764+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:05.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.764+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:05.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.764+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:05.767 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:05.765+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.165+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.165+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.165+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:06.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.165+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:06.168 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.166+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:06.169 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.166+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:06.968 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.966+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:06.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.966+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:06.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.966+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:06.969 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.966+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:06.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.967+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:06.970 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:06.967+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:08.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:08.568+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:08.571 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:08.568+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:08.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:08.568+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:08.572 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:08.568+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:08.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:08.568+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:08.573 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:08.569+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:11.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:11.771+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:11.773 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:11.771+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:11.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:11.772+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:11.774 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:11.772+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:11.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:11.772+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:11.775 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:11.772+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:12.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:12.118+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:06:12.120 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:12.118+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:06:18.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:18.177+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:18.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:18.177+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:18.180 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:18.177+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:18.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:18.177+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:18.181 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:18.178+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:18.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:18.178+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:20.752 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:20.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:20.753 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:20.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:20.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd400dba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:20.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd400dba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:20.754 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd400b7b0 2023-08-25T10:06:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4004bf0 2023-08-25T10:06:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.749+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1df80c3a40 2023-08-25T10:06:20.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.750+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:20.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.750+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:20.756 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.750+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:20.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.750+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:20.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.750+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:20.757 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.750+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:20.954 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.951+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.951+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.951+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.951+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.951+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:20.955 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:20.951+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:21.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:21.352+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:21.355 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:21.352+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:21.352+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:21.352+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:21.356 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:21.353+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:21.357 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:21.353+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:22.153+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:22.156 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:22.153+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:22.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:22.153+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:22.157 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:22.153+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:22.154+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:22.154+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:23.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:23.755+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:23.758 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:23.755+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:23.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:23.755+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:23.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:23.755+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:23.759 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:23.756+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:23.760 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:23.756+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:26.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:26.959+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:26.961 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:26.959+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:26.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:26.959+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:26.962 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:26.959+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:26.959+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:26.963 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:26.959+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:27.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:27.133+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:06:27.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:27.133+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:06:27.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:06:27.535 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:06:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:33.367+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:33.367+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:33.367+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:33.367+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:33.367+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:33.370 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:33.367+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:42.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:42.149+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:06:42.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:42.149+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:06:43.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:43.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df80c3a40 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:43.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd4008490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:43.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd4008490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:06:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:06:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:06:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd400b7b0 2023-08-25T10:06:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4004bf0 2023-08-25T10:06:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1dd4005200 2023-08-25T10:06:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.532+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.533+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.533+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.533+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.533+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:06:43.735 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.733+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:43.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.733+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:43.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.734+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:43.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.734+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:43.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.734+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:43.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:43.734+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:06:44.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.134+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:44.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.134+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:44.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.135+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:44.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.135+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:44.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.136+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:44.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.136+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:06:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.935+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.936+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.936+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.936+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.937+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:44.941 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:44.937+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:06:46.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:46.538+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:46.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:46.538+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:46.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:46.538+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:46.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:46.538+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:46.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:46.539+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:46.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:46.539+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:06:49.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:49.741+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:49.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:49.741+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:49.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:49.741+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:49.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:49.742+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:49.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:49.742+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:49.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:49.742+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:06:56.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:56.148+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:06:56.150 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:56.148+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:56.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:56.148+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:06:56.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:56.148+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:06:56.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:56.148+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:56.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:56.148+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:06:57.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:57.165+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:06:57.167 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:06:57.165+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:06:57.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:06:57.695 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:08.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:08.962+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:08.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:08.962+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:08.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:08.962+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:08.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:08.962+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:08.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:08.962+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:08.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:08.962+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:12.182 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:12.180+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:07:12.183 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:12.180+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:13.534 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:07:13.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:07:13.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:07:13.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:07:13.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd4004310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:07:13.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd4004310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:07:13.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:07:13.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:07:13.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:07:13.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd400b7b0 2023-08-25T10:07:13.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4004bf0 2023-08-25T10:07:13.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1dd4005200 2023-08-25T10:07:13.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:13.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:07:13.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:13.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:07:13.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:13.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.532+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:07:13.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.733+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:13.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.733+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:07:13.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.733+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:13.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.733+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:07:13.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.733+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:13.739 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:13.733+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:07:14.136 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.134+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:14.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.134+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:07:14.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.134+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:14.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.134+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:07:14.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.135+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:14.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.135+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:07:14.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.935+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:14.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.935+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:07:14.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.935+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:14.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.935+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:07:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.936+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:14.936+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:07:16.540 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:16.537+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:16.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:16.537+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:16.541 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:16.537+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:07:16.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:16.537+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:07:16.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:16.537+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:16.542 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:16.538+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-08-25T10:07:19.743 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:19.741+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:19.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:19.741+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:07:19.744 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:19.741+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:19.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:19.741+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:19.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:19.741+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:07:19.745 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:19.741+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-08-25T10:07:26.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:26.148+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:26.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:26.148+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:26.151 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:26.148+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:07:26.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:26.148+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:07:26.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:26.148+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:26.152 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:26.148+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-08-25T10:07:27.197 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:27.195+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:07:27.198 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:27.195+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:27.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:27.860 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:38.964 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:38.962+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:38.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:38.962+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:38.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:38.962+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:38.965 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:38.962+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:38.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:38.963+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:38.966 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:38.963+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:42.211+0000 7f1deffff640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 msgr2=0x7f1df8145d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:6832/786313351 2023-08-25T10:07:42.213 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:42.211+0000 7f1deffff640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:6832/786313351,v1:172.21.15.71:6833/786313351] conn(0x7f1dd806a050 0x7f1df8145d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-08-25T10:07:43.535 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd4008470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:07:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd4008470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:07:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:07:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004bf0 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:07:43.536 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-08-25T10:07:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-08-25T10:07:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 0x7f1dd400fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:07:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:07:43.537 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-08-25T10:07:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c070 con 0x7f1dd400b7b0 2023-08-25T10:07:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c1b0 con 0x7f1dd4004310 2023-08-25T10:07:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1dee7fc640 1 -- 172.21.15.71:0/3596385450 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd807c340 con 0x7f1dd4005200 2023-08-25T10:07:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:43.538 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:07:43.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:43.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:07:43.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.533+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 msgr2=0x7f1dd400fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:43.539 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.534+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 0x7f1dd400fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-08-25T10:07:43.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.734+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:43.736 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.734+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:07:43.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.734+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:43.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.734+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:07:43.737 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.735+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 msgr2=0x7f1dd400fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:43.738 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:43.735+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 0x7f1dd400fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-08-25T10:07:44.137 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.135+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:44.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.135+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:07:44.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.135+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:44.138 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.135+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:07:44.139 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.136+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 msgr2=0x7f1dd400fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:44.140 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.136+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 0x7f1dd400fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-08-25T10:07:44.938 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.936+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 msgr2=0x7f1dd40055e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.71:3300/0 2023-08-25T10:07:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.936+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1dd400b7b0 0x7f1dd40055e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:07:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.936+0000 7f1df77fe640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 msgr2=0x7f1dd40039f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2023-08-25T10:07:44.939 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.936+0000 7f1df77fe640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1dd4005200 0x7f1dd40039f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:07:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.937+0000 7f1df6ffd640 1 -- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 msgr2=0x7f1dd400fbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2023-08-25T10:07:44.940 INFO:teuthology.orchestra.run.smithi071.stderr:2023-08-25T10:07:44.937+0000 7f1df6ffd640 1 --2- 172.21.15.71:0/3596385450 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1dd4004310 0x7f1dd400fbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-08-25T10:07:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-08-25T10:07:45.583 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-08-25T10:07:45.599 INFO:teuthology.misc:Shutting down mds daemons... 2023-08-25T10:07:45.600 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-08-25T10:07:45.601 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.606 INFO:tasks.ceph.mds.a:Stopped 2023-08-25T10:07:45.607 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-08-25T10:07:45.608 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.609 INFO:tasks.ceph.mds.c:Stopped 2023-08-25T10:07:45.610 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-08-25T10:07:45.611 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.611 INFO:tasks.ceph.mds.e:Stopped 2023-08-25T10:07:45.612 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-08-25T10:07:45.613 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.614 INFO:tasks.ceph.mds.b:Stopped 2023-08-25T10:07:45.615 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-08-25T10:07:45.615 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.616 INFO:tasks.ceph.mds.d:Stopped 2023-08-25T10:07:45.617 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-08-25T10:07:45.618 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.619 INFO:tasks.ceph.mds.f:Stopped 2023-08-25T10:07:45.620 INFO:teuthology.misc:Shutting down osd daemons... 2023-08-25T10:07:45.621 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-08-25T10:07:45.621 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.622 INFO:tasks.ceph.osd.0:Stopped 2023-08-25T10:07:45.623 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-08-25T10:07:45.624 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.625 INFO:tasks.ceph.osd.1:Stopped 2023-08-25T10:07:45.625 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-08-25T10:07:45.626 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.627 INFO:tasks.ceph.osd.2:Stopped 2023-08-25T10:07:45.627 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-08-25T10:07:45.627 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.628 INFO:tasks.ceph.osd.3:Stopped 2023-08-25T10:07:45.628 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-08-25T10:07:45.629 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.630 INFO:tasks.ceph.osd.4:Stopped 2023-08-25T10:07:45.630 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-08-25T10:07:45.631 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.631 INFO:tasks.ceph.osd.5:Stopped 2023-08-25T10:07:45.632 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-08-25T10:07:45.632 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.633 INFO:tasks.ceph.osd.6:Stopped 2023-08-25T10:07:45.633 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-08-25T10:07:45.634 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.634 INFO:tasks.ceph.osd.7:Stopped 2023-08-25T10:07:45.635 INFO:teuthology.misc:Shutting down mgr daemons... 2023-08-25T10:07:45.635 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-08-25T10:07:45.636 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.636 INFO:tasks.ceph.mgr.y:Stopped 2023-08-25T10:07:45.637 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-08-25T10:07:45.637 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T10:07:45.638 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-08-25T10:07:45.660 INFO:tasks.ceph.mgr.x:Stopped 2023-08-25T10:07:45.661 INFO:teuthology.misc:Shutting down mon daemons... 2023-08-25T10:07:45.662 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-08-25T10:07:45.663 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.664 INFO:tasks.ceph.mon.a:Stopped 2023-08-25T10:07:45.664 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-08-25T10:07:45.665 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.676 INFO:tasks.ceph.mon.b:Stopped 2023-08-25T10:07:45.677 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-08-25T10:07:45.677 INFO:teuthology.orchestra.run:waiting for 300 2023-08-25T10:07:45.678 INFO:tasks.ceph.mon.c:Stopped 2023-08-25T10:07:45.678 INFO:tasks.ceph:Checking cluster log for badness... 2023-08-25T10:07:45.679 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-08-25T10:07:45.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T10:07:45.814 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-08-25T10:07:46.082 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T10:07:46.082 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-08-25T10:07:46.390 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T10:07:46.391 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-08-25T10:07:46.692 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T10:07:46.693 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-08-25T10:07:46.953 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T10:07:46.954 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-08-25T10:07:47.239 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T10:07:47.240 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-08-25T10:07:47.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T10:07:47.534 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-08-25T10:07:47.840 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T10:07:47.841 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-08-25T10:07:48.120 INFO:tasks.ceph:Archiving mon data... 2023-08-25T10:07:48.127 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/data/mon.a.tgz 2023-08-25T10:07:48.128 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2023-08-25T10:07:48.190 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.djXXUE09MJ 2023-08-25T10:07:48.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.djXXUE09MJ 2023-08-25T10:07:48.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.djXXUE09MJ 2023-08-25T10:07:48.548 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.djXXUE09MJ is 399KB 2023-08-25T10:07:48.611 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.djXXUE09MJ 2023-08-25T10:07:48.665 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/data/mon.b.tgz 2023-08-25T10:07:48.666 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2023-08-25T10:07:48.726 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.QZYXTAYs40 2023-08-25T10:07:48.727 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.QZYXTAYs40 2023-08-25T10:07:48.976 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.QZYXTAYs40 2023-08-25T10:07:49.136 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.QZYXTAYs40 is 415KB 2023-08-25T10:07:49.244 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.QZYXTAYs40 2023-08-25T10:07:49.302 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/data/mon.c.tgz 2023-08-25T10:07:49.303 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2023-08-25T10:07:49.356 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.JBRWWvUc0G 2023-08-25T10:07:49.356 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JBRWWvUc0G 2023-08-25T10:07:49.553 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.JBRWWvUc0G 2023-08-25T10:07:49.708 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.JBRWWvUc0G is 415KB 2023-08-25T10:07:49.817 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.JBRWWvUc0G 2023-08-25T10:07:49.877 INFO:tasks.ceph:Cleaning ceph cluster... 2023-08-25T10:07:49.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-08-25T10:07:49.887 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-08-25T10:07:49.985 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-08-25T10:07:49.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-08-25T10:07:49.992 DEBUG:teuthology.orchestra.run.smithi164:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-08-25T10:07:50.091 INFO:teuthology.orchestra.run.smithi071.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-08-25T10:07:50.100 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-08-25T10:07:50.114 INFO:tasks.ceph:Archiving crash dumps... 2023-08-25T10:07:50.118 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/remote/smithi071/crash 2023-08-25T10:07:50.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-08-25T10:07:50.206 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/remote/smithi164/crash 2023-08-25T10:07:50.207 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-08-25T10:07:50.309 INFO:tasks.ceph:Shutting down logrotate 2023-08-25T10:07:50.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:50.319 DEBUG:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:50.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:50.573 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-08-25T10:07:50.677 INFO:tasks.ceph:Compressing logs... 2023-08-25T10:07:50.678 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-25T10:07:50.687 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-25T10:07:50.776 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105136.log 2023-08-25T10:07:50.778 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110125.log 2023-08-25T10:07:50.779 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105136.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105136.log.gz 2023-08-25T10:07:50.779 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89517.log 2023-08-25T10:07:50.780 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108522.log 2023-08-25T10:07:50.781 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89306.log 2023-08-25T10:07:50.781 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113665.log 2023-08-25T10:07:50.781 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.110125.log.gz 2023-08-25T10:07:50.782 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99303.log 2023-08-25T10:07:50.783 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108522.log.gz 2023-08-25T10:07:50.783 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89397.log 2023-08-25T10:07:50.783 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89517.log.gz 2023-08-25T10:07:50.784 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89306.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-08-25T10:07:50.784 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.89306.log.gz 2023-08-25T10:07:50.785 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89397.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89397.log.gz 2023-08-25T10:07:50.785 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89347.log 2023-08-25T10:07:50.786 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104150.log 2023-08-25T10:07:50.786 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113665.log.gz 2023-08-25T10:07:50.787 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95193.log 2023-08-25T10:07:50.787 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99303.log.gz 2023-08-25T10:07:50.787 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104150.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104150.log.gz 2023-08-25T10:07:50.787 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-08-25T10:07:50.788 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97477.log 2023-08-25T10:07:50.789 INFO:teuthology.orchestra.run.smithi071.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.95193.log.gz 2023-08-25T10:07:50.789 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114455.log 2023-08-25T10:07:50.789 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-08-25T10:07:50.790 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89347.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-08-25T10:07:50.790 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89347.log.gz 2023-08-25T10:07:50.790 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-08-25T10:07:50.790 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-08-25T10:07:50.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-08-25T10:07:50.792 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97477.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97477.log.gz 2023-08-25T10:07:50.792 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94699.log 2023-08-25T10:07:50.792 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105312.log 2023-08-25T10:07:50.792 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.114455.log.gz 2023-08-25T10:07:50.793 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109855.log 2023-08-25T10:07:50.794 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89226.log 2023-08-25T10:07:50.794 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105312.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.94699.log.gz 2023-08-25T10:07:50.794 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105312.log.gz 2023-08-25T10:07:50.794 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105670.log 2023-08-25T10:07:50.796 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97645.log 2023-08-25T10:07:50.796 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109855.log.gz 2023-08-25T10:07:50.796 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105670.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105670.log.gz 2023-08-25T10:07:50.797 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95109.log 2023-08-25T10:07:50.797 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-08-25T10:07:50.797 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89226.log: 86.9% 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-08-25T10:07:50.798 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89226.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89437.log 2023-08-25T10:07:50.798 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-08-25T10:07:50.799 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-08-25T10:07:50.800 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104968.log 2023-08-25T10:07:50.800 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97645.log.gz 2023-08-25T10:07:50.800 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95109.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95109.log.gz 2023-08-25T10:07:50.800 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114287.log 2023-08-25T10:07:50.801 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113264.log 2023-08-25T10:07:50.801 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104968.log.gz 2023-08-25T10:07:50.802 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114287.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114287.log.gz -5 2023-08-25T10:07:50.802 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.93875.log 2023-08-25T10:07:50.803 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113264.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.113264.log.gz 2023-08-25T10:07:50.803 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113509.log 2023-08-25T10:07:50.803 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89437.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-08-25T10:07:50.803 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.89437.log.gz 2023-08-25T10:07:50.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-08-25T10:07:50.805 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112994.log 2023-08-25T10:07:50.805 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93875.log.gz 2023-08-25T10:07:50.805 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113509.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113509.log.gz 2023-08-25T10:07:50.805 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109289.log 2023-08-25T10:07:50.806 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104004.log 2023-08-25T10:07:50.806 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112994.log.gz 2023-08-25T10:07:50.807 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109289.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109289.log.gz 2023-08-25T10:07:50.807 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93661.log 2023-08-25T10:07:50.808 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113222.log 2023-08-25T10:07:50.808 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104004.log.gz 2023-08-25T10:07:50.809 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93661.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.93661.log.gz 2023-08-25T10:07:50.809 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111634.log 2023-08-25T10:07:50.810 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113222.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113222.log.gz 2023-08-25T10:07:50.810 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99219.log 2023-08-25T10:07:50.811 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93833.log 2023-08-25T10:07:50.812 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111634.log.gz 2023-08-25T10:07:50.812 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99219.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99219.log.gz 2023-08-25T10:07:50.813 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114533.log 2023-08-25T10:07:50.814 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93833.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93833.log.gz 2023-08-25T10:07:50.814 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112342.log 2023-08-25T10:07:50.815 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mon.b.log: 94.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-08-25T10:07:50.816 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-08-25T10:07:50.817 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114533.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114533.log.gz 2023-08-25T10:07:50.817 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112742.log 2023-08-25T10:07:50.817 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112342.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112342.log.gz 2023-08-25T10:07:50.817 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96275.log 2023-08-25T10:07:50.818 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101162.log 2023-08-25T10:07:50.818 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112742.log.gz 2023-08-25T10:07:50.819 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96275.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.102002.log -- replaced with /var/log/ceph/ceph-client.admin.96275.log.gz 2023-08-25T10:07:50.819 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:50.820 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101162.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.112430.log -- replaced with /var/log/ceph/ceph-client.admin.101162.log.gz 2023-08-25T10:07:50.820 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:50.821 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106659.log 2023-08-25T10:07:50.821 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102002.log.gz 2023-08-25T10:07:50.822 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112430.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.112430.log.gz 2023-08-25T10:07:50.823 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103825.log 2023-08-25T10:07:50.823 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106659.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106659.log.gz 2023-08-25T10:07:50.824 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107916.log 2023-08-25T10:07:50.825 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103825.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103825.log.gz 2023-08-25T10:07:50.825 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106083.log 2023-08-25T10:07:50.826 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107916.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107916.log.gz 2023-08-25T10:07:50.826 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109501.log 2023-08-25T10:07:50.827 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106083.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106083.log.gz 2023-08-25T10:07:50.827 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103962.log 2023-08-25T10:07:50.828 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109501.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109501.log.gz 2023-08-25T10:07:50.828 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99135.log 2023-08-25T10:07:50.829 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103962.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103962.log.gz 2023-08-25T10:07:50.830 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104362.log 2023-08-25T10:07:50.831 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99135.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99135.log.gz 2023-08-25T10:07:50.831 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107832.log 2023-08-25T10:07:50.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89477.log 2023-08-25T10:07:50.832 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104362.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104362.log.gz 2023-08-25T10:07:50.832 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-08-25T10:07:50.833 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107832.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107832.log.gz 2023-08-25T10:07:50.833 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109076.log 2023-08-25T10:07:50.834 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94213.log 2023-08-25T10:07:50.835 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109076.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109076.log.gz 2023-08-25T10:07:50.835 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97140.log 2023-08-25T10:07:50.837 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96597.log 2023-08-25T10:07:50.837 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97140.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.94213.log.gz 2023-08-25T10:07:50.838 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.log: 92.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89266.log 2023-08-25T10:07:50.839 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97140.log.gz 2023-08-25T10:07:50.839 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113162.log 2023-08-25T10:07:50.840 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96597.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96597.log.gz 2023-08-25T10:07:50.841 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113891.log 2023-08-25T10:07:50.841 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100551.log 2023-08-25T10:07:50.842 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113162.log.gz 2023-08-25T10:07:50.842 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113891.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113891.log.gz 2023-08-25T10:07:50.843 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94941.log 2023-08-25T10:07:50.844 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100551.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.100551.log.gz --verbose 2023-08-25T10:07:50.844 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.95969.log 2023-08-25T10:07:50.845 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94941.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94941.log.gz 2023-08-25T10:07:50.846 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89477.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-08-25T10:07:50.846 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-08-25T10:07:50.846 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89477.log.gz 2023-08-25T10:07:50.847 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93791.log 2023-08-25T10:07:50.847 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95969.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-08-25T10:07:50.848 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95969.log.gz 2023-08-25T10:07:50.848 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93791.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.93791.log.gz 2023-08-25T10:07:50.849 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105220.log 2023-08-25T10:07:50.850 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107790.log 2023-08-25T10:07:50.851 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105220.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105220.log.gz 2023-08-25T10:07:50.852 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105052.log 2023-08-25T10:07:50.853 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107790.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107790.log.gz 2023-08-25T10:07:50.854 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96614.log 2023-08-25T10:07:50.855 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97603.log 2023-08-25T10:07:50.855 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105052.log.gz 2023-08-25T10:07:50.856 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96614.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96614.log.gz 2023-08-25T10:07:50.857 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110335.log 2023-08-25T10:07:50.858 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97603.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97603.log.gz 2023-08-25T10:07:50.858 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110065.log 2023-08-25T10:07:50.859 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.89266.log: /var/log/ceph/ceph-mds.f.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89266.log.gz 2023-08-25T10:07:50.860 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113975.log 2023-08-25T10:07:50.860 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110335.log.gz 2023-08-25T10:07:50.861 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110065.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110065.log.gz 2023-08-25T10:07:50.862 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104236.log 2023-08-25T10:07:50.863 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113975.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113975.log.gz 2023-08-25T10:07:50.864 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101440.log 2023-08-25T10:07:50.865 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104236.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104236.log.gz 2023-08-25T10:07:50.866 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103113.log 2023-08-25T10:07:50.867 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112388.log 2023-08-25T10:07:50.867 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101440.log.gz 2023-08-25T10:07:50.869 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103113.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103113.log.gz 2023-08-25T10:07:50.869 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103071.log 2023-08-25T10:07:50.870 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112388.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112388.log.gz 2023-08-25T10:07:50.870 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99009.log 2023-08-25T10:07:50.872 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103071.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.103071.log.gz 2023-08-25T10:07:50.872 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106575.log 2023-08-25T10:07:50.873 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99009.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99009.log.gz 2023-08-25T10:07:50.873 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112514.log 2023-08-25T10:07:50.874 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106575.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106575.log.gz 2023-08-25T10:07:50.875 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96264.log 2023-08-25T10:07:50.876 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100993.log 2023-08-25T10:07:50.876 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112514.log.gz 2023-08-25T10:07:50.877 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96264.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96264.log.gz 2023-08-25T10:07:50.878 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97038.log 2023-08-25T10:07:50.879 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100993.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100993.log.gz 2023-08-25T10:07:50.880 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112910.log 2023-08-25T10:07:50.881 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97038.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97038.log.gz 2023-08-25T10:07:50.881 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104194.log 2023-08-25T10:07:50.882 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112910.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112910.log.gz 2023-08-25T10:07:50.883 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99555.log 2023-08-25T10:07:50.884 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89184.log 2023-08-25T10:07:50.885 INFO:teuthology.orchestra.run.smithi071.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104194.log.gz 2023-08-25T10:07:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99555.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99555.log.gz 2023-08-25T10:07:50.886 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95801.log 2023-08-25T10:07:50.887 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.89184.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.89184.log.gz 2023-08-25T10:07:50.888 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107537.log 2023-08-25T10:07:50.889 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93110.log 2023-08-25T10:07:50.889 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95801.log.gz 2023-08-25T10:07:50.890 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107537.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107537.log.gz 2023-08-25T10:07:50.891 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110209.log 2023-08-25T10:07:50.892 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114391.log 2023-08-25T10:07:50.892 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93110.log.gz 2023-08-25T10:07:50.893 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110209.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110209.log.gz 2023-08-25T10:07:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93919.log 2023-08-25T10:07:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114391.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114391.log.gz 2023-08-25T10:07:50.895 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96603.log 2023-08-25T10:07:50.896 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93919.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93919.log.gz 2023-08-25T10:07:50.897 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108724.log 2023-08-25T10:07:50.898 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106198.log 2023-08-25T10:07:50.898 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96603.log.gz 2023-08-25T10:07:50.900 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108724.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108724.log.gz 2023-08-25T10:07:50.900 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109981.log 2023-08-25T10:07:50.901 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106198.log: gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.97182.log -- replaced with /var/log/ceph/ceph-client.admin.106198.log.gz 2023-08-25T10:07:50.901 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:50.902 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109981.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109981.log.gz 2023-08-25T10:07:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107748.log 2023-08-25T10:07:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102842.log 2023-08-25T10:07:50.905 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97182.log.gz 2023-08-25T10:07:50.906 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98163.log 2023-08-25T10:07:50.906 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.107748.log.gz 2023-08-25T10:07:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97435.log 2023-08-25T10:07:50.908 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102842.log.gz 2023-08-25T10:07:50.909 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110689.log 2023-08-25T10:07:50.909 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98163.log.gz 2023-08-25T10:07:50.911 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108396.log 2023-08-25T10:07:50.911 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97435.log.gz 2023-08-25T10:07:50.912 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111338.log 2023-08-25T10:07:50.913 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110689.log.gz 2023-08-25T10:07:50.914 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114307.log 2023-08-25T10:07:50.914 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108396.log.gz 2023-08-25T10:07:50.915 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111338.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111338.log.gz 2023-08-25T10:07:50.915 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101348.log 2023-08-25T10:07:50.916 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95151.log 2023-08-25T10:07:50.917 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114307.log.gz 2023-08-25T10:07:50.918 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101348.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101348.log.gz 2023-08-25T10:07:50.918 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95709.log 2023-08-25T10:07:50.919 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111465.log 2023-08-25T10:07:50.920 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95151.log.gz 2023-08-25T10:07:50.920 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95709.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95709.log.gz 2023-08-25T10:07:50.921 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110251.log 2023-08-25T10:07:50.922 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107099.log 2023-08-25T10:07:50.922 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111465.log.gz 2023-08-25T10:07:50.924 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97350.log 2023-08-25T10:07:50.924 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110251.log.gz 2023-08-25T10:07:50.925 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107225.log 2023-08-25T10:07:50.926 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107099.log.gz 2023-08-25T10:07:50.926 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97350.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.97350.log.gz 2023-08-25T10:07:50.927 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114017.log 2023-08-25T10:07:50.928 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107225.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-08-25T10:07:50.928 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107225.log.gz 2023-08-25T10:07:50.929 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114017.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.114017.log.gz 2023-08-25T10:07:50.930 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114143.log 2023-08-25T10:07:50.931 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105459.log 2023-08-25T10:07:50.932 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113791.log 2023-08-25T10:07:50.933 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.114143.log.gz 2023-08-25T10:07:50.934 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111862.log 2023-08-25T10:07:50.934 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105459.log.gz 2023-08-25T10:07:50.936 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113791.log: gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113791.log.gz 2023-08-25T10:07:50.937 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.110293.log 2023-08-25T10:07:50.938 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111862.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111862.log.gz 2023-08-25T10:07:50.938 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102800.log 2023-08-25T10:07:50.940 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95335.log 2023-08-25T10:07:50.941 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110293.log.gz 2023-08-25T10:07:50.942 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102800.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102800.log.gz 2023-08-25T10:07:50.942 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103867.log 2023-08-25T10:07:50.942 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108808.log 2023-08-25T10:07:50.943 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95335.log.gz 2023-08-25T10:07:50.944 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97561.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103867.log.gz 2023-08-25T10:07:50.944 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:50.945 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108808.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108808.log.gz 2023-08-25T10:07:50.946 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110899.log 2023-08-25T10:07:50.947 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106370.log 2023-08-25T10:07:50.947 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97561.log.gz 2023-08-25T10:07:50.949 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110899.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110899.log.gz 2023-08-25T10:07:50.949 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.109897.log 2023-08-25T10:07:50.950 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106370.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106370.log.gz 2023-08-25T10:07:50.951 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109416.log 2023-08-25T10:07:50.952 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107874.log 2023-08-25T10:07:50.952 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109897.log.gz 2023-08-25T10:07:50.954 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101288.log 2023-08-25T10:07:50.954 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109416.log.gz 2023-08-25T10:07:50.955 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107874.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107874.log.gz 2023-08-25T10:07:50.955 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108102.log 2023-08-25T10:07:50.957 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107706.log 2023-08-25T10:07:50.957 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101288.log.gz 2023-08-25T10:07:50.958 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108102.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.108102.log.gz /var/log/ceph/ceph-client.admin.109332.log 2023-08-25T10:07:50.959 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:50.960 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107706.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107706.log.gz 2023-08-25T10:07:50.960 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.105712.log 2023-08-25T10:07:50.962 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109332.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109332.log.gz 2023-08-25T10:07:50.962 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100908.log 2023-08-25T10:07:50.963 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109162.log 2023-08-25T10:07:50.963 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105712.log.gz 2023-08-25T10:07:50.965 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100908.log: gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100908.log.gz 2023-08-25T10:07:50.965 INFO:teuthology.orchestra.run.smithi071.stderr: --verbose -- /var/log/ceph/ceph-client.admin.93747.log 2023-08-25T10:07:50.966 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109162.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109162.log.gz 2023-08-25T10:07:50.967 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107141.log 2023-08-25T10:07:50.968 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93747.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-08-25T10:07:50.968 INFO:teuthology.orchestra.run.smithi071.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93747.log.gz 2023-08-25T10:07:50.970 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107141.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.107141.log.gz -- 2023-08-25T10:07:50.970 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.111778.log 2023-08-25T10:07:50.971 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109813.log 2023-08-25T10:07:50.972 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111778.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111778.log.gz 2023-08-25T10:07:50.974 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105178.log 2023-08-25T10:07:50.974 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109813.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109813.log.gz 2023-08-25T10:07:50.975 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113581.log 2023-08-25T10:07:50.976 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105178.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105178.log.gz 2023-08-25T10:07:50.977 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108681.log 2023-08-25T10:07:50.979 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113581.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113581.log.gz 2023-08-25T10:07:50.979 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105586.log 2023-08-25T10:07:50.980 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108681.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108681.log.gz 2023-08-25T10:07:50.981 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112030.log 2023-08-25T10:07:50.982 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105586.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105586.log.gz 2023-08-25T10:07:50.983 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111381.log 2023-08-25T10:07:50.984 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112030.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112030.log.gz 2023-08-25T10:07:50.984 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111025.log 2023-08-25T10:07:50.986 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111381.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111381.log.gz 2023-08-25T10:07:50.986 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96604.log 2023-08-25T10:07:50.988 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111025.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111025.log.gz 2023-08-25T10:07:50.988 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96273.log 2023-08-25T10:07:50.989 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96604.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96604.log.gz 2023-08-25T10:07:50.990 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109543.log 2023-08-25T10:07:50.991 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96273.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96273.log.gz 2023-08-25T10:07:50.991 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93449.log 2023-08-25T10:07:50.993 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109543.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110773.log 2023-08-25T10:07:50.993 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109543.log.gz 2023-08-25T10:07:50.995 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93449.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93449.log.gz 2023-08-25T10:07:50.995 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.111296.log 2023-08-25T10:07:50.996 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110773.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110773.log.gz 2023-08-25T10:07:50.997 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105010.log 2023-08-25T10:07:50.998 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111296.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111296.log.gz 2023-08-25T10:07:51.000 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99471.log 2023-08-25T10:07:51.000 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105010.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105010.log.gz 2023-08-25T10:07:51.003 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105354.log 2023-08-25T10:07:51.003 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99471.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99471.log.gz 2023-08-25T10:07:51.003 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93208.log 2023-08-25T10:07:51.006 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105354.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105354.log.gz 2023-08-25T10:07:51.007 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107351.log 2023-08-25T10:07:51.007 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93208.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93208.log.gz 2023-08-25T10:07:51.007 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109939.log 2023-08-25T10:07:51.008 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107351.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107351.log.gz 2023-08-25T10:07:51.009 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107309.log 2023-08-25T10:07:51.009 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109939.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109939.log.gz 2023-08-25T10:07:51.010 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94171.log 2023-08-25T10:07:51.012 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107309.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107309.log.gz 2023-08-25T10:07:51.012 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109458.log 2023-08-25T10:07:51.014 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94171.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94171.log.gz 2023-08-25T10:07:51.014 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107267.log 2023-08-25T10:07:51.016 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109458.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109458.log.gz 2023-08-25T10:07:51.016 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114617.log 2023-08-25T10:07:51.018 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107267.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107267.log.gz 2023-08-25T10:07:51.018 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93535.log 2023-08-25T10:07:51.018 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114617.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114617.log.gz 2023-08-25T10:07:51.019 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99429.log 2023-08-25T10:07:51.020 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93535.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.93535.log.gz 2023-08-25T10:07:51.020 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107622.log 2023-08-25T10:07:51.021 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99429.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99429.log.gz 2023-08-25T10:07:51.022 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100722.log 2023-08-25T10:07:51.023 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107622.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107622.log.gz 2023-08-25T10:07:51.024 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109771.log 2023-08-25T10:07:51.025 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100722.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100722.log.gz 2023-08-25T10:07:51.026 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100782.log 2023-08-25T10:07:51.027 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109771.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.109771.log.gz 2023-08-25T10:07:51.028 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108354.log 2023-08-25T10:07:51.029 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100782.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100782.log.gz 2023-08-25T10:07:51.030 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96994.log 2023-08-25T10:07:51.031 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108354.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108354.log.gz 2023-08-25T10:07:51.032 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110587.log 2023-08-25T10:07:51.033 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96994.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96994.log.gz 2023-08-25T10:07:51.033 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112258.log 2023-08-25T10:07:51.034 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110587.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110587.log.gz 2023-08-25T10:07:51.035 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103029.log 2023-08-25T10:07:51.036 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112258.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112258.log.gz 2023-08-25T10:07:51.037 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96277.log 2023-08-25T10:07:51.038 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103029.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103029.log.gz 2023-08-25T10:07:51.039 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113036.log 2023-08-25T10:07:51.041 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96277.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96277.log.gz 2023-08-25T10:07:51.041 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98035.log 2023-08-25T10:07:51.042 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113036.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113036.log.gz 2023-08-25T10:07:51.043 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102672.log 2023-08-25T10:07:51.044 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98035.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-08-25T10:07:51.044 INFO:teuthology.orchestra.run.smithi071.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.98035.log.gz 2023-08-25T10:07:51.045 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110377.log 2023-08-25T10:07:51.047 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102672.log: /var/log/ceph/ceph.audit.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102672.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113749.log 2023-08-25T10:07:51.047 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:51.049 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110377.log: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-08-25T10:07:51.050 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114216.log 2023-08-25T10:07:51.051 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.110377.log.gz 2023-08-25T10:07:51.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107579.log 2023-08-25T10:07:51.052 INFO:teuthology.orchestra.run.smithi071.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.113749.log.gz 2023-08-25T10:07:51.052 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114216.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114216.log.gz 2023-08-25T10:07:51.053 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103909.log 2023-08-25T10:07:51.055 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107579.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.107579.log.gz 2023-08-25T10:07:51.055 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.101119.log 2023-08-25T10:07:51.056 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97729.log 2023-08-25T10:07:51.056 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103909.log.gz 2023-08-25T10:07:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99513.log 2023-08-25T10:07:51.058 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101119.log.gz 2023-08-25T10:07:51.059 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96612.log 2023-08-25T10:07:51.059 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97729.log.gz 2023-08-25T10:07:51.060 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99513.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112682.log 2023-08-25T10:07:51.061 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.99513.log.gz 2023-08-25T10:07:51.062 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108438.log 2023-08-25T10:07:51.063 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96612.log.gz 2023-08-25T10:07:51.064 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112952.log 2023-08-25T10:07:51.064 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112682.log.gz 2023-08-25T10:07:51.065 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104692.log 2023-08-25T10:07:51.066 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108438.log.gz 2023-08-25T10:07:51.067 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112114.log 2023-08-25T10:07:51.067 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112952.log.gz 2023-08-25T10:07:51.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104692.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96901.log 2023-08-25T10:07:51.069 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104692.log.gz 2023-08-25T10:07:51.070 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112114.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112114.log.gz 2023-08-25T10:07:51.070 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.100866.log 2023-08-25T10:07:51.071 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96901.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96901.log.gz 2023-08-25T10:07:51.072 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102969.log 2023-08-25T10:07:51.073 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100866.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102927.log 2023-08-25T10:07:51.074 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.100866.log.gz 2023-08-25T10:07:51.075 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99345.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102969.log.gz 2023-08-25T10:07:51.076 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:07:51.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102927.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102927.log.gz 2023-08-25T10:07:51.078 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98883.log 2023-08-25T10:07:51.078 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99345.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.99345.log.gz 2023-08-25T10:07:51.079 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.83170.log 2023-08-25T10:07:51.081 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98883.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98883.log.gz 2023-08-25T10:07:51.082 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.102587.log 2023-08-25T10:07:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.tmp-client.admin.83170.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.83170.log.gz 2023-08-25T10:07:51.082 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100596.log 2023-08-25T10:07:51.084 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102587.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102587.log.gz 2023-08-25T10:07:51.085 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.97771.log 2023-08-25T10:07:51.085 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100596.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100596.log.gz 2023-08-25T10:07:51.086 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110461.log 2023-08-25T10:07:51.088 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97771.log: gzip -5 --verbose -- 91.0% -- replaced with /var/log/ceph/ceph-client.admin.97771.log.gz 2023-08-25T10:07:51.088 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-mds.c.log 2023-08-25T10:07:51.089 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110461.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110461.log.gz 2023-08-25T10:07:51.090 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111904.log 2023-08-25T10:07:51.092 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95885.log 2023-08-25T10:07:51.093 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111904.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111904.log.gz 2023-08-25T10:07:51.094 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105816.log 2023-08-25T10:07:51.095 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97687.log 2023-08-25T10:07:51.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95885.log: /var/log/ceph/ceph-client.admin.105816.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95885.log.gz 2023-08-25T10:07:51.097 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105816.log.gz 2023-08-25T10:07:51.098 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101035.log 2023-08-25T10:07:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107183.log 2023-08-25T10:07:51.100 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97687.log: /var/log/ceph/ceph-client.admin.101035.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.97687.log.gz 2023-08-25T10:07:51.101 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101035.log.gz 2023-08-25T10:07:51.101 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104320.log 2023-08-25T10:07:51.103 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106041.log 2023-08-25T10:07:51.104 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107183.log: /var/log/ceph/ceph-client.admin.104320.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107183.log.gz 2023-08-25T10:07:51.105 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104320.log.gz 2023-08-25T10:07:51.105 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94983.log 2023-08-25T10:07:51.107 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98799.log 2023-08-25T10:07:51.109 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94983.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106041.log.gz 2023-08-25T10:07:51.109 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94983.log.gz 2023-08-25T10:07:51.110 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93493.log 2023-08-25T10:07:51.111 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110983.log 2023-08-25T10:07:51.112 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98799.log: /var/log/ceph/ceph-client.admin.93493.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98799.log.gz 2023-08-25T10:07:51.113 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93493.log.gz 2023-08-25T10:07:51.114 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97266.log 2023-08-25T10:07:51.115 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105270.log 2023-08-25T10:07:51.117 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110983.log: /var/log/ceph/ceph-client.admin.97266.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110983.log.gz 2023-08-25T10:07:51.118 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97266.log.gz 2023-08-25T10:07:51.118 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108000.log 2023-08-25T10:07:51.120 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95293.log 2023-08-25T10:07:51.121 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105270.log: /var/log/ceph/ceph-client.admin.108000.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105270.log.gz 2023-08-25T10:07:51.121 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108000.log.gz 2023-08-25T10:07:51.122 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96945.log 2023-08-25T10:07:51.124 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96011.log 2023-08-25T10:07:51.125 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95293.log: /var/log/ceph/ceph-client.admin.96945.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95293.log.gz 2023-08-25T10:07:51.126 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96945.log.gz 2023-08-25T10:07:51.127 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95067.log 2023-08-25T10:07:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95667.log 2023-08-25T10:07:51.129 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96011.log: /var/log/ceph/ceph-client.admin.95067.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96011.log.gz 2023-08-25T10:07:51.130 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95067.log.gz 2023-08-25T10:07:51.130 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108934.log 2023-08-25T10:07:51.132 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97519.log 2023-08-25T10:07:51.133 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95667.log: /var/log/ceph/ceph-client.admin.108934.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95667.log.gz 2023-08-25T10:07:51.134 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108934.log.gz 2023-08-25T10:07:51.134 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103664.log 2023-08-25T10:07:51.136 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94045.log 2023-08-25T10:07:51.137 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97519.log: /var/log/ceph/ceph-client.admin.103664.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97519.log.gz 2023-08-25T10:07:51.138 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.103664.log.gz 2023-08-25T10:07:51.139 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94279.log 2023-08-25T10:07:51.140 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96817.log 2023-08-25T10:07:51.141 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94045.log: /var/log/ceph/ceph-client.admin.94279.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94045.log.gz 2023-08-25T10:07:51.143 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94279.log.gz 2023-08-25T10:07:51.143 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105501.log 2023-08-25T10:07:51.144 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108270.log 2023-08-25T10:07:51.145 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96817.log: /var/log/ceph/ceph-client.admin.105501.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.96817.log.gz 2023-08-25T10:07:51.147 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105501.log.gz 2023-08-25T10:07:51.147 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95025.log 2023-08-25T10:07:51.149 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108480.log 2023-08-25T10:07:51.151 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108270.log: /var/log/ceph/ceph-client.admin.95025.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108270.log.gz 2023-08-25T10:07:51.151 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95025.log.gz 2023-08-25T10:07:51.152 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99639.log 2023-08-25T10:07:51.153 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103741.log 2023-08-25T10:07:51.154 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108480.log: /var/log/ceph/ceph-client.admin.99639.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108480.log.gz 2023-08-25T10:07:51.155 INFO:teuthology.orchestra.run.smithi071.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99639.log.gz 2023-08-25T10:07:51.156 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109374.log 2023-08-25T10:07:51.157 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110647.log 2023-08-25T10:07:51.158 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103741.log: /var/log/ceph/ceph-client.admin.109374.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103741.log.gz 2023-08-25T10:07:51.159 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109374.log.gz 2023-08-25T10:07:51.159 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112216.log 2023-08-25T10:07:51.160 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111254.log 2023-08-25T10:07:51.161 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110647.log: /var/log/ceph/ceph-client.admin.112216.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110647.log.gz 2023-08-25T10:07:51.162 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112216.log.gz 2023-08-25T10:07:51.163 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97837.log 2023-08-25T10:07:51.164 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105628.log 2023-08-25T10:07:51.166 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111254.log: /var/log/ceph/ceph-client.admin.97837.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111254.log.gz 2023-08-25T10:07:51.167 INFO:teuthology.orchestra.run.smithi071.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.97837.log.gz 2023-08-25T10:07:51.167 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104278.log 2023-08-25T10:07:51.169 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102543.log 2023-08-25T10:07:51.170 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105628.log: /var/log/ceph/ceph-client.admin.104278.log: 88.2% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105628.log.gz 2023-08-25T10:07:51.170 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104278.log.gz 2023-08-25T10:07:51.171 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104926.log 2023-08-25T10:07:51.173 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109246.log 2023-08-25T10:07:51.174 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102543.log: /var/log/ceph/ceph-client.admin.104926.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102543.log.gz 2023-08-25T10:07:51.175 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104926.log.gz 2023-08-25T10:07:51.175 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96278.log 2023-08-25T10:07:51.177 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111820.log 2023-08-25T10:07:51.178 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109246.log: /var/log/ceph/ceph-client.admin.96278.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.109246.log.gz 2023-08-25T10:07:51.179 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96278.log.gz 2023-08-25T10:07:51.180 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97903.log 2023-08-25T10:07:51.181 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111109.log 2023-08-25T10:07:51.182 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111820.log: /var/log/ceph/ceph-client.admin.97903.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111820.log.gz 2023-08-25T10:07:51.184 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98925.log 2023-08-25T10:07:51.184 INFO:teuthology.orchestra.run.smithi071.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.97903.log.gz 2023-08-25T10:07:51.186 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97969.log 2023-08-25T10:07:51.186 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98925.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.111109.log.gz 2023-08-25T10:07:51.187 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98925.log.gz 2023-08-25T10:07:51.188 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95461.log 2023-08-25T10:07:51.190 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96610.log 2023-08-25T10:07:51.191 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97969.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.97969.log.gz 2023-08-25T10:07:51.192 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95461.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95461.log.gz 2023-08-25T10:07:51.193 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96274.log 2023-08-25T10:07:51.194 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112868.log 2023-08-25T10:07:51.194 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96610.log.gz 2023-08-25T10:07:51.196 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93703.log 2023-08-25T10:07:51.197 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112868.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96274.log.gz 2023-08-25T10:07:51.197 INFO:teuthology.orchestra.run.smithi071.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.112868.log.gz 2023-08-25T10:07:51.199 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114575.log 2023-08-25T10:07:51.200 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99177.log 2023-08-25T10:07:51.201 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.93703.log.gz 2023-08-25T10:07:51.202 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95927.log 2023-08-25T10:07:51.203 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114575.log: /var/log/ceph/ceph-client.admin.99177.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.114575.log.gz 2023-08-25T10:07:51.203 INFO:teuthology.orchestra.run.smithi164.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-08-25T10:07:51.204 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99177.log.gz 2023-08-25T10:07:51.205 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108892.log 2023-08-25T10:07:51.206 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111212.log 2023-08-25T10:07:51.207 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95927.log: /var/log/ceph/ceph-client.admin.108892.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95927.log.gz 2023-08-25T10:07:51.208 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108892.log.gz 2023-08-25T10:07:51.208 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113933.log 2023-08-25T10:07:51.210 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110857.log 2023-08-25T10:07:51.211 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.111212.log.gz 2023-08-25T10:07:51.213 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96053.log 2023-08-25T10:07:51.213 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113933.log.gz 2023-08-25T10:07:51.214 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110857.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.110857.log.gz 2023-08-25T10:07:51.214 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113364.log 2023-08-25T10:07:51.216 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94129.log 2023-08-25T10:07:51.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96053.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96053.log.gz 2023-08-25T10:07:51.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113078.log 2023-08-25T10:07:51.219 INFO:teuthology.orchestra.run.smithi071.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113364.log.gz 2023-08-25T10:07:51.220 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94129.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.94129.log.gz 2023-08-25T10:07:51.220 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102044.log 2023-08-25T10:07:51.222 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98841.log 2023-08-25T10:07:51.222 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113078.log.gz 2023-08-25T10:07:51.224 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101246.log 2023-08-25T10:07:51.225 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8%/var/log/ceph/ceph-client.admin.98841.log: -- replaced with /var/log/ceph/ceph-client.admin.102044.log.gz 2023-08-25T10:07:51.226 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98841.log.gz 2023-08-25T10:07:51.226 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106295.log 2023-08-25T10:07:51.228 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96859.log 2023-08-25T10:07:51.228 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101246.log.gz 2023-08-25T10:07:51.230 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105878.log 2023-08-25T10:07:51.231 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96859.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106295.log.gz 2023-08-25T10:07:51.231 INFO:teuthology.orchestra.run.smithi071.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96859.log.gz 2023-08-25T10:07:51.233 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111549.log 2023-08-25T10:07:51.234 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101390.log 2023-08-25T10:07:51.234 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105878.log.gz 2023-08-25T10:07:51.236 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94003.log 2023-08-25T10:07:51.237 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101390.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111549.log.gz 2023-08-25T10:07:51.237 INFO:teuthology.orchestra.run.smithi071.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101390.log.gz 2023-08-25T10:07:51.239 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104826.log 2023-08-25T10:07:51.240 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100638.log 2023-08-25T10:07:51.241 INFO:teuthology.orchestra.run.smithi071.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94003.log.gz 2023-08-25T10:07:51.242 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89453.log 2023-08-25T10:07:51.243 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100638.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104826.log.gz 2023-08-25T10:07:51.244 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100638.log.gz 2023-08-25T10:07:51.244 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100951.log 2023-08-25T10:07:51.246 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.89453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98567.log 2023-08-25T10:07:51.247 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.89453.log.gz 2023-08-25T10:07:51.249 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108144.log 2023-08-25T10:07:51.249 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3%/var/log/ceph/ceph-client.0.98567.log: -- replaced with /var/log/ceph/ceph-client.admin.100951.log.gz 2023-08-25T10:07:51.261 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95504.log 2023-08-25T10:07:51.263 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108144.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108144.log.gz 2023-08-25T10:07:51.264 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108624.log 2023-08-25T10:07:51.266 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95504.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95504.log.gz 2023-08-25T10:07:51.268 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107053.log 2023-08-25T10:07:51.270 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108624.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.108624.log.gz 2023-08-25T10:07:51.272 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100680.log 2023-08-25T10:07:51.274 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107053.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107053.log.gz 2023-08-25T10:07:51.276 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113833.log 2023-08-25T10:07:51.278 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100680.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100680.log.gz 2023-08-25T10:07:51.279 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108582.log 2023-08-25T10:07:51.282 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113833.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113833.log.gz 2023-08-25T10:07:51.283 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112784.log 2023-08-25T10:07:51.285 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108582.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108582.log.gz 2023-08-25T10:07:51.287 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99093.log 2023-08-25T10:07:51.289 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112784.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112784.log.gz 2023-08-25T10:07:51.291 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94087.log 2023-08-25T10:07:51.293 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99093.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99093.log.gz 2023-08-25T10:07:51.295 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108186.log 2023-08-25T10:07:51.297 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.94087.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94087.log.gz 2023-08-25T10:07:51.298 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104404.log 2023-08-25T10:07:51.300 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108186.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108186.log.gz 2023-08-25T10:07:51.302 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98757.log 2023-08-25T10:07:51.304 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104404.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104404.log.gz 2023-08-25T10:07:51.306 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114101.log 2023-08-25T10:07:51.308 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98757.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98757.log.gz 2023-08-25T10:07:51.310 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107435.log 2023-08-25T10:07:51.311 INFO:teuthology.orchestra.run.smithi164.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-08-25T10:07:51.311 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114101.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.114101.log.gz 2023-08-25T10:07:51.313 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-08-25T10:07:51.315 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107435.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.107435.log.gz 2023-08-25T10:07:51.317 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111067.log 2023-08-25T10:07:51.333 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110503.log 2023-08-25T10:07:51.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111067.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111067.log.gz 2023-08-25T10:07:51.349 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113425.log 2023-08-25T10:07:51.351 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110503.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110503.log.gz 2023-08-25T10:07:51.361 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-08-25T10:07:51.370 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103621.log 2023-08-25T10:07:51.375 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113425.log.gz 2023-08-25T10:07:51.378 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105754.log 2023-08-25T10:07:51.392 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97098.log 2023-08-25T10:07:51.392 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103621.log.gz 2023-08-25T10:07:51.393 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105754.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105754.log.gz 2023-08-25T10:07:51.405 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106862.log 2023-08-25T10:07:51.407 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97098.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97098.log.gz 2023-08-25T10:07:51.413 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109204.log 2023-08-25T10:07:51.424 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110815.log 2023-08-25T10:07:51.427 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109204.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.106862.log.gz 2023-08-25T10:07:51.437 INFO:teuthology.orchestra.run.smithi071.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109204.log.gz 2023-08-25T10:07:51.437 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-08-25T10:07:51.439 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110815.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.110815.log.gz 2023-08-25T10:07:51.445 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89552.log 2023-08-25T10:07:51.453 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113306.log 2023-08-25T10:07:51.461 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.89552.log: -- replaced with /var/log/ceph/ceph.log.gz 2023-08-25T10:07:51.467 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89552.log.gz 2023-08-25T10:07:51.467 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112640.log 2023-08-25T10:07:51.469 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113306.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113306.log.gz 2023-08-25T10:07:51.475 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101204.log 2023-08-25T10:07:51.489 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112640.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112640.log.gz 2023-08-25T10:07:51.490 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97308.log 2023-08-25T10:07:51.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101204.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101204.log.gz 2023-08-25T10:07:51.500 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109729.log 2023-08-25T10:07:51.511 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110545.log 2023-08-25T10:07:51.514 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109729.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97308.log.gz 2023-08-25T10:07:51.524 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109729.log.gz 2023-08-25T10:07:51.528 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112826.log 2023-08-25T10:07:51.530 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110545.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110545.log.gz 2023-08-25T10:07:51.536 INFO:teuthology.orchestra.run.smithi071.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-08-25T10:07:51.539 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108766.log 2023-08-25T10:07:51.553 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112826.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112826.log.gz 2023-08-25T10:07:51.554 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111507.log 2023-08-25T10:07:51.555 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108766.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108766.log.gz 2023-08-25T10:07:51.567 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111946.log 2023-08-25T10:07:51.568 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111507.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111507.log.gz 2023-08-25T10:07:51.582 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93961.log 2023-08-25T10:07:51.583 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111946.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111946.log.gz 2023-08-25T10:07:51.590 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100824.log 2023-08-25T10:07:51.604 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93961.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93961.log.gz 2023-08-25T10:07:51.610 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112300.log 2023-08-25T10:07:51.612 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.100824.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100824.log.gz 2023-08-25T10:07:51.632 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109018.log 2023-08-25T10:07:51.634 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112300.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112300.log.gz 2023-08-25T10:07:51.646 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113623.log 2023-08-25T10:07:51.648 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109018.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109018.log.gz 2023-08-25T10:07:51.664 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112472.log 2023-08-25T10:07:51.666 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113623.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113623.log.gz 2023-08-25T10:07:51.679 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108976.log 2023-08-25T10:07:51.680 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112472.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112472.log.gz 2023-08-25T10:07:51.690 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96613.log 2023-08-25T10:07:51.704 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108976.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108976.log.gz 2023-08-25T10:07:51.709 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97224.log 2023-08-25T10:07:51.711 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96613.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96613.log.gz 2023-08-25T10:07:51.731 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99051.log 2023-08-25T10:07:51.742 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107958.log 2023-08-25T10:07:51.745 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99051.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97224.log.gz 2023-08-25T10:07:51.749 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114059.log 2023-08-25T10:07:51.755 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107958.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99051.log.gz 2023-08-25T10:07:51.759 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108228.log 2023-08-25T10:07:51.763 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114059.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107958.log.gz 2023-08-25T10:07:51.774 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114059.log.gz 2023-08-25T10:07:51.774 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110419.log 2023-08-25T10:07:51.776 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108228.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108228.log.gz 2023-08-25T10:07:51.782 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110731.log 2023-08-25T10:07:51.784 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110419.log.gz 2023-08-25T10:07:51.785 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106240.log 2023-08-25T10:07:51.786 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110731.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.110731.log.gz 2023-08-25T10:07:51.797 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113467.log 2023-08-25T10:07:51.799 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106240.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.106240.log.gz 2023-08-25T10:07:51.811 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107393.log 2023-08-25T10:07:51.813 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113467.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113467.log.gz 2023-08-25T10:07:51.823 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102501.log 2023-08-25T10:07:51.837 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107393.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.107393.log.gz 2023-08-25T10:07:51.841 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99681.log 2023-08-25T10:07:51.843 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102501.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102501.log.gz 2023-08-25T10:07:51.853 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93619.log 2023-08-25T10:07:51.864 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99681.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102629.log 2023-08-25T10:07:51.868 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93619.log: -- replaced with /var/log/ceph/ceph-client.admin.99681.log.gz 2023-08-25T10:07:51.874 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-08-25T10:07:51.877 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102629.log: 88.4% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93619.log.gz 2023-08-25T10:07:51.887 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.102629.log.gz 2023-08-25T10:07:51.894 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95625.log 2023-08-25T10:07:51.916 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101077.log 2023-08-25T10:07:51.917 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95625.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.95625.log.gz 2023-08-25T10:07:51.928 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112156.log 2023-08-25T10:07:51.939 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101077.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101077.log.gz 2023-08-25T10:07:51.941 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111170.log 2023-08-25T10:07:51.943 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112156.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.112156.log.gz 2023-08-25T10:07:51.952 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110023.log 2023-08-25T10:07:51.965 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111170.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111170.log.gz 2023-08-25T10:07:51.966 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110941.log 2023-08-25T10:07:51.975 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114223.log 2023-08-25T10:07:51.980 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110941.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110023.log.gz 2023-08-25T10:07:51.982 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104064.log 2023-08-25T10:07:51.988 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114223.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110941.log.gz 2023-08-25T10:07:51.992 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98967.log 2023-08-25T10:07:51.996 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104064.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114223.log.gz 2023-08-25T10:07:52.002 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96265.log 2023-08-25T10:07:52.006 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.98967.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104064.log.gz 2023-08-25T10:07:52.016 INFO:teuthology.orchestra.run.smithi071.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98967.log.gz 2023-08-25T10:07:52.023 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109585.log 2023-08-25T10:07:52.025 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96265.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96265.log.gz 2023-08-25T10:07:52.039 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105544.log 2023-08-25T10:07:52.040 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109585.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109585.log.gz 2023-08-25T10:07:52.048 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112072.log 2023-08-25T10:07:52.053 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105544.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105544.log.gz 2023-08-25T10:07:52.068 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96600.log 2023-08-25T10:07:52.068 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112072.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112072.log.gz 2023-08-25T10:07:52.075 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99723.log 2023-08-25T10:07:52.088 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96096.log 2023-08-25T10:07:52.088 INFO:teuthology.orchestra.run.smithi071.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96600.log.gz 2023-08-25T10:07:52.096 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99723.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93307.log 2023-08-25T10:07:52.101 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96096.log: -- replaced with /var/log/ceph/ceph-client.admin.99723.log.gz 2023-08-25T10:07:52.109 INFO:teuthology.orchestra.run.smithi071.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96096.log.gz 2023-08-25T10:07:52.114 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95583.log 2023-08-25T10:07:52.116 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93307.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.93307.log.gz 2023-08-25T10:07:52.126 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102885.log 2023-08-25T10:07:52.140 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95583.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95583.log.gz 2023-08-25T10:07:52.149 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93577.log 2023-08-25T10:07:52.151 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102885.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.102885.log.gz 2023-08-25T10:07:52.157 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114659.log 2023-08-25T10:07:52.168 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95251.log 2023-08-25T10:07:52.171 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114659.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93577.log.gz 2023-08-25T10:07:52.175 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112598.log 2023-08-25T10:07:52.182 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95251.log: 94.9% -- replaced with /var/log/ceph/ceph-client.admin.114659.log.gz 2023-08-25T10:07:52.189 INFO:teuthology.orchestra.run.smithi071.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95251.log.gz 2023-08-25T10:07:52.189 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105417.log 2023-08-25T10:07:52.191 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112598.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112598.log.gz 2023-08-25T10:07:52.205 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111694.log 2023-08-25T10:07:52.217 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93166.log 2023-08-25T10:07:52.218 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111694.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111694.log.gz 2023-08-25T10:07:52.219 INFO:teuthology.orchestra.run.smithi071.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105417.log.gz 2023-08-25T10:07:52.233 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108312.log 2023-08-25T10:07:52.247 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93166.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93166.log.gz 2023-08-25T10:07:52.252 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108850.log 2023-08-25T10:07:52.253 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108312.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108312.log.gz 2023-08-25T10:07:52.265 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107664.log 2023-08-25T10:07:52.267 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108850.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.108850.log.gz 2023-08-25T10:07:52.285 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106474.log 2023-08-25T10:07:52.287 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107664.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107664.log.gz 2023-08-25T10:07:52.301 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101941.log 2023-08-25T10:07:52.303 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.106474.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106474.log.gz 2023-08-25T10:07:52.317 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102087.log 2023-08-25T10:07:52.319 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101941.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.101941.log.gz 2023-08-25T10:07:52.335 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105094.log 2023-08-25T10:07:52.350 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.102087.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102087.log.gz 2023-08-25T10:07:52.352 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112556.log 2023-08-25T10:07:52.353 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.105094.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105094.log.gz 2023-08-25T10:07:52.363 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113707.log 2023-08-25T10:07:52.365 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.112556.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112556.log.gz 2023-08-25T10:07:52.371 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97392.log 2023-08-25T10:07:52.372 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113707.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113707.log.gz 2023-08-25T10:07:52.385 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111423.log 2023-08-25T10:07:52.387 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.97392.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97392.log.gz 2023-08-25T10:07:52.403 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93407.log 2023-08-25T10:07:52.404 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111423.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111423.log.gz 2023-08-25T10:07:52.418 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111736.log 2023-08-25T10:07:52.420 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93407.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93407.log.gz 2023-08-25T10:07:52.427 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95377.log 2023-08-25T10:07:52.435 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101499.log 2023-08-25T10:07:52.440 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95377.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111736.log.gz 2023-08-25T10:07:52.448 INFO:teuthology.orchestra.run.smithi071.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95377.log.gz 2023-08-25T10:07:52.448 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111591.log 2023-08-25T10:07:52.450 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101499.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101499.log.gz 2023-08-25T10:07:52.460 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114372.log 2023-08-25T10:07:52.461 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111591.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111591.log.gz 2023-08-25T10:07:52.467 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96262.log 2023-08-25T10:07:52.478 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.114372.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108060.log 2023-08-25T10:07:52.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.96262.log: -- replaced with /var/log/ceph/ceph-client.admin.114372.log.gz 2023-08-25T10:07:52.485 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107477.log 2023-08-25T10:07:52.491 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.108060.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96262.log.gz 2023-08-25T10:07:52.495 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113120.log 2023-08-25T10:07:52.500 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.107477.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108060.log.gz 2023-08-25T10:07:52.509 INFO:teuthology.orchestra.run.smithi071.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107477.log.gz 2023-08-25T10:07:52.516 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109120.log 2023-08-25T10:07:52.518 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.113120.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113120.log.gz 2023-08-25T10:07:52.530 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95759.log 2023-08-25T10:07:52.532 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109120.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109120.log.gz 2023-08-25T10:07:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111988.log 2023-08-25T10:07:52.547 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95759.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95759.log.gz 2023-08-25T10:07:52.555 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109645.log 2023-08-25T10:07:52.571 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.111988.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111988.log.gz 2023-08-25T10:07:52.577 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99597.log 2023-08-25T10:07:52.579 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109645.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109645.log.gz 2023-08-25T10:07:52.587 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95843.log 2023-08-25T10:07:52.601 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99597.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99597.log.gz 2023-08-25T10:07:52.608 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95419.log 2023-08-25T10:07:52.610 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95843.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95843.log.gz 2023-08-25T10:07:52.625 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110167.log 2023-08-25T10:07:52.627 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.95419.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95419.log.gz 2023-08-25T10:07:52.635 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93000.log 2023-08-25T10:07:52.649 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.110167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101899.log 2023-08-25T10:07:52.649 INFO:teuthology.orchestra.run.smithi071.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110167.log.gz 2023-08-25T10:07:52.650 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.93000.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93000.log.gz 2023-08-25T10:07:52.656 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104106.log 2023-08-25T10:07:52.667 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.101899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99261.log 2023-08-25T10:07:52.670 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.104106.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101899.log.gz 2023-08-25T10:07:52.680 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109687.log 2023-08-25T10:07:52.681 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104106.log.gz/var/log/ceph/ceph-client.admin.99261.log: 2023-08-25T10:07:52.682 INFO:teuthology.orchestra.run.smithi071.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99261.log.gz 2023-08-25T10:07:52.710 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103783.log 2023-08-25T10:07:52.712 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.109687.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109687.log.gz 2023-08-25T10:07:52.725 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99387.log 2023-08-25T10:07:52.726 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.103783.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103783.log.gz 2023-08-25T10:07:52.749 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.99387.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99387.log.gz 2023-08-25T10:07:52.752 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-08-25T10:07:53.179 INFO:teuthology.orchestra.run.smithi071.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-08-25T10:07:54.663 INFO:teuthology.orchestra.run.smithi071.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-08-25T10:07:55.584 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-08-25T10:07:55.922 INFO:teuthology.orchestra.run.smithi071.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-08-25T10:07:56.876 INFO:teuthology.orchestra.run.smithi071.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-08-25T10:07:57.289 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-08-25T10:07:57.865 INFO:teuthology.orchestra.run.smithi164.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-08-25T10:07:58.027 INFO:teuthology.orchestra.run.smithi164.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-08-25T10:07:59.650 INFO:teuthology.orchestra.run.smithi071.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-08-25T10:07:59.665 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-08-25T10:07:59.668 INFO:teuthology.orchestra.run.smithi071.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-08-25T10:08:00.270 INFO:teuthology.orchestra.run.smithi071.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-08-25T10:08:00.335 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-08-25T10:08:00.340 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-08-25T10:08:00.340 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m9.599s 2023-08-25T10:08:00.340 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m43.227s 2023-08-25T10:08:00.340 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m3.372s 2023-08-25T10:08:00.840 INFO:teuthology.orchestra.run.smithi071.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-08-25T10:08:16.945 INFO:teuthology.orchestra.run.smithi071.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.0.98567.log.gz 2023-08-25T10:08:16.950 INFO:teuthology.orchestra.run.smithi071.stderr: 2023-08-25T10:08:16.951 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m26.215s 2023-08-25T10:08:16.951 INFO:teuthology.orchestra.run.smithi071.stderr:user 1m10.746s 2023-08-25T10:08:16.951 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m6.990s 2023-08-25T10:08:16.952 INFO:tasks.ceph:Archiving logs... 2023-08-25T10:08:16.953 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/remote/smithi071/log 2023-08-25T10:08:16.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-25T10:08:23.987 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/remote/smithi164/log 2023-08-25T10:08:23.989 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-25T10:08:28.496 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi071 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-08-25T10:08:28.509 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-08-25T10:08:28.529 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_rishabh-d-dave_ceph_4b369cf18ed1391a426ab4ae86da834e9c074f81/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-08-25T10:08:28.539 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-08-25T10:08:28.540 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-25T10:08:28.557 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-25T10:08:28.717 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-08-25T10:08:28.797 INFO:teuthology.task.clock:Checking final clock skew... 2023-08-25T10:08:28.798 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T10:08:28.809 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-25T10:08:28.859 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2023-08-25T10:08:28.869 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T10:08:28.870 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2023-08-25T10:08:28.870 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +3091us[+3091us] +/- 79ms 2023-08-25T10:08:28.870 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -2376us[-2408us] +/- 53ms 2023-08-25T10:08:28.871 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +200us[ +168us] +/- 64ms 2023-08-25T10:08:28.871 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T10:08:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2023-08-25T10:08:28.879 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-25T10:08:28.880 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2023-08-25T10:08:28.880 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +3061us[+3061us] +/- 79ms 2023-08-25T10:08:28.880 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -2356us[-2361us] +/- 53ms 2023-08-25T10:08:28.881 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +180us[ +175us] +/- 64ms 2023-08-25T10:08:28.881 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-08-25T10:08:28.883 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-08-25T10:08:28.903 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-08-25T10:08:28.904 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-08-25T10:08:28.927 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-25T10:08:29.212 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-25T10:08:29.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T10:08:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T10:08:29.591 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-25T10:08:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T10:08:29.701 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-08-25T10:08:29.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-08-25T10:08:29.773 INFO:teuthology.task.internal:Duration was 1987.922572 seconds 2023-08-25T10:08:29.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-08-25T10:08:29.808 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-08-25T10:08:29.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-25T10:08:29.818 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-25T10:08:29.984 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T10:08:29.992 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-25T10:08:30.351 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-08-25T10:08:30.352 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2023-08-25T10:08:30.353 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-25T10:08:30.420 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2023-08-25T10:08:30.421 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-25T10:08:30.508 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-08-25T10:08:30.509 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-25T10:08:30.518 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-25T10:08:30.608 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-08-25T10:08:30.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-25T10:08:30.614 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-25T10:08:31.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-08-25T10:08:31.483 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-25T10:08:31.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-25T10:08:31.577 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-25T10:08:31.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-08-25T10:08:31.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-25T10:08:31.699 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-25T10:08:31.799 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2023-08-25T10:08:31.811 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2023-08-25T10:08:31.884 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-25T10:08:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T10:08:31.939 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-25T10:08:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-25T10:08:31.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-08-25T10:08:32.010 INFO:teuthology.task.internal:Transferring archived files... 2023-08-25T10:08:32.011 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/remote/smithi071 2023-08-25T10:08:32.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-25T10:08:32.120 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327/remote/smithi164 2023-08-25T10:08:32.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-25T10:08:32.230 INFO:teuthology.task.internal:Removing archive directory... 2023-08-25T10:08:32.230 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-25T10:08:32.240 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2023-08-25T10:08:32.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-08-25T10:08:32.308 INFO:teuthology.task.internal:Not uploading archives. 2023-08-25T10:08:32.308 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-08-25T10:08:32.336 INFO:teuthology.task.internal:Tidying up after the test... 2023-08-25T10:08:32.338 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-25T10:08:32.347 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-25T10:08:32.398 INFO:teuthology.orchestra.run.smithi071.stdout: 263665 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 25 10:08 /home/ubuntu/cephtest 2023-08-25T10:08:32.407 INFO:teuthology.orchestra.run.smithi164.stdout: 264347 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 25 10:08 /home/ubuntu/cephtest 2023-08-25T10:08:32.411 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-08-25T10:08:32.422 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-25T10:08:32.449 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2023-08-25T10:08:32.449 INFO:teuthology.nuke:Checking targets against current locks 2023-08-25T10:08:32.478 DEBUG:teuthology.nuke:shortname: smithi071 2023-08-25T10:08:32.478 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T10:08:32.500 DEBUG:teuthology.nuke:shortname: smithi164 2023-08-25T10:08:32.501 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-25T10:08:32.504 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 09:26:12.730620', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGK5rZRhD9wTEpAxb8Xg5U6DSQJKyvneMkKBiMVKngkXaBc4/SOTGnZ0oZgIeugfCUb2mfTjamoEClErqT06Dgg='} 2023-08-25T10:08:32.523 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-25_06:38:25-fs-wip-rishabh-2023aug3-b5-testing-default-smithi/7379327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-25 09:26:12.732064', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKTJ01r1w1pvwRpAElBNol36OwGpEbAZ+Te5ggmgsQD6Q7jm4EX3vtmJk4+R/SANSZWTU3UzOFyPs4IP3cYEwg='} 2023-08-25T10:08:32.555 INFO:teuthology.orchestra.console.smithi071:Power off 2023-08-25T10:08:32.556 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-25T10:08:32.579 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2023-08-25T10:08:32.580 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:32.598 INFO:teuthology.orchestra.console.smithi164:Power off 2023-08-25T10:08:32.599 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-25T10:08:32.608 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-08-25T10:08:32.619 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2023-08-25T10:08:32.620 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:32.642 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2023-08-25T10:08:36.610 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:36.643 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:36.723 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-08-25T10:08:36.757 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2023-08-25T10:08:40.724 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:40.758 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:40.837 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2023-08-25T10:08:40.871 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2023-08-25T10:08:40.972 INFO:teuthology.orchestra.console.smithi164:Power off completed 2023-08-25T10:08:44.840 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-25T10:08:44.951 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2023-08-25T10:08:45.052 INFO:teuthology.orchestra.console.smithi071:Power off completed 2023-08-25T10:08:45.159 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1987.9225718975067 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi071 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b369cf18ed1391a426ab4ae86da834e9c074f81 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default mon.a-kernel-sha1: a6a69670761025e039dc5ae86619b0cf04c46dc2 mon.b-kernel-sha1: a6a69670761025e039dc5ae86619b0cf04c46dc2 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b118dbef594b45b7b0dd722658514201 status: fail success: false 2023-08-25T10:08:45.160 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-25T10:08:45.241 INFO:teuthology.run:FAIL